How our bot works

What ShipwardlyBot is

ShipwardlyBot is the scanner behind Shipwardly surveys. It visits a site when someone asks us to scan it, and every request it makes identifies itself with this User-Agent:

It does not crawl the web or follow links from one site to another. The only scans it runs on a schedule are weekly monitoring scans: once a week, and only of verified sites whose owners have turned weekly monitoring on. Each one first checks that the owner's DNS record or file is still in place, and does not scan if it is not.

What it fetches

On a free scan, only your homepage and the files it points to:

• the homepage HTML and its response headers
• the JavaScript and CSS files that page links to
• the source maps those files name, on your own site only, at most 5 per scan, to confirm they really download

It does not log in, submit forms or guess at other paths. Deeper checks, such as looking for an exposed /.env file, run only after the owner of the site has verified the domain with us.

What a verified scan requests

A verified scan runs only on a site whose owner has proved it is theirs, with a DNS TXT record or a file in /.well-known, and only when that owner starts it. We check the proof again first. Then, besides everything a free scan fetches, it makes this fixed list of requests.

Every one goes to the owner's own site (or their own Supabase project), identifies itself as ShipwardlyBot and never follows a redirect. Each file is confirmed by what comes back, so an ordinary page is never mistaken for an exposed file, and we keep only what we found, never the contents.

Two made-up addresses

/shipwardly-probe-<random> Sent first, to learn how your site answers an address that does not exist.

Environment files

/.env /.env.local /.env.production /.env.backup Confirmed only by lines such as NAME=value. We keep the variable names, never a value.

Git repository

/.git/HEAD /.git/config The second only when the first is really there, to confirm the repository can be downloaded.

Folder index

/.DS_Store Confirmed by the file's own signature.

Backups and database dumps

/backup.zip /backup.tar.gz /site.zip /db.sql /dump.sql /database.sql /wp-config.php.bak /config.php.bak /.env.bak Confirmed by the file's signature (a zip or gzip header, SQL statements), never by its name alone.

Diagnostic pages

/phpinfo.php /info.php Confirmed only by a real phpinfo() page.

Folders

/assets/ /static/ /uploads/ /images/ /js/ /css/ /files/ Your site's own asset folders first, then these, at most 8 in all, to see whether any lists its contents. We count the entries and keep no listing.

Admin addresses

/admin /administrator /wp-admin/ /dashboard /backoffice /manage /phpmyadmin/ /adminer.php /_admin /cms Only to see whether each one asks for a login.

Your own API

Up to 5 addresses under /api or /rest that your own code calls, one GET each with the header Origin: https://shipwardly-cors-check.invalid, an address no one can own. To see which other websites they trust. Never with cookies or credentials.

Supabase, if your app uses it

One GET of https://<your-project>.supabase.co/rest/v1/ for the list of tables, when your project publishes it, then one HEAD per table your code names, at most 40: /rest/v1/<table>?select=*&limit=1 with Prefer: count=exact. With the public anon key already in your page source, sent only to your own *.supabase.co project. A HEAD request returns no rows: we record whether each table answered and its row count, never any of your data.

The limits it keeps to

• At most one anonymous scan of a domain every 10 minutes.
• A 10 second timeout on each request.
• At most 5 MB downloaded per file.
• On a verified scan, at most 60 requests to your site and 45 to your Supabase project, within the same 90 seconds as the rest of the scan.
• It never uses keys it discovers. The one exception is the Supabase check a verified owner authorises, which uses the public anon key your site already publishes, and asks only for a row count.

How to opt out

Add this DNS TXT record to your domain and ShipwardlyBot will not scan it. We check for it before every scan.

If you cannot change your DNS, or you think the bot has misbehaved, email [email protected] with your domain and we will add it to our block list by hand.