Published by the HackerMe team (Agura), the makers of HackerMe, a free automated website security scanner. The checks below are vendor-neutral. Each one is labeled: **safe on any public site* (it only reads what the site already publishes to everyone), your own sites only (it requests paths that could return sensitive data, so run it only on systems you own or have written permission to test), or maintainer check (it needs access to the server, the deployment pipeline or the repository).*

Files that were never meant to be public sometimes end up in the web root: a .git directory copied with a deployment, a .env file, a backup.zip left after a migration, or an editor copy such as config.php~. The OWASP Web Security Testing Guide points out a specific risk with copies like login.asp.old: the server may return them as plain text instead of executing them, which hands out source code. This post covers six checks, what you should see, what counts as a finding and what a clean result does not prove.

Ground rules before you start

  • Request a single, small file per path and stop there. Do not download a whole exposed repository or open archive contents on a system you are not responsible for.
  • If you find real secrets, treat them as compromised. Deleting the file does not undo the exposure.

1. Read robots.txt (safe on any public site) 

curl -s https://example.com/robots.txt
  • What to look for: Disallow lines that name paths such as /backup, /admin or /old.
  • Finding: robots.txt is being used to hide something sensitive. RFC 9309 is explicit that these rules "are not a form of access authorization" and that listing paths "exposes them publicly and thus makes the paths discoverable."
  • Doesn't prove: anything about paths that are not listed. A short robots.txt is not evidence that nothing is exposed.

2. Git metadata (your own sites only)

A Git repository keeps its history in the .git directory, including the object store (Git: repository layout). Request only the small text files:

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/.git/HEAD curl -s https://example.com/.git/config | head -n 5
  • Expected: 404 or 403, with no repository content in the body.
  • Finding: HEAD returns something like ref: refs/heads/main, or config returns a [core] section. If the directory is readable, assume the history, and anything ever committed to it, may be retrievable.
  • Doesn't prove: that other version-control folders (for example .svn or .hg) are blocked. Check those separately.

3. Environment and config files (your own sites only) 

for p in .env .env.production .env.local config.php.bak wp-config.php.bak; do printf "%s " "$p"; curl -s -o /dev/null -w "%{http_code}\n" "https://example.com/$p" done
  • Expected: 404 or 403 for every path.
  • Finding: any 200 that returns real KEY=value lines or configuration code.
  • Careful: some sites return 200 with a generic page for every unknown path. Compare the body with a path that certainly doesn't exist, such as /this-should-not-exist-123, before you conclude anything.
  • Doesn't prove: that secrets aren't exposed elsewhere, for example in a JavaScript bundle or a public API response.

4. Backups, archives and editor copies (your own sites only)

OWASP lists common leftover extensions including ~, bak, txt, src, dev, old, inc, orig, copy, tmp and swp, applied "before, after, and instead of" the real extension. Start with names that match your own files:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/backup.zip curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/index.php.old
  • Expected: 404 or 403.
  • Finding: a 200 with a size that matches an archive or a source file. Remove it and find out how it got there.
  • Doesn't prove: that no backup exists under a name you didn't guess. Guessing finds only what you thought to ask for.

5. Directory listing (your own sites only)

Request folders that commonly hold user or build files, such as /uploads/, /backup/ or /static/.

  • Expected: a 403, a 404, or a normal page, with no automatically generated file index.
  • Finding: a page titled "Index of /..." or similar that lists files. It makes enumeration trivial, even if every file looks harmless today.

6. Deployment and repository hygiene (maintainer check)

The external checks above only find what is reachable now. The durable fix lives in the build:

  • Deploy a build artifact, not a working copy of the repository, and keep .git, .env and archives out of it.
  • Add a server rule that denies dot-files and known backup extensions as a second layer.
  • Keep secrets out of the repository in the first place (OWASP Secrets Management Cheat Sheet).
  • Doesn't prove: that secrets were never committed earlier. History keeps them unless it is rewritten.

Example: reading one result 

$ curl -s -o /dev/null -w "%{http_code}\n" https://example.com/.git/HEAD 200 $ curl -s https://example.com/.git/HEAD ref: refs/heads/main

The repository metadata is publicly readable. That is a finding to fix: block the path, remove the directory from the web root, then rotate every credential the repository has ever contained. GitHub's guidance on removing sensitive data also puts revoking or rotating the secret as the first step. A 404 here would only mean this one path is closed. It would say nothing about the rest of the site.

Make it routine

Re-run these checks after deployment changes, migrations or server moves, and keep a list of the paths you check. A clean result is a starting point, not a guarantee.

Disclosure: we are the team behind HackerMe, a free automated website security scanner: https://hackerme.ai. The checks above work without it.

What would you add to this list? We would like to hear it in the comments.