My site answered 200 on all 163 URLs. Crawlers never saw a single page.

For two months I ran a small data site: 163 URLs, 152 generated company pages, a 1.7 MB CSV, valid sitemap.xml, and an IndexNow submission that returned 200 every single week.

Every check I had was green. And every search engine had zero of my pages.

This is the story of what was wrong, because I think it is a common failure and the symptom looks exactly like success.

The clue was one request away

On a hunch I fetched the same file twice:

curl -s https://example.site/robots.txt | wc -c # 26 curl -s "https://example.site/robots.txt?probe=1" | wc -c # 162

Two different files from the same path. The 26-byte one was:

User-agent: * Disallow: /

The 162-byte one was mine, the one I had deployed:

User-agent: * Allow: / Sitemap: https://example.site/sitemap.xml

The host was rewriting the bare path. ?probe=1 bypassed whatever was doing it, so every "is my robots.txt right?" check I had ever run against the bare URL was reading the host's stub, not my file. Crawlers read the bare path. So for two months I was politely asking every crawler in the world to go away, while my dashboard said the site was up.

Three checks confirmed it was the platform, not a cache:

The injected response had no charset, no Cache-Control, no ETag; mine had all three.
Deploying the identical build to a brand-new subdomain produced the same 26 bytes — so it was not a stale CDN object.
The response headers of the injected copy came from the platform's edge, not from my deployment.

The second bug, in the same audit

While auditing I submitted my sitemap to IndexNow and got:

HTTP 422 {"errorCode":"InvalidRequestParameters", "message":"One or more URLs are not related to your verified domain."}

Seven URLs in a 163-URL sitemap still pointed at the old hostname. The sitemap generator only rewrote the section it owned and left a hand-written header block from weeks earlier intact.

And here is the embarrassing part: my own residue check had already reported "0 occurrences". I had written:

Select-String -Path site\*.html -Pattern 'old-host\.com' -SimpleMatch

-SimpleMatch treats the pattern as a literal string — backslashes included. So I was searching for old-host\.com with an actual backslash in it, which matches nothing, ever. The check was a no-op wearing a green badge.

What I changed

The real fix was not the migration. It was making the failure impossible to miss. Every deploy now runs a gate before publishing:

# robots.txt must not block the site, and the bare path must match ?probe=1 if re.search(r"(?im)^\s*disallow:\s*/\s*$", robots): problems.append("robots.txt blocks the whole site") if not re.search(r"(?im)^\s*allow:\s*/\s*$", robots): problems.append("no site-wide Allow: /") _, fresh = fetch(robots_url + "?probe=1") if fresh != raw: problems.append("bare URL differs from ?probe=1 (crawlers read the stale one)") # sitemap must be single-host, and sampled pages must not be noindex hosts = {u.split("/")[2] for u in locs if "://" in u} if hosts != {site_host}: problems.append(f"foreign host in sitemap: {hosts}")

That gate is step 8 of 10 in the weekly pipeline now, and it is allowed to fail the run.

I moved to a host whose robots.txt I actually control end to end (a GitHub Pages user site, served at the domain root — a project site under /repo-name/ would have broken every root-absolute link I have). Same bytes, different owner:

$ curl -s https://my-site/robots.txt User-agent: * Allow: / Sitemap: https://my-site/sitemap.xml

Four things I would tell my past self

1. 200 OK is not a statement about crawlability. It is a statement about HTTP. Everything that decides whether a page enters an index — robots.txt, noindex, canonical tags, sitemap host consistency, whether the CDN serves a different body on the bare path — lives beside the status code, not in it.

2. Verify what machines read, not what your browser showed you. A browser hides this class of bug completely, because you looked at ?probe=1 when you debugged, and the crawler looks at /.

3. Compare the bare URL with a query-string URL. It costs one request and catches platform-injected responses, stale edge objects, and rewrite rules. I now do it on robots.txt on every deploy.

4. A dashboard that is green in every cell can hide a total failure. My "163/163 URLs 200" metric was true and useless. The metric that would have caught this is "does the file a crawler fetches contain what I deployed".

The upside of finding it

The audit also forced two fixes I would not have made otherwise: a sitemap that can never drift to a foreign host, and a robots.txt with no BOM (a UTF-8 BOM in robots.txt silently breaks the first directive, which is its own quiet version of this bug).

The dataset I was trying to get indexed is public: 8,040 job postings that state a salary, from 266 companies, normalised to annual USD — dataset page, raw CSV on GitHub, CC BY 4.0.

If you are running a content site and your traffic is zero while every check is green, spend one request on robots.txt?probe=1. It is the cheapest test in this post and it took me two months to run it.