Every page we serve advertises an RSS feed our own robots.txt forbids

CogniPrep's robots.txt is generated by a 122-line app/robots.ts and arrives at the edge as 850 lines and 24,687 bytes. 48 of those lines are Allow and 796 are Disallow, because the whole policy is written twice: once for User-Agent: * and once for User-Agent: Bingbot, which crawls more predictably when it has an explicit group. The two groups hold 422 rules each and are identical, rule for rule.

A rules file that size is not something you read. It is something you query. So I wrote the matcher.

The matcher

Three rules are enough for the real thing:

1. A pattern is a literal prefix, with * matching any run of characters and $ anchoring the end.
2. Of all matching rules, the longest pattern wins.
3. If an Allow and a Disallow tie on length, Allow wins.

def decide(rules, path): best = None for kind, pattern, regex in rules: if regex.match(path): length = len(pattern.replace("*", "").replace("$", "")) if best is None or length > best[0] or (length == best[0] and kind == "allow"): best = (length, kind, pattern) return best or (0, "allow", "(default)") 

Then I fed it every URL in our live sitemap. All 530 are allowed, in both groups. That is the result you want and it is also the boring one, so I ran the inverse query: what does this file block that we also point at?

1. Our own feed

Disallow: /feed.xml 

The root layout puts this in the <head> of every page:

<link rel="alternate" type="application/rss+xml" title="CogniPrep Blog RSS Feed" href="/feed.xml" /> 

I sampled 25 of the 530 live URLs and all 25 carry it. The feed itself answers 200 and lists our articles. So every page on the site tells a crawler the feed exists, and the rules file in the same deployment tells that crawler not to fetch it.

A feed reader a person has subscribed in their own client is not affected. A search engine that discovers feeds by following rel="alternate" is: it finds the pointer and then declines to read the target. The Disallow was added with /manifest.ts in a block commented "Feed and manifest files", and nothing was pointing at the feed at the time.

2. A wildcard that reaches further than it looks

Disallow: /*.json 

That is 6 characters of pattern after the wildcard is stripped, which beats Allow: / at 1. It covers the JSON our API returns, which is the intent, and it also covers /.well-known/assetlinks.json, the file Android reads to associate the installed app with the domain. Android does not consult robots.txt, so nothing breaks. It is still a rule with a wider blast radius than its comment suggests.

3. The 372 rules that are right

The longest block in the file is one Disallow per practice game, because each game route is auth-gated and a crawler that fetches one only sees a shell. Those lines are generated, not typed:

const GAME_ROUTES = GAME_LIBRARY.map((game) => game.route).sort(); 

I checked the invariant the generation is supposed to buy, by diffing the 372 /games/ disallow lines in the live file against the 372 route values in the library. Both directions of comm came back empty. A game added tomorrow is disallowed the day it ships, which is exactly what the comment claims.

4. The claim right under it that is not true

The same comment block ends with a sentence about defence in depth:

The pages also send robots: noindex, nofollow and the public pages that link to them use rel="nofollow", so nothing depends on this file alone.

The first half is enforced in page metadata. The second half is a convention, and conventions are not checked by anything. So I checked it: I pulled every <a href> out of all 530 live pages, kept the ones pointing at a /games/<game-id> route, and read the tag.

13 pages carry 30 such links. Not one has a rel attribute of any kind.

They are all hand-written <Link href={game.route}> calls in individual page files, most of them from our newest batch of employer pages. Nothing is leaking: the targets are auth-gated and carry noindex. What it costs is internal linking. A link into a disallowed path is a dead end for a crawler, so those 30 links do nothing for the pages that spend them, while the provider hub three paragraphs further down the same page is the page we actually want crawled.

The fix is not interesting (either rel="nofollow" or link the hub instead). The interesting part is that a generated block of 372 rules held perfectly and the sentence of prose next to it did not.

See it

curl -s https://cogniprep.app/robots.txt | wc -l # 850 curl -s https://cogniprep.app/robots.txt | grep -c '^Disallow' # 796 curl -s https://cogniprep.app/robots.txt | grep -n 'feed.xml' curl -sI https://cogniprep.app/feed.xml | head -1 # 200 

Then open the SPAR employer guide, view source and search for /games/gitp-. Five links, no rel on any of them. The page you want a crawler to reach from there is the GITP hub, which is the one link on the page that robots.txt allows.

Last thing worth copying from this: the matcher is 10 lines and it turned a 24 KB file into a question I can ask in a test. If your robots.txt is generated from application data, the sitemap is the input you should be feeding it.