User-agent: * doesn't apply to crawlers you named in robots.txt

A Disallow under User-agent: * does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and * is only the fallback for crawlers you didn't name.

I learned this from my own logs. I had blocked my /go/ redirect links under *, and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships.

The version that doesn't work

User-agent: Googlebot Allow: / User-agent: ClaudeBot Allow: / User-agent: * Allow: / Disallow: /go/ 

It's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under *. In this file, /go/ is allowed for both Googlebot and ClaudeBot.

Both specs say so:

• RFC 9309, section 2.2.1: if no matching group exists, crawlers MUST obey the * group.
• Google's robots.txt spec: only one group is valid for a particular crawler, and the others are ignored.

Groups don't inherit from each other, and * is not a shared base.

The fix

Write each blocked path into every named group.

User-agent: Googlebot Allow: / Disallow: /go/ User-agent: ClaudeBot Allow: / Disallow: /go/ User-agent: * Allow: / Disallow: /go/ 

Keeping Allow: / next to Disallow: /go/ is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for /go/amazon/... the four-character Disallow: /go/ beats the one-character Allow: /.

Once Googlebot re-read the fixed file, it stopped:

It took about four hours to stop, which fits Google caching robots.txt for up to 24 hours.

A script that finds the group you forgot

With a dozen named groups, you will eventually miss one. This script collects every User-agent in the file and checks the paths you want blocked against each of them:

import re import sys from protego import Protego # pip install protego  def named_agents(text): """Every User-agent name in robots.txt except *.""" names = re.findall(r"(?im)^\s*user-agent\s*:\s*([^\s#]+)", text) return sorted({n for n in names if n != "*"}, key=str.lower) def check(text, base, paths): rp = Protego.parse(text) agents = named_agents(text) + ["SomeUnnamedBot"] # last one exercises the * group  leaks = [] for ua in agents: for p in paths: ok = rp.can_fetch(base + p, ua) print(f"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}") if ok: leaks.append((ua, p)) return leaks if __name__ == "__main__": robots_file, base, *paths = sys.argv[1:] leaks = check(open(robots_file, encoding="utf-8").read(), base, paths) if leaks: print(f"\n{len(leaks)} blocked path(s) are still allowed") sys.exit(1) 

python check_groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY 

On my old file, all 13 named groups came back allow and only SomeUnnamedBot was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build.

Don't use urllib.robotparser for this

Python's standard urllib.robotparser says everything is allowed, even with the fixed file:

from urllib.robotparser import RobotFileParser rp = RobotFileParser() rp.parse(open("robots.txt", encoding="utf-8").read().splitlines()) rp.can_fetch("Googlebot", "/go/amazon/B0CL6H35DY") # True, even after the fix 

It reads a group top to bottom and stops at the first matching line. With Allow: / first, it never reaches Disallow: /go/. Use Protego, the longest-match parser Scrapy uses. I tested with 0.7.0.

nofollow and X-Robots-Tag don't stop the fetch

My redirect links had rel="sponsored nofollow" and returned X-Robots-Tag: noindex, nofollow. Googlebot still fetched 67 different redirect URLs.

• Since March 1, 2020, Google treats nofollow as a hint for crawling. Google decides whether to follow.
• X-Robots-Tag comes back with the response, so by the time a crawler reads it, the request has already been made.

If you need the request itself to stop, use robots.txt.

The hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot (it stopped two days earlier), and the unnamed GoogleOther that showed up anyway are on Aulvem → Aulvem | User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times