On 29 October 2026 we fetched the robots.txt of 151 large websites (news, shops, SaaS, docs, social networks, education, SEO) and ran every file through a linter. 135 files could be read.
The short version
- 44% block at least one of the six best-known AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, OAI-SearchBot) from
/. - ClaudeBot is blocked more often than GPTBot: 55 sites (41%) vs. 42 (31%).
- Training yes, search no: 22 sites block GPTBot but leave OAI-SearchBot (ChatGPT search) open. 20 block both.
- News blocks, software does not: 19 of 34 news sites block GPTBot, 2 of 33 SaaS sites do, and no developer-docs or SEO site does.
- A third carry dead lines: 44 files still name
anthropic-aiorClaude-Web, which are not on Anthropic's current crawler list. - The syntax is fine: one line in 135 files silently does nothing. The bigger issue is delivery: 9 sites answered
/robots.txtwith HTTP 403, 3 with an HTML page.
Who blocks which crawler
| Crawler | Blocked from / | Share of 135 |
|---|---|---|
| ClaudeBot | 55 | 41% |
| Bytespider | 55 | 41% |
| CCBot | 53 | 39% |
| Applebot-Extended | 48 | 36% |
| meta-externalagent | 43 | 32% |
| GPTBot | 42 | 31% |
| Google-Extended | 38 | 28% |
| PerplexityBot | 38 | 28% |
| Claude-SearchBot | 30 | 22% |
| ChatGPT-User | 26 | 19% |
| OAI-SearchBot | 20 | 15% |
| Googlebot | 1 | 1% |
Crawlers that collect training data are blocked two to three times as often as the ones behind AI search and user-triggered fetches.
A 403 on robots.txt does not keep bots out
Nine sites, including cnn.com, nih.gov and npmjs.com, answered our identifiable non-browser client with HTTP 403 for /robots.txt. Under RFC 9309 a 4xx answer means a crawler may fetch everything, and Google treats it the same way. A firewall that blocks /robots.txt hides the rules instead of enforcing them. (Real crawlers may get a different answer; we can only see what an unknown client gets.)
Gzip nobody asked for
www.forbes.com sends its robots.txt gzip-compressed even when the request asks for no compression. Real crawlers decode it. Our own CLI did not, until this study: before 0.3.1 it read the file as binary and found no rules. If you test robots.txt with your own scripts, decode Content-Encoding.
Check your own file
npx crawlerpeek lint https://example.com # syntax, legacy tokens, extensions npx crawlerpeek scan https://example.com # every sitemap URL against robots.txt Or in the browser, without installing anything: robots.txt validator and sitemap robots.txt check.
The full write-up with the sector table, the 13 sites that block every AI crawler, Content-Signal and RSL adoption, and the method is on crawlerpeek.com. The CLI and MCP server are open source: github.com/crawlerpeek/crawlerpeek.
