A robots.txt that allows GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot only tells you what the site permits. It says nothing about what the server actually serves. A WAF or CDN bot rule can return a 403 to a crawler that robots.txt welcomes, and you would never see it in the file.

Here is a small script that checks both layers for any URL: what robots.txt says per bot, and what HTTP status the server returns when the request carries that bot's user agent string.

The script 

import sys import requests from protego import Protego # pip install protego requests  BOTS = { "GPTBot": "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)", "OAI-SearchBot": "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)", "ClaudeBot": "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)", "PerplexityBot": "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)", } def check(base, path="/"): robots = requests.get(base + "/robots.txt", timeout=10).text rp = Protego.parse(robots) rows = [] for name, ua in BOTS.items(): allowed = rp.can_fetch(base + path, name) r = requests.get(base + path, headers={"User-Agent": ua}, timeout=10) rows.append((name, allowed, r.status_code, len(r.text))) return rows if __name__ == "__main__": base = sys.argv[1].rstrip("/") path = sys.argv[2] if len(sys.argv) > 2 else "/" bad = 0 print(f"{'bot':15s} {'robots.txt':11s} {'HTTP':5s} bytes") for name, allowed, status, size in check(base, path): flag = "" if (allowed and status == 200) else " <-- check" bad += bool(flag) print(f"{name:15s} {'allow' if allowed else 'DISALLOW':11s} {status:<5d} {size}{flag}") sys.exit(1 if bad else 0)

Run it with python check_ai_access.py https://example.com /some-article. It exits with code 1 if any bot is disallowed or gets a non-200 response, so you can drop it into CI.

What the output looks like

I tested the script against a local server that allows everything in robots.txt but returns a 403 to any request with an OAI-SearchBot user agent, which is what a strict WAF rule can do:

bot robots.txt HTTP bytes GPTBot allow 200 31 OAI-SearchBot allow 403 14 <-- check ClaudeBot allow 200 31 PerplexityBot allow 200 31

robots.txt says yes for all four. The server says no to one. That is the gap this check is built to catch.

Why Protego and not urllib.robotparser

Python's built-in urllib.robotparser reads rules top to bottom and stops at the first match, so an early Allow: / can hide a later, more specific Disallow. Protego follows the longest-match behaviour described in RFC 9309, which is closer to how real crawlers read the file.

What this check does not prove

Be careful with the result:

  • It sends the bot's user agent string from your own IP. Many CDNs verify real bots by IP range or reverse DNS, so a 200 here does not guarantee the real crawler gets a 200. A 403 here is a stronger signal than a 200 is a reassurance.
  • A 200 does not mean the content is readable. If the article text only appears after JavaScript runs, a crawler that does not render JS gets an empty shell. Compare the byte count against a normal browser fetch.
  • Your server logs are the ground truth. Filter them by user agent and status code to see what the real crawlers received.

What to do with a flagged row

  1. Check your WAF or CDN bot management rules for the user agent token.
  2. Allow the crawlers you want by verified IP range or the CDN's verified-bot category, instead of loosening rules for everyone.
  3. Re-run the script, then confirm in the logs after the next real crawl.

If your robots.txt has several named groups, also check that each blocked path is repeated in every named group, because a crawler reads only the group that matches its own name and ignores *.