TL;DR: "Block AI bots" is three different decisions, not one. Training crawlers, AI search crawlers and user-triggered fetchers each do a different job. Block the wrong one and your product disappears from ChatGPT and Google's AI answers, while your content still ends up in training data anyway.
Why is this suddenly a developer problem?
For years, robots.txt was a set-and-forget file. Then AI crawlers arrived, and someone on the marketing or legal side asked engineering to "block the AI bots".
The traffic numbers explain the pressure. Cloudflare's July 2026 crawler report found that 52% of crawler requests on its network were for AI training in June 2026, up from 22% in spring 2025. More than half of all web traffic is now non-human.
(Note that Cloudflare sells bot-management products and argues for blocking by default, and the figures cover only its own network. They are still the best public view we have.)
The catch: the quick fix, a blanket Disallow for anything that looks like AI, usually blocks the bots that put you into AI answers.
What are the three types of AI bot?
Before touching the file, label every AI user agent by what it actually does:
| Type | What it does | Example user agents | If you block it |
|---|---|---|---|
| Training | Collects pages to train future models | GPTBot | Your content is (per the vendor) not used for training. AI answers are unaffected in the short term. |
| Search / answer | Indexes pages so the assistant can cite them in live answers | OAI-SearchBot | You stop appearing in ChatGPT search answers |
| User-triggered | Fetches a page when a user asks the assistant about it | ChatGPT-User | Inconsistent: OpenAI notes that robots.txt rules may not apply to these fetches |
Google works differently, and this is where most teams slip up:
-
Googlebotgoverns Search, including AI Overviews and AI Mode. Block it and you vanish from all three. -
Google-Extendedis a control token for some of Google's AI uses. Blocking it does not remove you from AI Overviews or AI Mode.
Vendors rename and add agents regularly. Check the official lists, such as OpenAI's crawler docs and Google's crawler overview, before you edit anything.
What does a sensible default look like?
If you're a business whose buyers use ChatGPT or Gemini to research vendors, you usually want to be found in AI answers. A reasonable starting point:
# --- AI search / answer crawlers: allow (keeps you in live AI answers) User-agent: OAI-SearchBot Allow: / # --- User-triggered fetchers: allow (a real person asked about your page) User-agent: ChatGPT-User Allow: / # --- Training crawlers: a business decision (see below) # Option A: allow (your content can shape how future models describe you) User-agent: GPTBot Allow: / # Option B: block training only # User-agent: GPTBot # Disallow: / # --- Google: never block Googlebot if you want Search + AI Overviews User-agent: Googlebot Allow: / # Google-Extended: opt-out token for some AI uses; does NOT affect AI Overviews # User-agent: Google-Extended # Disallow: / # Keep private areas private for everyone User-agent: * Disallow: /admin/ Disallow: /api/ Treat this as a template, not gospel. Publishers whose content is the product (news, research, courses) have a very different calculation from a SaaS or services company.
Why isn't robots.txt enough on its own?
This is the step most audits miss: your CDN or WAF can override robots.txt.
If Cloudflare, Akamai, Fastly or your WAF has a "block AI bots" toggle switched on, your carefully written Allow lines do nothing. The bot never reaches the file. So check both layers:
- robots.txt: what you say is allowed
- CDN / firewall bot rules: what is actually allowed through
A quick way to see what a bot experiences is to request a page with its user agent and check the status code:
curl -s -o /dev/null -w "%{http_code}\n" \ -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \ https://example.com/ A 403 or a challenge page means something upstream is blocking it, whatever robots.txt says. This isn't a perfect test: some CDNs verify real bots by IP address, so a spoofed user agent can be treated differently. It does catch the most common misconfiguration, a blanket bot block.
What's a 5-step checklist for your next sprint?
- Read robots.txt and the CDN/WAF bot settings together. The network-level rule wins.
- List every AI user agent you allow or block, and tag each one training, search or user-triggered.
- Allow search crawlers and user fetchers unless you have a specific reason not to.
- Make the training decision with the business, meaning whoever owns content licensing. It isn't purely an engineering call.
- Recheck after every change and every quarter. New agents appear, and defaults change.
How can you check your site quickly?
If you want a quick read on where you stand, the team at Sigzen AI has written up the full decision table behind this post, including what happens with each block/allow combination.
They also built a free AI crawler check. Enter a domain and it reads your robots.txt against 16 named AI crawlers (OpenAI, Google, Microsoft, Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance, Common Crawl and others), returning allowed, blocked or partly blocked for each. It only reads robots.txt, so pair it with the curl test above to catch CDN-level blocks.
How does your team handle this? Do you block training bots, allow everything, or let the CDN decide? I'd like to hear in the comments.
Written by the Sigzen AI editorial team. Sigzen AI helps B2B brands get described accurately in ChatGPT, Gemini, Copilot and Google AI answers, and builds AI agents for operations teams. More of our research is at sigzenai.com/insights.

