On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt, based on work from the IETF's AIPREF working group. Coverage the same day reported that Microsoft and OpenAI will honour it too, and that Anthropic has started building support.
If you build or maintain websites, here's what it means and what to do (all free).
The rule
User-Agent: __usage_ai-training Disallow: / Instead of a crawler name, the User-Agent line holds a policy token. Normal robots.txt path matching applies, so you can opt out site-wide and still allow a folder:
User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ Timeline (it's not live yet)
- Microsoft: from February 1, 2027
- Google: "early 2027" per its crawler documentation note; the blog says 2027
- OpenAI: "in the coming months"
- Anthropic: implementation started, no date
It's an IETF draft, so the exact token or details could still change.
The Google-Extended catch
Google says that if you use the shared token to opt out and don't explicitly specify Google-Extended, it will treat that as if you disallowed Google-Extended. Google also says sites that allow Google-Extended are eligible to appear and receive clicks via grounding in Gemini Apps.
So if you want to opt out broadly but still be cited in Gemini, you'd add an explicit allow, knowing that it also lets Google use the content for Gemini:
# Opt out of AI training with all participating AI providers User-Agent: __usage_ai-training Disallow: / # Optional: still allow Google's Gemini (training + citations in Gemini Apps) User-Agent: Google-Extended Allow: / Checklist
- Open
yoursite.com/robots.txtand see what's already there. - Decide: protect your content from training, or stay visible in AI answers?
- If opting out, add the shared token group.
- Decide on Google-Extended explicitly, so nothing happens by default.
- Google Search rankings aren't affected. Re-check the spec in early 2027.
What would you choose for your projects: block, allow, or a mix by folder?
