Sanu VargheseIT & AI notes
Blog › AI
AI1 November 2026 · 5 min read

One robots.txt line to opt out of AI training: what it does and what it doesn't

If you look after a website, there has never been a simple way to say "please don't train AI on this". You had to know every AI company's crawler name and write a rule for each one. A new proposal, now backed by Google, OpenAI, Microsoft and Anthropic, aims to fix that with one line in robots.txt. Here's what it does, what it doesn't, and what I'd do in the meantime.

The problem it solves

Today, opting out of AI training means a list of separate rules. OpenAI has GPTBot. Google has Google-Extended, which is a control token rather than a crawler of its own: Google says it decides whether content it crawls can be used to train future Gemini models and for grounding, and that it does not affect whether a site appears in Google Search. Every new AI company adds another name to learn, and a site owner who misses one has, in effect, said yes.

What's new

On 30 October, Kevin Kelley, Anthropic's technical lead for standards, published an IETF draft and announced it on the IETF's AI preferences mailing list. It adds a special group to robots.txt using the token __usage_ai-training. You list the paths you don't want used for AI training under that group, and every company that honours the signal reads the same rule. The example in the reporting blocks a whole site except its press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The timelines differ by company:

  • Microsoft says it will honour the preference from 1 February 2027, replacing the NoArchive and NoCache tags it has used since 2023.
  • Google says early 2027, and its crawler documentation already says it intends to support the signal. If you use the token without mentioning Google-Extended, Google will treat it as a block on Google-Extended.
  • OpenAI says "in the coming months", and its existing GPTBot controls still apply.
  • Anthropic: the reports differ. Engadget says Anthropic has already started supporting it. The Next Web quotes Kelley saying support "will take us some time". I'd treat it as not yet guaranteed.

What it doesn't do

This is the part worth reading twice.

  • It is a preference, not a lock. It says how you want your content used. It doesn't stop crawling, and it relies on companies choosing to honour it.
  • It doesn't reach backwards. As Engadget puts it, the directive cannot undo any past scraping.
  • Silence means no preference. If your robots.txt has no such group, you have said nothing. Your existing rules for all crawlers don't count.
  • It isn't a standard yet. The draft is an individual submission with no formal standing at the IETF. Separately, Google's Gary Illyes and Mozilla's Martin Thomson have a different IETF working draft that would add a Content-Usage rule to robots.txt. So expect the details to change.
  • It isn't payment. Microsoft says the preferences do not set up licensing or payment. It also warns that blocking AI grounding will significantly reduce your visibility in AI-generated answers, though not your Bing ranking.

There is also a trust question. In submissions reported on 1 October, Australian public broadcasters ABC and SBS told a parliamentary committee that robots.txt rules are "routinely ignored or bypassed". A shared signal helps the companies that play fair. It does nothing about the ones that don't.

What I'd do now

  1. Decide what you actually want. For a business site, being cited in AI answers may be worth more than being left out. For a site with paid or original content, the answer may be different. This is a business decision, so ask the owner, don't guess.
  2. Keep your existing rules. Until the new signal is live everywhere, crawler-specific rules such as GPTBot and Google-Extended are still what works.
  3. Add the new group when you're ready. It does no harm to add it alongside the old rules, as long as you test the file. A broken robots.txt can block search engines too, so check it in Google Search Console after any change.
  4. Use the CDN if you need teeth. If you need to actually stop bots rather than ask politely, look at bot controls at your CDN or firewall. Robots.txt is a request, not a wall.
  5. Write it down. Note what you set and why, so the next person doesn't undo it by accident.

One shared signal is a real improvement on a list of crawler names nobody can keep up with. Just be clear with whoever owns the site about what it does: it states a preference, it doesn't enforce one.

Sources

← All posts
Esc