AI Training Opt-Out: 4 AI Giants Agree on One robots.txt Rule

For years, website owners who wanted to keep their content out of AI training had to chase a moving list of crawler names, one company at a time. That may finally be ending. On October 30, Google, Microsoft, OpenAI and Anthropic each said they will honor a single, shared AI training opt-out signal written into robots.txt, based on a proposal Anthropic submitted to the Internet Engineering Task Force (IETF) the same day.

The signal is one new line any site can add today. Support rolls out unevenly through 2027, and it only applies to future training, not to anything already collected.

Key takeaways

• A new robots.txt user-agent, __usage_ai-training, lets a site say “don’t train generative AI on this” to every provider that honors it.
• Anthropic has started implementing it. Microsoft starts on February 1, 2027. Google is aiming for early 2027, and OpenAI says “in the coming months.”
• It controls how content is used, not whether it is crawled, and it does not affect normal search rankings.
• The proposal is an individual IETF draft, not yet an official standard, and private licensing deals can override it.

What happened

Kevin Kelley, Anthropic’s technical lead for standards, published an Internet-Draft titled “Machine-Readable Training Preference Signal” and announced on the IETF’s AI preferences mailing list that Anthropic has begun implementing it. The same day, the other three companies published their own commitments: Google on its Search Central blog, Microsoft on the Bing blog, and OpenAI in a post on its site.

The mechanism is deliberately boring. It adds no new grammar to robots.txt. Instead, it reuses the existing format (standardised as RFC 9309) and introduces a special “policy token” that sits where a crawler name would normally go:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example, taken from the draft, tells every participating company not to use the site for generative AI training except for the press kit folder. The normal path rules and precedence of robots.txt still apply, so you can be as broad or as granular as you like.

How each company will apply it

The four commitments share one token but differ in timing and detail:

• Anthropic: Implementation has started, with no public launch date. The token works alongside its existing ClaudeBot, Claude-User and Claude-SearchBot controls.
• Google: According to Google’s John Mueller, if a site opts out with the shared token and doesn’t name Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google says it aims to support the control in 2027, and its crawler documentation points to early 2027, as Search Engine Roundtable reports. Note that Google-Extended also governs grounding in Gemini Apps and Vertex AI, so the opt-out may reach further at Google than “training” alone suggests.
• Microsoft: Starts honoring the preference on February 1, 2027, and goes furthest by adding separate page-level controls for training and for “grounding” (using content to inform AI answers). Its post says, “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”
• OpenAI: Plans to implement the signal “in the coming months” and says its existing robots.txt controls remain in place.

Why it matters

Until now, opting out meant knowing every crawler’s name (GPTBot, Google-Extended, ClaudeBot and many more), keeping that list current, and hoping new entrants published theirs. A single signal that several of the largest model makers agree to read moves the default burden from publishers toward AI companies. Any new provider that wants to look responsible now has an obvious line to honor.

The draft also makes a useful split between crawling and use. The token is not a crawl directive. A site can stay fully crawlable and searchable while saying it does not want its pages used to change the learned weights of a generative model. Microsoft states plainly that neither the training nor the grounding preference affects eligibility for regular Bing results. For publishers who have watched AI answers eat into search traffic, that separation matters.

Critics have a fair point too. As Paul Thurrott noted, the change arrives years after these companies collected the open web at scale. The signal only applies to content retrieved after it takes effect, and it is voluntary. Crawlers that ignore robots.txt today will keep ignoring it.

The fine print you should know

• Wildcards don’t count. Rules under User-Agent: * are not read as an AI training preference. You must name the token explicitly.
• Training is not grounding. The draft defines training as modifying a generative model’s learned parameters. Real-time retrieval to answer a question (grounding) sits outside it, which is why Microsoft built a separate grounding control.
• Non-generative models are excluded. Training classifiers, ranking or scoring systems is out of scope.
• Deals override it. A private licensing agreement can supersede what your robots.txt says.
• It isn’t a standard yet. This is an individual Internet-Draft with no formal IETF endorsement. The current version expires on May 3, 2027, unless it is updated or adopted.

What it means for developers and site owners

If you run a site, a docs portal or a SaaS marketing page, this is a short to-do list:

1. Decide your policy first. Blocking training is not the same as blocking AI visibility. Many businesses want their docs and product pages used by models, because that is how they get recommended. If that’s you, do nothing, or add an explicit Allow.
2. If you want out, add the token now. It costs nothing, Anthropic is already implementing it, and the others will read it as they switch on. Keep your existing vendor-specific rules as well, since the shared token supplements them.
3. Bing users have a deadline. If you rely on NoArchive or NoCache to keep content out of Microsoft’s AI features, those tags stop counting as AI controls after March 1, 2027. Microsoft is opening a temporary transition option in Bing Webmaster Tools on December 15, 2026.
4. If you build crawlers, plan to respect it. Teams scraping the web to fine-tune or pre-train generative models should parse the token. With the biggest labs on board, ignoring it will be hard to defend to customers or regulators.

This fits a broader shift we’ve tracked in how sites get found in AI search: visibility, training and grounding are becoming three separate decisions instead of one.

What to watch next

Three things will show whether this sticks. First, whether the IETF’s AI preferences working group adopts the draft, turning a vendor agreement into a real standard. Second, whether other model builders, including Meta, Apple, Amazon and the major open-weight labs, sign on. Third, how Google’s mapping to Google-Extended plays out in practice, since that token touches Gemini grounding as well as training.

For now, the message to site owners is simple: one line in robots.txt is about to carry more weight than it ever has. Decide what you want it to say before the 2027 rollouts begin.