🔍 Read the full analysis: How To Find And Set The Shared Opt-Out For AI Training on ThorstenMeyerAI.com
TL;DR
OpenAI, Google, Microsoft and Anthropic have each said they will honor a shared robots.txt signal expressing a publisher’s preference not to have its content used to train generative AI models. The signal is voluntary, does not block crawling or AI answers, and will take effect on different schedules; Meta has voiced support but has not committed to honor it.
OpenAI, Google, Microsoft and Anthropic say they will honor a shared instruction in websites’ robots.txt files asking that content not be used to train generative AI models. The proposal, published as an Internet-Draft on October 30, gives publishers one signal to use with participating companies instead of maintaining a separate training opt-out for each crawler. It is a voluntary preference, not a technical block, and each company has set or described a different implementation timeline.
To express the preference, a publisher can add this rule group to its robots.txt file:
User-Agent: __usage_ai-training
Disallow: /
The group uses a shared token, __usage_ai-training, rather than naming each company’s crawler. The proposal appears in draft-kelley-aipref-training-preference-signal-00, titled “Machine-Readable Training Preference Signal.” It was written by Kevin Kelley of Anthropic and published through the Internet Engineering Task Force’s AI Preferences working group. The document is an individual Internet-Draft, not an adopted IETF standard.
The companies describe different rollout plans. Anthropic says it has started implementing support, while noting that extending it across its services will take time. OpenAI says it aims to implement the signal “in the coming months.” Microsoft says it will honor the training preference from February 1, 2027, as it crawls sites. Google says it aims to support the control in 2027; its crawler documentation describes the target as early 2027.
Microsoft also announced a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says both preferences take effect February 1, 2027. Its announcement says NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027. Publishers using those directives may need to add the newer controls before then.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
One Rule, Separate AI Uses
The shared signal could reduce the work publishers face when stating that their material should not be used for model training. A common rule is easier to maintain than separate instructions for GPTBot, Google-Extended and ClaudeBot, but it does not automatically reach companies that have not agreed to honor it. Meta has expressed support for publishers’ ability to state preferences, but has not publicly committed to follow this signal. The status of other AI services is also unconfirmed in the cited reporting.
Publishers should also distinguish training from other uses. The shared token is described as a preference against training; it does not, by itself, express a separate choice about content being retrieved or used to produce AI-generated answers. Microsoft’s announcement provides a distinct grounding control. Search indexing is another separate question: Microsoft says its preferences do not change eligibility for Bing search results, and Google says Google-Extended does not affect Google Search.
The distinction matters because adding the shared rule should not be treated as a universal “do not use” switch. Nor does an opt-out instruction establish a right to payment. Microsoft says neither choice guarantees a publisher inclusion, traffic or payment. The proposal records a publisher’s stated preference; its practical effect depends on each company’s implementation and compliance.
Popular right now
Bose QuietComfort Ultra Wireless Noise-Cancelling Headphones with Spatial Audio, Black
TCL QM64L 85-Inch QD-Mini LED 4K Smart Fire TV
De'Longhi 15-Bar Pump Espresso Machine with Milk Frother & Temperature Control
As an affiliate, we earn on qualifying purchases.
Before this proposal, publishers commonly used company-specific controls. A site could name GPTBot for OpenAI or Google-Extended for Google, for example, and use Anthropic’s crawler rules for Claude. Microsoft’s earlier NoArchive and NoCache tags served different purposes from a robots.txt crawler rule. The new proposal offers a shared label for one defined use—training—without replacing all those controls immediately.
The draft sits within work at the IETF AI Preferences working group, which considers ways to communicate preferences about automated use of web content. The working group’s process has not yet settled the proposal. At its October 21 meeting, one of the chairs said, “We haven’t declared consensus on anything yet.” The existence of company announcements does not make the draft an official internet standard.
Publishers’ practical approach is to decide separately about training, AI answers and search, then keep existing company-specific rules while services roll out the shared one. The source article also discloses that the draft was written by an Anthropic standards lead and that the article’s publisher is part of a network of publisher websites. Those interests do not establish whether the signal will be effective, so the companies’ published commitments and the draft’s status remain distinct points to track.
„“has already started implementing support for this draft”“
— Anthropic, describing its implementation
Adoption and Enforcement Remain Open
The signal is a request, not a lock. Robots.txt communicates a site’s preference; the material provided does not establish a technical mechanism that prevents a company from crawling or using content despite that preference. The draft itself has no formal standing in the IETF standards process, and working-group consensus has not been declared.
It is also unclear exactly when each service will apply the rule across all products and crawlers. Anthropic says implementation is underway but may take time; OpenAI has given a broad “coming months” target; and Google and Microsoft point to 2027 dates. The source article reports that Meta voiced support without saying it would honor the instruction, and it found no commitment in the reviewed sources from Apple, Amazon, xAI or Perplexity. Those absences do not establish what those companies may decide later.
Finally, the shared training preference does not remove material from models that have already been trained, and it does not cover every use of web content. Publishers should not assume it governs live retrieval for AI answers, search eligibility or payment. The companies’ operational details and any later revisions to the draft remain subject to change.
Watch Rollouts Through 2027
Publishers can add the shared rule now if they want to state a preference against AI training, while retaining company-specific crawler directives until participating services have implemented the new token. Sites using Microsoft’s NoArchive or NoCache tags should review the company’s separate grounding and training controls ahead of its stated March 1, 2027, change in how those older directives count.
The next milestones are the companies’ actual rollouts: Anthropic’s broader implementation, OpenAI’s planned deployment in the coming months, and Google’s and Microsoft’s 2027 schedules. Publishers will also need to watch for whether Meta and other AI providers make explicit commitments. Separately, the IETF working group may continue discussing the draft; its publication alone does not show that the group has adopted it as a standard.
For now, the practical step is to identify which uses a site wants to restrict—training, AI answer grounding, or search-related crawling—and configure the relevant controls separately. The shared line simplifies one preference for companies that honor it, but how much protection it provides will become clearer only as implementation and standards discussions progress.
Key Questions
What should a website add to opt out of AI training?
The proposed robots.txt group is User-Agent: __usage_ai-training followed by Disallow: /. It expresses a preference against using the site’s content for generative AI training by companies that honor the signal.
Does the shared rule block AI companies from crawling a site?
No. The source describes it as a preference, not a technical lock. It also does not remove content from models that were trained before the rule was applied.
Does the rule stop content from appearing in AI answers?
Not by itself. The shared signal is for training. Microsoft announced a separate AI Grounding preference for content used to inform answers in Bing, Copilot and Microsoft 365.
When will the companies honor the signal?
Anthropic says implementation has begun; OpenAI says it aims to implement the signal in the coming months. Microsoft’s date is February 1, 2027, and Google’s target is 2027, described in its crawler documentation as early 2027.
Has Meta committed to honor the shared opt-out?
No commitment is reported in the source material. Meta’s communications chief expressed support for publishers stating preferences and called a common signal an important step, but did not say Meta would honor it.
Source: ThorstenMeyerAI.com
