Last checked . Picks are based on the prices, rates, fees and terms each provider publishes, checked at the source. We never rank products by commission. See how we’re funded.
A robots.txt AI training opt-out now exists in draft form, and Google, Microsoft, OpenAI, and Anthropic have said they will honor it. The Next Web reported on Oct. 31, 2026, that the four companies announced support for a single robots.txt token, written as __usage_ai-training, which lets a website state that its content should not be used to train generative AI models. Before this, a site owner needed a separate rule for each company’s crawler. This guide covers the exact rule, how to add it, when each company says it will start honoring it, and what it does not do.
What the robots.txt AI training opt-out is
The Next Web reports that Anthropic’s Kevin Kelley wrote the proposal as an Internet-Draft for the Internet Engineering Task Force (IETF) and announced it on the group’s AI preferences mailing list. It is an individual submission. The IETF has not endorsed it, and it has no formal standing in the standards process. How2Shout, in its guide to the rule, adds that the draft expires in May 2027 unless it is updated, so its future as a standard is uncertain.
The token expresses a preference about how content is used for training. It does not control who can crawl a site. According to How2Shout, Googlebot and Bingbot indexing is unaffected, and robots.txt remains a request that each company chooses to respect.
How to add the robots.txt AI training opt-out
How2Shout gives the syntax from the draft. The group goes in the robots.txt file at the root of the domain, and existing rules do not need to change:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example blocks the whole site from AI training except the press kit. The same article lists variations in the table below.
| Goal | Rule inside the group |
|---|---|
| Opt the whole site out | Disallow: / |
| Opt out except one section | Add an Allow: line, such as Allow: /press/ |
| Opt one section out | Disallow: /premium/ |
After saving the file, How2Shout advises checking it at the domain’s /robots.txt address. A wildcard group does not work for this purpose. The Next Web notes that a site without the new group is treated as having expressed no preference, and existing rules for all crawlers do not count, so a User-Agent: * block is not an opt-out.
When each company says it will honor the robots.txt AI training opt-out
The companies are on different schedules, according to The Next Web and How2Shout. The table below combines the two reports.
| Company | Reported timing | Existing controls that still apply |
|---|---|---|
| Anthropic | Work has started; no date given | Its existing crawler controls |
| OpenAI | “In the coming months” | GPTBot and its other controls |
| 2027, with its documentation saying early 2027 | Google-Extended | |
| Microsoft | Honors the preference from Feb. 1, 2027 | NoArchive and NoCache tags, which stop being read after March 1, 2027 |
Google has said that a site using the shared token without naming Google-Extended will be treated as having blocked Google-Extended. How2Shout also reports that Microsoft is opening a Bing Webmaster Tools setting on Dec. 15, 2026, to keep grounding blocked in the interim. Grounding is the use of a live page in an AI answer, and Microsoft treats it as a separate page-level setting.
What the robots.txt AI training opt-out does not do
The reports list several limits on the robots.txt AI training opt-out:
- It does not establish licensing or payment. Microsoft says the preferences “do not establish licensing or payment arrangements,” per The Next Web.
- It can be overridden by private agreements between a site and an AI company.
- It shapes future use only, according to How2Shout. Neither report says it removes content from models that have already been trained.
- It has no stated enforcement mechanism. Compliance depends on each company’s voluntary implementation.
- Blocking Microsoft’s grounding setting does not change a page’s place in Bing search results, though Microsoft says it may reduce visibility in AI-generated experiences.
How Microsoft’s page-level tags fit with the robots.txt AI training opt-out
The Next Web reports that for Microsoft, the new preferences replace the NoArchive and NoCache tags that the company has used since 2023, and they add a separate page-level setting for grounding. How2Shout names the page-level tags as ai-training and ai-grounding, and it says they can override robots.txt. A site that uses both a robots.txt group and page tags would therefore follow the more specific page tag for Microsoft. The reports do not say whether the other companies plan page-level equivalents.
Choosing a robots.txt AI training opt-out rule by site type
The reports do not recommend a rule for any type of site, so the table below is the Jolt Journal’s own mapping of the three documented variations to common situations.
| Situation | Documented rule that fits |
|---|---|
| A site that wants no training use at all | Disallow: / |
| A site that wants its press or media pages available | Disallow: / plus an Allow: line for that folder |
| A site with a paid or members-only section | Disallow: for that section only |
In every case the rule belongs in the robots.txt file at the root of the domain, and the file should be reloaded in a browser afterward to confirm that the new group appears. Because compliance with the robots.txt AI training opt-out is voluntary and timing differs by company, a rule added today may have no effect on some systems until their stated start dates in 2027.
Which rules to keep alongside the robots.txt AI training opt-out
How2Shout recommends adding the shared token now and keeping the GPTBot, Google-Extended, and Bing rules until each company confirms that its support is live. Company-specific rules can also be more specific than the shared token, which allows an exception for one company. The Jolt Journal’s reading of that advice is that the robots.txt AI training opt-out is an addition and not a replacement for now.
The Jolt Journal has followed the training-data questions behind this change. Its report on OpenAI’s plan to stop training on the live internet covers one company’s stated approach, and its report on the Canadian publishers’ lawsuit against Anthropic covers a dispute over training content.
Editor’s take
The robots.txt AI training opt-out is cheap to add and limited in effect. Adding three lines costs a site owner nothing, and it signals a preference to four major developers at once. But it is a draft with no formal standing, three of the four companies have not started honoring it, and the reports describe no way to verify compliance. Site owners who care about the question should add the token, keep the older crawler rules, and recheck each company’s documentation in early 2027.
This is general information, not legal advice. The IETF draft and each company’s crawler documentation have the final word on syntax and support.
