AI training opt-out token ignores 'User-agent: *', so blanket blocks express no preference

A new signal for the robots exclusion file, meant for opting out of AI training does not treat the catch-all "User-agent: *" rule as an opt-out, according to the Internet-Draft that defines it. Published on October 30, 2026, the draft says wildcard groups must not be read as matching its token, `__usage_ai-training`, so a site has to name the token to state a preference.

That detail matters because the announcement was framed as one simple switch for publishers. The switch exists, but a site that blocks every bot with a wildcard has said nothing about training under this scheme.

What the draft actually asks sites to write

The mechanism borrows the format those files already use. A site adds a group headed by the token, followed by an ordinary Disallow or Allow rule for the whole site or for specific paths. Disallow signals a preference against AI training, and Allow signals a preference in favor.

The draft is explicit that this is about use, not access. It calls the token "not a directive for crawling behavior" and tells operators who want to limit crawling to keep using rules aimed at named crawlers. A wildcard block can still keep compliant crawlers out, but under this draft it does not count as a statement about training.

It also leaves room for exceptions. Private agreements may override a preference expressed through the token, the text says.

A signal with no formal standing yet

The document was written by Kevin Kelley of Anthropic and filed as an individual draft. The IETF's tracker describes that status as not endorsed by the IETF and without formal standing, and this version expires on May 3, 2027.

Support is arriving unevenly. Microsoft says its new preferences start taking effect on February 1, 2027, as pages are recrawled, so publishing a setting does not activate it right away. Google's crawler documentation says it intends to support the token in early 2027. A report carried by Tech Xplore says OpenAI promised a rollout "in the coming months" and that Anthropic has "already started".

Silence still means yes

Microsoft's guide states the default plainly: when no applicable preference is published, both training and grounding are allowed. Grounding is the use of content as input when an AI answer is created, and Microsoft offers no site-wide file rule for it. It is set per page with a meta tag or header instead.

Microsoft separates the costs of those two choices. Opting out of training leaves a site eligible for traditional Bing results, while restricting grounding "will significantly reduce your content's visibility in AI-generated experiences," the guide warns. It also says none of these settings create licensing or payment arrangements.

There is a deadline for sites relying on older tags too. After March 1, 2027, Microsoft will stop reading NoArchive and NoCache as training or grounding preferences. The draft itself does not explicitly address models trained before a preference was published, which remains the open question for anyone whose pages were crawled years ago.