AI companies plan a shared robots.txt signal for training preferences

What happened?

Website publishers may get a common way to express a preference against generative AI training across several providers. The proposed robots.txt group is written exactly as “User-Agent: __usage_ai-training” followed on the next line by “Disallow: /”. The token expresses a use preference; it is not the name of a crawler and does not automatically prevent crawling, search indexing or use by a party that ignores the signal.

The mechanism appears in draft-kelley-aipref-training-preference-signal-00, an Internet-Draft rather than an adopted IETF standard. The draft uses existing robots.txt syntax, including path-specific rules. Implementation and interpretation may evolve as standards work continues.

Google says it aims to support the control in 2027. OpenAI says it aims to implement it in the coming months. Microsoft says it will honor the new training and separate grounding preferences beginning February 1, 2027 as it crawls. Anthropic’s Kevin Kelley wrote that Anthropic has started implementing support, with work across services still ahead. These announcements do not mean a universal block is operating today.

Existing provider-specific controls remain relevant. Google says its Google-Extended setting continues alongside the shared signal. Microsoft’s new grounding preference is separate from training; publishers using its older NoArchive or NoCache controls should review Microsoft’s transition instructions and deadlines instead of assuming the old settings carry over.

Source: IETF ↗ · Source: Google ↗ · Source: OpenAI ↗ · Source: Microsoft Bing ↗ · Source: Kevin Kelley, Anthropic, via IETF ↗

Why does it matter?

A shared preference could reduce the maintenance burden for publishers, but it works only where a provider implements and respects it. Operators should check each provider’s rollout and retain the appropriate existing controls during the transition.