OpenAI, Google, Microsoft, and Anthropic announced support for a new standard that lets publishers opt out of AI scraping across multiple companies at once. The "policy token" signal, proposed by Kevin Kelley at Anthropic as part of the Internet Engineering Task Force, will live in a website's robots.txt file. Publishers can currently only signal AI training preferences to each company separately.
The shared token lets publishers apply their preferences across all supporting AI crawlers instead of setting them individually. Anthropic has already adopted the signal. Google and Microsoft plan to support it next year, while OpenAI says it will implement it in the "coming months."
Microsoft will offer granular controls allowing publishers to opt out of AI training or AI grounding, the process of pointing an AI model to verified sources to reduce hallucinations. Microsoft noted that websites will still appear in Bing search results regardless of their AI preferences. Google previously implemented similar controls that let sites manage whether their content appears in AI search features or is used for AI training without impacting search visibility.
Meta communications head Andy Stone signaled support for the new setting but did not confirm whether Meta will honor it. "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step," Stone said. "And we look forward to working with other AI companies on the standards and their implementation across the industry."
The announcement comes as publishers deal with increasing AI crawler activity scraping their content. Some have used the Really Simple Licensing standard to charge crawlers, while others like Cloudflare have blocked AI crawlers by default.