Major AI firms agree to single opt-out rule for web scraping

OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new standard that lets publishers stop AI companies from scraping their websites. The change allows site owners to apply their preferences across multiple AI providers using a single setting, rather than contacting each company individually.

The new mechanism uses a shared policy token placed within a website's robots.txt file. This text file has long been used to tell search engines and other bots which parts of a site they can access. Kevin Kelley, a technical lead for standards at Anthropic, proposed the signal through the Internet Engineering Task Force.

Currently, publishers must signal their AI training preferences separately to each major AI firm. The new standard streamlines this process by providing a common signal that multiple crawlers will recognize. If a crawler encounters the token, the AI company commits to restricting its access to that site.

Meta also weighed in on the development. Andy Stone, the company's communications head, said Meta believes creating a common signal is an important step. However, Stone did not explicitly confirm that Meta will honor the setting. "We look forward to working with other AI companies on the standards and their implementation across the industry," Stone said.

Microsoft plans to offer detailed controls under the new system. Publishers will be able to opt out of AI training or AI grounding, a process where models point to verified sources to reduce errors. Microsoft noted that websites will still appear in Bing search results even if they opt out of training. Google has already implemented similar controls that allow sites to manage their presence in AI search features without affecting standard search visibility.

Anthropic has already adopted the new signal. Google and Microsoft plan to support it next year. OpenAI says it will implement the standard in the coming months. The move follows recent efforts by other organizations to manage AI crawler traffic, including the Really Simple Licensing standard and Cloudflare's default blocking of AI bots.

The agreement comes as publishers deal with a growing number of AI crawlers collecting content for training models. The new standard aims to give site owners more direct control over how their content is used by large technology companies.