OpenAI, Google, Microsoft and Anthropic back new opt out signal for AI scraping

OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI. With the control, publishers can apply their AI training preferences across multiple AI companies instead of setting them individually. The shared signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force.

How the signal works

The signal will live within a website's robots.txt file, which has long signaled which web crawlers can and cannot scrape a site for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company. The companies say they will restrict their AI crawlers from accessing a site if they encounter the shared token.

Company plans and controls

Anthropic has already adopted the signal, while Google and Microsoft plan to support it next year. OpenAI says it will implement it in the coming months. Microsoft says it will offer granular controls that let publishers opt out of AI training or AI grounding, which it describes as pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing's search results regardless of their preferences. Google previously implemented similar controls that allow sites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search.

Support and other efforts

Meta communications head Andy Stone also signaled support for the new setting, but did not say whether the company will honor it. Stone said creating a common signal that conveys such preferences to multiple providers at once is an important step, and that Meta looks forward to working with other AI companies on standards and their implementation across the industry. The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default.

Sources: The Verge · Techmeme