OpenAI, Google, Microsoft, Anthropic Back Unified Opt-Out Standard for AI Scraping

OpenAI, Google, Microsoft, and Anthropic announced today that they will support a common technical signal letting publishers express AI‑training preferences once, instead of configuring separate rules for each company. The “policy token” lives in a site’s robots.txt file and is being proposed through the Internet Engineering Task Force (IETF) by Anthropic standards lead Kevin Kelley [6].

A Shared Signal in robots.txt

Until now, publishers that wanted to block or limit AI training crawlers had to identify each provider’s user‑agent string and maintain distinct directives for every one. The new signal aims to let a site declare its preference—allow, disallow, or restrict—and have all participating crawlers honor it automatically [6][7].

OpenAI said the change “reduces that work for operators” and will not replace its existing per‑crawler controls, which will continue to function [7]. Microsoft added that it will offer granular options to opt out of AI training or AI grounding (the practice of pointing a model to a verified source to reduce hallucinations) while keeping the site visible in Bing search results [6]. Google noted it already provides similar controls for AI search features versus model training [6].

Staggered Timelines, Voluntary Adoption

The four companies are not moving in lockstep. Anthropic says it has already adopted the signal. Google and Microsoft plan to support it “next year.” OpenAI says it will implement the token in the “coming months” and will share details later [6]. All commitments are voluntary; the announcements do not mention any regulatory or contractual enforcement.

The IETF working group on AI Preferences is still developing the standard. OpenAI described today’s announcement as strengthening its “continued participation in the IETF standards discussions” and said it will “adapt its practices as those conversations and the landscape continue to evolve” [7].

Meta Signals Support Without Commitment

Meta communications head Andy Stone said the company believes “creating a common signal that conveys such preferences to multiple providers at once is an important step” and that it looks forward to “working with other AI companies on the standards and their implementation across the industry.” He did not say whether Meta’s crawlers will honor the token [6].

Context: Publisher Pressure and Existing Alternatives

The announcement comes as publishers contend with a surge of AI crawlers. Some have turned to the Really Simple Licensing (RSL) standard, which lets sites charge crawlers for access, while Cloudflare now blocks known AI crawlers by default [6].

What Remains Unclear

• Whether the IETF will finalize the policy‑token specification and on what timeline.
• How many smaller AI companies, search indexes, or academic crawlers will adopt the signal.
• Whether “voluntary” adherence will hold if competitive pressures shift.
• How the standard interacts with existing copyright law and the growing body of lawsuits over training data.

The four labs frame the move as a practical step toward a simpler web. Publishers get a single lever; crawler operators get a standardized signal. Whether it becomes a durable norm or a temporary gesture depends on adoption beyond the initial signatories and on whether the IETF process produces a stable, widely implemented RFC.