Major AI companies back shared standard for publishers to opt out of scraping

What's Happening

• OpenAI, Google, Microsoft, and Anthropic plan to adopt a new standard that lets publishers opt out of AI scraping across multiple platforms at once.
• Anthropic technical lead Kevin Kelley proposed the shared policy token to the Internet Engineering Task Force to live within a website's robots.txt file.
• Microsoft will offer granular controls distinguishing between AI training and AI grounding, while keeping opted-out sites in Bing search results.

Why It Matters

Website owners manage opt-outs for every AI crawler one by one, creating an administrative burden. A unified standard simplifies this process and gives publishers practical control over how their content feeds generative AI models.

The Context

• Publishers face a continuous influx of AI crawlers scraping their content to train models.
• Previous attempts to manage unauthorized scraping include the Really Simple Licensing standard and Cloudflare blocking AI crawlers by default.

What They're Saying

• Meta communications head Andy Stone called the common signal an important step but stopped short of committing to honor the token.

What to Watch

• Monitor whether Meta and other AI developers implement the standard across the industry to respect publisher preferences.

The Bottom Line

A unified opt-out standard marks a rare moment of coordination among top AI labs to address publisher concerns over data scraping.