ai training robots: Giants Back Shared Opt-Out Signal

A shared ai training robots preference is moving from working-group talk into vendor roadmaps. OpenAI, Google, Microsoft and Anthropic say they will honor a common robots.txt signal so publishers can opt out of generative AI training crawlers without maintaining a separate rule for every bot.

What actually happened

On October 30, 2026, the companies lined up behind work from the IETF AI Preferences (AIPREF) effort. The Verge reported that the shared policy token was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as an individual draft — not a finished internet standard.

Today's status quo forces site owners to list crawlers one by one: GPTBot here, Google-Extended there, plus whatever new UA string launches next quarter. The new signal is meant to collapse that into one preference that participating providers interpret as "do not use this content for foundational generative AI training."

OpenAI's own post says it will honor the common robots.txt signal, that existing crawler controls remain in force and that implementation details will come in a later update. Google's Search Central blog says that when a site opts out with the shared token and does not explicitly allow Google-Extended, Google plans to treat that like a Google-Extended disallow. Microsoft is pairing the work with more granular choices around training versus AI grounding. Anthropic is already on the signal, according to The Verge.

AIpresso's November 2 digest adds the useful boundary: the draft focus is generative training scrapers, while separate product controls still gate whether a page can be fetched to answer a live question.

What the signal is — and is not

The ai training robots preference does not replace search robots rules, licensing deals or lawsuits. It is a voluntary industry convention riding on robots.txt, the same fragile file the web has used for decades. Crawlers that ignore robots.txt will still ignore this token.

It also does not automatically block every AI-adjacent fetch. Grounding, retrieval for citations and non-generative tasks are being discussed as separate switches. Microsoft and OpenAI both emphasize that training opt-outs and live-answer fetches can be gated differently. Google notes that sites which explicitly allow Google-Extended can still be eligible for Gemini grounding experiences.

Because this is an IETF individual draft, the exact vocabulary can still change as the working group debates the rest of AIPREF. Publishers who adopt early should watch for syntax updates, not carve the first blog-post example into stone forever.

Why publishers care now

Unauthorized training scrapes have pushed sites toward default blocks, paid licensing experiments and bot fights at the CDN layer. Cloudflare's default AI-crawler blocking and Really Simple Licensing are part of that same pressure wave. A shared opt-out will not end those fights, but it lowers the clerical cost of saying no to the largest labs at once.

Policy context still matters. We have tracked broader governance noise around white house ai safety pact and market expectations in year end ai model odds. A robots token is narrower: it is an operational control, not a statute. Funding and product timelines for the labs, including coverage like openai funding target cut, will shape how quickly "coming months" and "2027" become real crawler behavior.

Meta's posture is the reminder that endorsement is not enforcement. Andy Stone called a common signal an important step; The Verge notes Meta did not say it will honor the token. Until a provider ships parser support and documents the UA behavior, treat promises as roadmaps.

One line in robots.txt only works against bots that choose to read it.

What to do as a reader

If you run a site, read the OpenAI note, Google's Search Central post and The Verge's roundup before you rewrite production robots.txt. Keep your existing per-bot rules until each provider documents the shared token in its crawler docs.

If you follow AI policy, watch for implementation changelogs from Anthropic, OpenAI, Google and Microsoft rather than the announcement headlines alone. The ai training robots story becomes real on the day logs stop showing those crawlers after an opt-out — not on the day the blog posts go live.

Nothing here is legal advice. robots.txt preferences are technical signals, not a substitute for contracts or regulation.