What the new AI scraping opt-out means for your content

OpenAI, Google, Microsoft, and Anthropic say they will honor a single shared signal that lets a website opt out of AI scraping by all of them at once, according to The Verge. The signal is called a policy token. It lives in a site’s robots.txt file, and Kevin Kelley, a technical lead for standards at Anthropic, proposed it through the Internet Engineering Task Force. One fact decides what this means for you, and it is about timing. Per The Verge, only Anthropic has adopted the token so far. Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the “coming months.”

The background you need

The report assumes you already know how websites talk to AI companies. Here is the short version.

Crawlers. A crawler is an automated program that visits web pages and copies their content. Search engines use crawlers to build their index. Archives use them to save old pages. AI companies use them to collect text for their models.

robots.txt. This is a plain text file at the root of a website. It tells crawlers which parts of the site they may visit. The Verge notes it has long been used to signal access for search indexing, archiving, and AI training. Think of it as a sign on your office door listing who may come in. The key point is that it is an honor system. Nothing technically stops a crawler that ignores the sign. So when large companies publicly commit to obeying a rule, that commitment is most of what gives the rule its force.

How opting out worked until now. According to The Verge, sites currently have to signal their AI training preferences separately to each AI company. In practice, each company’s crawler goes by its own name, and a site has to write a rule for each name. Picture a do-not-call list where you must register separately with every telemarketer, and new telemarketers keep appearing. The shared token is meant to work like one list that every participating company checks.

The analogy breaks in one place. A do-not-call list is backed by regulators. This token is backed only by the companies’ own promises.

Training versus grounding. These are two different uses of your content. Training means your text helps build the model’s general knowledge. Grounding, as The Verge describes it, means pointing an AI model to a verified source of information to reduce hallucinations. A hallucination is a confident answer that is wrong. In short, training shapes what the model knows, and grounding is the model looking something up when it answers.

The IETF. The Internet Engineering Task Force is an open standards body that writes many of the technical rules the internet runs on. A proposal there is a draft standard. It is not a law, and it does not bind anyone who has not agreed to it.

Why this is happening now. The Verge says publishers are dealing with a continuing influx of AI crawlers feeding their content into AI models. Other responses already exist. The Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default. The pressure shows up in other places too. TechCrunch reports that Reddit is moving to a mainly logged-in experience, partly to protect its data from AI scraping and to support licensing deals with model providers. Reddit said its search referrals dropped by more than 1 million per week on average in October.

What actually changed

The new part is narrow but useful. There is now one token, written once, that several major AI companies say they will respect. The Verge reports that OpenAI, Google, Microsoft, and Anthropic all say they will restrict their AI crawlers from accessing a site when they encounter it.

Microsoft is adding more detail. It says it will offer granular controls, so publishers can opt out of AI training or AI grounding separately. Microsoft also says sites will still appear in Bing search results whatever they choose.

Meta is in a different position. Its communications head, Andy Stone, signaled support and called a common signal “an important step.” According to The Verge, though, he did not say whether Meta will honor it.

Several things were already true before this announcement:

• robots.txt already existed and was already used for AI opt-outs, one company at a time.
• Google had already introduced similar controls. Per The Verge, they let sites decide whether their content appears in AI search features or is used for training, without affecting search visibility.
• Compliance was voluntary before, and it still is.

The main thing missing is timing. Of the four companies, only Anthropic’s support is live today.

What it means for PMs, analysts, and knowledge workers

Someone at your company has to make this call. This applies if your team publishes anything public: a marketing site, product docs, a help center, research reports, a blog. Whether to opt out of AI training is now a business decision, not just a technical setting. You probably do not edit robots.txt yourself. You can ask who does, and whether anyone has decided your company’s AI policy on purpose.

Do not tear out what works. If you add only the shared token today, it covers Anthropic, according to The Verge. Google and Microsoft plan to support it next year, and OpenAI in the coming months. Until then, existing per-company rules still matter. The safe approach is to add the token alongside your current rules, not in place of them.

Opting out of training does not have to mean vanishing from search. Microsoft says Bing results are unaffected, and Google already separates AI features from search visibility. That removes one common fear. The grounding choice is less simple. If you opt out of grounding, your pages presumably will not be used as a source when an AI tool answers a question. With AI summaries already reducing click-throughs, as TechCrunch reports, being cited may be one of the few ways left to get noticed. Weigh both sides before you block everything.

For analysts who rely on AI tools. As more sites opt out, AI assistants could have less access to some publishers’ content. When an answer feels thin or leans on secondary sources, go to the original publisher directly.

What to watch. Watch for the Google and Microsoft rollouts, OpenAI’s implementation, any firm commitment from Meta, and whether the IETF proposal becomes a finished standard.

What we don’t know yet

• The exact syntax. The reports do not show what the token looks like, or whether it covers training only or also grounding and AI search features across all four companies.
• Meta’s decision. Meta supports the idea in public but has not committed to honoring it.
• Everyone else. The sources do not say whether smaller AI companies or lesser-known crawlers will adopt it.
• Content already collected. Nothing in the reports says whether opting out affects material that was gathered before a site added the token.
• Verification. There is no word on how a publisher could confirm that a company is actually complying.
• How it fits with paid licensing. It is unclear how the token will interact with approaches like RSL, where publishers charge crawlers instead of blocking them.