Anthropic, Google, Microsoft and OpenAI back AI training opt-out token for publishers

Anthropic, Google, Microsoft and OpenAI have agreed to support a new policy token that allows publishers to opt out of having their website content used to train AI models. The proposed standard, submitted to the Internet Engineering Task Force (IETF), lets publishers set permissions in their robots.txt file for AI training access in a single step. While the companies have announced support, the standard is not yet official and is still under IETF review.

The policy token aims to simplify how publishers express preferences but does not guarantee compliance. Data from ScalePost shows that only 0.15% of crawler requests from the main AI firms checked robots.txt files, raising doubts about practical enforcement. Critics, including publisher groups, argue that the move is more symbolic than effective, as robots.txt is not legally binding and many crawlers ignore it.

For regulated industries, the debate reflects ongoing concerns about data governance, intellectual property and the limits of voluntary standards in AI. The lack of enforceability may leave banks, insurers and healthcare firms exposed if their data is scraped without consent. Monitoring the IETF process and future regulatory responses will be important for risk management.