Abstract:
Google, Microsoft, OpenAI, and Anthropic are supporting a new AI training preference statement that allows websites to indicate whether to allow content to be used by AI companies to train models by modifying robots.txt. Under the draft, website administrators would only need to add a set of rules to make a statement to multiple AI companies supporting the rules.
The statement reads as follows:
# Site-wide ban rules
User
-
Agent
:
__usage_ai
-
training
Disallow
:
/
# Allow specific directory example
User
-
Agent
:
__usage_ai
-
training
Allow
:
/archive/
User
-
Agent
:
__usage_ai
-
training
disallow
:
/archive/
# Allowing specific directories does not conflict with the site-wide ban, that is, /archive/ is allowed but all other directories are prohibited
Add this content to the robots.txt file of the website to declare that the entire site content is prohibited from being used for training AI models. The website administrator can also modify the allowed or disallowed directories. For example, some directories are allowed to be used for AI training, and some content is not allowed to be used for AI training.
The support hours of each company are different:
The plan is still an Internet Draft, but Microsoft has said that the company will comply with the statement starting on February 1, 2027, so websites that do not want their content to be used for AI training can now modify the file to add the relevant statement.
Google’s plan is to provide support starting in early 2027, but the specific time has yet to be completed. OpenAI says it will implement the plan in the coming months, and Anthropic appears to have begun supporting the draft.
Prohibiting training and quoting content to generate answers belong to different purposes:
The above statement only expresses the website's choice of content usage. According to the draft instructions, setting the above statement alone will not directly prevent crawlers from accessing the web page. If the website wants to restrict specific crawlers from crawling content, it still needs to be configured through traditional robots.txt rules.
Training content and AI answer reference web pages also have different purposes. Microsoft provides additional AI Grounding controls for its own services, allowing websites to decide whether the content is allowed to be referenced when AI-generated answers are used. This control is for specific pages and only applies to Microsoft services.
Google stated that if a website uses the above statement to reject AI training but does not explicitly set Google-Extended, Google plans to handle it according to the website's rejection of Google-Extended rules. Google-Extended involves Gemini model training, and also involves the web reference use of some Gemini answers.
Comments