Add as preferred source

New AI Training Control in robots.txt: What Website Owners Need to Know

Neon teal and cyan illustration explaining a proposed shared AI training preference in robots.txt, with a policy token connected to AI provider symbols and publisher controls.

Website owners have always needed ways to control how search engines and automated bots access their content. Now, as artificial intelligence becomes a bigger part of the internet, content publishers are getting another important option: the ability to express their preferences about AI model training.

On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt. The proposed mechanism aims to let website owners communicate their AI training preferences to multiple participating AI providers through a single policy token.

This development could make content management easier for publishers, bloggers, businesses and digital marketing professionals.

However, there is an important detail: Google plans to introduce support in 2027. The shared control is not yet ready for general implementation.

1. What Is the New AI Training Control in robots.txt?

The new AI training control is a proposed shared mechanism that allows website owners to express whether their website content can be used to train generative AI models.

Currently, website owners may need to configure separate robots.txt instructions for different AI providers.

  • Google uses Google-Extended.
  • OpenAI has GPTBot.
  • Anthropic has ClaudeBot.
  • Other AI providers have their own crawler controls.

Managing individual instructions can become difficult, especially for businesses managing multiple websites.

Google’s new proposal aims to simplify this process through a shared policy token. Instead of maintaining separate training preferences for every participating provider, publishers could express a common preference through one robots.txt group.

2. What Did Google Announce on October 30, 2026?

Google Search Advocate John Mueller announced the proposed shared AI training control through the official Google Search Central Blog.

The announcement explained that website owners would be able to communicate their AI training preferences across multiple participating providers.

The proposal builds on Google’s existing Google-Extended control, which has been available since 2023.

Google also clarified three important points:

  1. Website owners will be able to express broad AI training preferences using a shared policy token.
  2. Existing company-specific controls will continue to work.
  3. Google aims to support the new mechanism alongside other AI providers in 2027.

The proposal is based on work from the IETF AIPREF working group.

Read the original announcement: Google Search Central: Supporting a Shared AI Training Control for Websites.

3. How Does the New robots.txt Policy Token Work?

The proposed mechanism introduces a shared policy token:

User-Agent: __usage_ai-training
Disallow: /

This expresses a preference against using the website’s content for AI training across the paths covered by the rule.

Website owners could also create path-specific preferences. For example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

In this example, the website expresses a preference against AI training for the entire website, while the /press-kit/ directory is explicitly allowed for AI training.

The proposed mechanism uses existing robots.txt group syntax rather than introducing entirely new directive names.

Important: These examples explain the IETF proposal. They should not be treated as a live, universally supported instruction. Google has announced a 2027 implementation target, and the IETF document remains an Internet-Draft.

4. Which AI Companies Are Supporting This Initiative?

The announcement represents an important move towards shared AI content preferences. Several major AI companies have discussed supporting the proposed mechanism.

Company Announced position
Google Plans to support the shared signal in 2027
OpenAI Plans to honor the signal in the coming months
Microsoft Announced support, with implementation planned from February 1, 2027
Anthropic An employee authored the IETF proposal; company implementation details should be checked separately

The timeline and implementation commitments are not identical across providers. The initiative is intended to reduce the need for publishers to manage separate AI training preferences for every company.

5. What Is the Difference Between Google-Extended and the New Shared Control?

Google already provides a mechanism called Google-Extended. It allows website owners to control whether content crawled by Google can be used for certain generative AI purposes.

The new shared policy token has a broader objective.

Feature Google-Extended Shared AI training control
Purpose Google-specific AI usage preferences Shared AI training preferences
Coverage Google services covered by its documentation Participating AI providers
Configuration Google-specific robots.txt group Shared policy-token group
Availability Existing control Planned provider implementations
Main benefit Control over Google’s covered AI uses One common preference across participating providers

Google has also explained that if a website opts out using the shared token and does not explicitly specify Google-Extended, Google plans to interpret that preference as a Google-Extended disallow.

Existing company-specific instructions will continue to function. This means publishers can combine a broad preference with individual provider-specific controls.

6. Will Blocking AI Training Affect Google Search Rankings?

This is one of the most important questions for SEO professionals.

Website owners should understand the difference between AI training, crawling and search visibility. These are separate activities.

  • AI training: Using website content to modify the learned parameters of a generative AI model.
  • Crawling: Automated access to website pages to retrieve their content.
  • Search indexing: Processing and storing information so that it can potentially appear in search results.
  • AI grounding: Retrieving or referencing website information when generating an AI response.

The proposed shared AI training signal is specifically designed to communicate AI training preferences. It is not a general crawling directive.

Google has also stated that Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

However, website owners should distinguish training preferences from other AI-related controls. For example, Google-Extended also affects eligibility for certain Gemini Apps grounding uses. Google’s documentation explains that websites explicitly allowing Google-Extended can be eligible to appear in Gemini responses and receive clicks through grounding.

Therefore, website owners should not assume that every AI-related control has the same effect.

7. What Should Website Owners Do Right Now?

Since the shared mechanism is still moving towards implementation, website owners should prepare rather than rush to modify their robots.txt files.

1. Review Your Existing robots.txt File

Check which automated crawlers currently have access to your website. Look for existing instructions related to Googlebot, Google-Extended, GPTBot, ClaudeBot, Bingbot and other AI-related crawlers. Understand what each rule does before making changes.

2. Understand Your AI Content Strategy

Decide how you want AI providers to use your published content. Consider whether you allow AI model training, want your content available for AI-powered search, want AI systems to retrieve it for answering user questions, or want different preferences for different website sections.

These decisions should be made separately because AI training and AI search visibility are not identical.

3. Monitor the IETF Standard

The shared policy token comes from the IETF AIPREF working group’s work. The relevant Internet-Draft is titled Machine-Readable Training Preference Signal. The latest revision available on November 22, 2026, is draft-01, updated November 16, 2026. It is still an Internet-Draft, not an approved final internet standard.

Monitor its development: IETF – Machine-Readable Training Preference Signal.

4. Keep Provider-Specific Controls in Place

Do not remove existing AI crawler instructions simply because a shared policy mechanism has been announced. Continue using the controls documented by individual providers until their shared-signal implementations are available and documented.

5. Prepare a Content Usage Policy

Businesses that publish original research, guides, proprietary information and educational content should establish clear internal preferences. This helps website owners make informed decisions about AI training and content distribution.

6. Review the Implementation Before Deployment

When providers begin supporting the shared token, review their official documentation. Test the robots.txt configuration and ensure that existing crawler instructions continue to work as expected. Avoid copying unverified configurations into a live website.

8. What Does This Mean for SEO and Digital Marketing?

This development introduces another important consideration for technical SEO. SEO professionals will increasingly need to understand how website content is discovered, crawled, indexed, retrieved and used by AI systems.

Content ownership and control

Publishers may have a simpler way to communicate AI training preferences across participating providers.

AI search visibility

Website owners will need to understand the difference between allowing content for model training and allowing it to be retrieved or referenced in AI-powered search experiences.

Technical SEO

robots.txt management will become more important as websites manage search crawlers, AI crawlers and machine-readable content usage preferences.

Content strategy

Businesses should review which content they want to make available for different AI uses, particularly original research, proprietary resources and premium information.

9. What Does This Mean for WordPress Website Owners?

WordPress powers millions of websites, including business websites, blogs and e-commerce stores.

Most WordPress websites already have a robots.txt configuration, either generated virtually by WordPress or managed through SEO plugins and server configuration.

Website owners should take the following steps:

  1. Check the current robots.txt file.
  2. Review existing Google-Extended and other AI crawler rules.
  3. Understand the difference between training and search-related controls.
  4. Follow official Google and AI provider implementation announcements.
  5. Update the configuration only when the shared signal is supported.

For websites using Rank Math, Yoast SEO or other SEO plugins, check how the plugin or hosting environment manages robots.txt before editing the file.

Do not add the proposed shared policy token and assume that all AI providers will immediately recognise it.

10. Final Thoughts

Google’s proposed shared AI training control is an important development for website owners and content publishers.

Instead of managing separate AI training preferences for every provider, publishers could eventually communicate their preferences through one shared policy token in robots.txt.

However, the new mechanism should not be confused with a general crawling restriction or an AI search visibility control.

Google aims to support the mechanism in 2027, while the IETF proposal continues to develop.

For now, website owners should review their existing robots.txt configuration, understand their AI content preferences and monitor official implementation updates.

The future of website management will involve more than controlling who can crawl your content. It will also involve making clear decisions about how your content can be used by AI systems.

Frequently Asked Questions

1. What is the new AI training control in robots.txt?

It is a proposed shared policy token that allows website owners to communicate AI model training preferences to participating AI providers through a robots.txt file.

2. What is the new robots.txt AI training token?

The proposed policy token is __usage_ai-training. It is designed to communicate whether website content may be used for generative AI model training.

3. When will Google’s shared AI training control be available?

Google announced on October 30, 2026, that it aims to support the shared control in 2027. No exact launch date was provided.

4. Does the new AI training control block Google from crawling my website?

No. The proposed policy token is not a crawling directive. Website owners must continue using crawler-specific robots.txt instructions to manage crawling.

5. Does blocking AI training affect Google rankings?

Google states that Google-Extended is not a Google Search ranking signal and does not affect inclusion in Google Search. The proposed shared token is also specifically about AI training preferences, not general search ranking instructions.

6. What is the difference between Google-Extended and the shared AI training control?

Google-Extended is a Google-specific control. The proposed shared policy token is designed to communicate AI training preferences across multiple participating providers.

7. Should I add the new token to my robots.txt file now?

Not as a live, universally supported instruction. Google has announced plans for 2027 implementation. Continue using currently documented provider-specific controls and wait for official implementation guidance.

8. Can I allow AI training for some pages and block it for others?

The IETF proposal supports path-specific preferences using Allow and Disallow rules within the policy-token group. Provider implementation and support will determine how these preferences are applied.

9. Is the new AI training control an official IETF standard?

No. As of November 22, 2026, the mechanism is documented in an active Internet-Draft. It has not become an approved IETF standard.

10. Why is this update important for SEO professionals?

It introduces a shared way to express AI training preferences. SEO professionals will need to understand the difference between AI training, crawling, indexing and AI search visibility when managing websites.

Related SEO Guides

Sources and Further Reading

  1. Google Search Central – Supporting a Shared AI Training Control for Websites
  2. IETF – Machine-Readable Training Preference Signal
  3. OpenAI – Making It Easier for Websites to Express AI Training Preferences
  4. Search Engine Land – Google Gives Publishers More Control Over How AI Uses Their Content
  5. Google – Google-Extended Documentation

Share this article

LinkedIn X WhatsApp Email
Anand Bajrangi

Written by

Anand Bajrangi

SEO consultant, author and co-founder of EvolvCode Solutions

Anand is a self-taught SEO consultant from Bihar, India. He started learning SEO in 2020, and he and his team have since helped more than 100 businesses get found on Google. He wrote two books on SEO, The Truth About SEO and SEO Words Made Simple, and builds free SEO tools, a WordPress SEO plugin and the SEO Inspector Chrome extension.

See more of Anand’s articles on Google

Choose anandbajrangi.com as a preferred source, and Google will show you more of its articles in Top Stories and in “From your sources”.

Add as preferred source

Want Anand to look at your site?

Request a free SEO audit. You get a plain-English list of what to fix first, by email.

Get free SEO audit