Short answer: sometimes. Website owners can often restrict a provider’s use of content for model training while keeping separate routes open for search or live grounding. The controls differ by platform, and a robots.txt preference does not guarantee a mention or citation in an AI answer.
On October 30, 2026, OpenAI, Google and Microsoft announced support for a proposed shared robots.txt signal: __usage_ai-training. It gives publishers a common way to express a preference about training use. It does not replace provider-specific crawler rules, and it does not settle the separate question of whether a page can be retrieved to answer a user’s query.
That distinction matters for any company trying to protect its content without accidentally removing itself from AI-powered discovery.
What the shared AI-training signal does
The proposal uses an ordinary robots.txt group with a purpose token in the User-agent field. For example:
User-agent: __usage_ai-training
Disallow: /A site can use standard path rules to express exceptions. The proposed token describes a preference about using content to train generative AI models. It is not itself a command that prevents a crawler from requesting a page. Crawler access still depends on the rules for the provider’s actual crawler or product token, along with other access controls.
The signal is described in an individual IETF Internet-Draft. It is work in progress, not a finalized or IETF-endorsed standard. The draft says existing parsers that do not recognize the token will ignore it. A wildcard group such as User-agent: * does not count as expressing this specific preference under the draft; the purpose token must be named.
OpenAI calls its support a voluntary effort and says the new signal will not replace its current crawler controls. Its announced timing is “in the coming months,” with implementation details to follow. That means the token should not be treated as a universal control that is already active across providers.
Training and grounding are different uses
Training uses content to modify a model’s learned parameters for future responses. Grounding retrieves or supplies content when a system is generating a response, so the answer can draw on information beyond what the model learned during training. Crawling, indexing, ranking and displaying a citation are separate steps again.
| Activity | What it means | What a preference can affect |
|---|---|---|
| Crawling | A system requests a page or file. | Whether a crawler is permitted to fetch it under the relevant access rules. |
| Indexing | A search system stores and organizes information. | Whether a page is eligible to appear in a search index. |
| Training | Content contributes to the learned parameters of a generative model. | Potential use in future model training, where a provider recognizes and honors the preference. |
| Grounding | Content is retrieved or supplied to help answer a current query. | Whether content may inform some generated answers. |
| Selection and citation | A system chooses sources and may show links or attribution. | No training preference guarantees selection, a citation, traffic or payment. |
In practice, a site can express a training preference and remain eligible for some kinds of search or grounding. Whether that combination is available depends on the provider’s controls.
How the announced controls differ
| Provider | Training control | Grounding or search implications | Announced timing |
|---|---|---|---|
| OpenAI | Plans to honor the shared signal. Existing GPTBot controls remain in place. | OAI-SearchBot is a separate control for surfacing websites in ChatGPT search. Blocking GPTBot does not, by itself, block OAI-SearchBot. | “Coming months”; no specific date announced. |
| Plans to support the shared signal in early 2027. | Google-Extended currently covers both Gemini model training and grounding in specified Gemini products. Google says a shared training opt-out without an explicit Google-Extended rule will be treated as a Google-Extended disallow. That can affect Gemini grounding as well as training. It does not affect ordinary Google Search inclusion or ranking. | Early 2027. | |
| Microsoft Bing | Plans to honor the shared signal, with training preferences also available at page level. | Separate page-level grounding choices: Allow; DisallowExtended, which limits use to Microsoft-operated experiences; and Disallow. Microsoft says these preferences do not change eligibility for traditional Bing results. | Preferences start taking effect as Microsoft crawls pages from February 1, 2027. |
OpenAI’s own crawler documentation distinguishes GPTBot, used for potential model training, from OAI-SearchBot, used to surface websites in ChatGPT search. Google’s announcement explains how its planned shared signal interacts with Google-Extended; the Google-Extended documentation describes that control’s scope across training and grounding. Microsoft’s publisher announcement explains the separate training and grounding choices and their rollout.
The IETF draft lists an Anthropic affiliation for its author. That fact alone is not a company adoption commitment. Publishers should look for an official product or crawler-policy announcement before assuming any provider honors the shared token.
Does blocking AI training remove a site from AI answers?
Not universally. OpenAI currently documents separate controls for training and ChatGPT search. Microsoft has announced independent training and grounding preferences. Google is the important exception to watch: Google-Extended currently controls both Gemini training and certain grounding uses. Under Google’s announced mapping, a shared training opt-out without an explicit Google-Extended rule may therefore reduce eligibility for Gemini grounding.
Google Search also has controls for its AI Overviews and AI Mode that are separate from Google-Extended. Google’s Search documentation says site owners can manage inclusion in those features through Search Console. These product-specific controls should not be confused with ordinary Google Search ranking or the shared training preference.
What website owners should do now
- Decide which use you want to restrict. Training, live grounding, general search discovery and user-triggered retrieval are not interchangeable choices.
- Inventory current rules. Review robots.txt and page-level controls for GPTBot, OAI-SearchBot, Google-Extended and Bingbot. Include any other provider controls relevant to your audience.
- Map each rule to a product. Record whether it affects training, search, grounding or more than one of these. Do not copy a single policy across providers without checking its consequences.
- Keep a dated baseline. Save the current robots.txt file and page-level directives. A public declaration is evidence of what the site asked for, not proof that every provider complied.
- Recheck rollout and recrawling. OpenAI has not given a specific launch date. Google targets early 2027. Microsoft says its new preferences begin taking effect as it crawls from February 1, 2027.
- Check the Microsoft transition if you use NoArchive or NoCache. Microsoft says temporary Bing Webmaster Tools grounding preferences open December 15, 2026. Sites relying on those older controls to restrict grounding need to follow Microsoft’s transition steps by March 1, 2027.
For Google, blocking Google-Extended does not remove a site from ordinary Google Search, but it can limit the specified Gemini training and grounding uses. For Microsoft, allowing grounding makes content eligible to inform responses; Microsoft explicitly says it does not guarantee answer inclusion, links, traffic or payment. For every provider, an allowed crawler is a prerequisite in some workflows, not a promise that a page will be selected or cited.
How to make the decision safely
Before changing a production robots.txt file, write down the desired outcome for each provider: “exclude from future training,” “remain eligible for live grounding,” “remain discoverable in search,” or “restrict both training and grounding.” Then verify that the provider’s documented controls actually support that combination.
This is the central point: an AI-training opt-out is not automatically an AI-search opt-out, but the two can overlap on some platforms. A careful review should test each rule against its actual product scope and rollout status, then revisit the settings after providers recrawl the affected pages.
Ranknizer’s AI Search Visibility Audit examines how a brand appears across defined prompts and AI search experiences. For the broader measurement context, see the guide to AI Search Visibility metrics and KPIs. The right technical choice starts with the result a publisher wants, not a blanket assumption that training controls determine citation eligibility.
Prepared by Muhammad Sukhera, AI Search Visibility Consultant at Ranknizer. Last verified: October 31, 2026. Provider plans and implementation dates can change; check the linked official documentation before changing site controls.
Primary sources
- IETF: Machine-Readable Training Preference Signal (Internet-Draft)
- OpenAI: Making it easier for websites to express AI training preferences
- OpenAI: Overview of crawlers
- Google: Supporting a shared AI training control for websites
- Google: Google-Extended and common crawlers
- Google Search Central: Control the content you share on Search
- Microsoft Bing: Your content, your choice: clearer preferences for training and grounding