"Should we block AI bots?" is usually asked as a single decision. It is at least four, each made with a different token and a different consequence. Blocking the wrong line can switch off visibility you wanted to keep, and the line you thought mattered may change nothing.
Every statement in this guide comes from the company's own documentation, checked on 11 November 2026. These documents change, so re-read the linked sources before acting.
First, three different jobs: training, search index, user request
AI companies run bots that do three different jobs, and their documentation names them separately:
- Training crawlers collect content that may be used to train models. Blocking them signals that you do not want your content used for training.
- Search crawlers build the index that lets an AI search product show and link your site. Block them and you are absent from that product.
- User-triggered fetchers visit a page at the moment a user asks a question. Some companies state plainly that these may not be bound by robots.txt.
Which bot does what
| Company | Token | Used for | What to know |
|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | This token is the control for Search features | |
Google-Extended | Training of Gemini models, and grounding in Gemini Apps and Vertex AI | Does not affect inclusion or ranking in Search; has no separate HTTP user agent | |
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT's search features | If opted out, the site is not shown in ChatGPT search answers; changes take about 24 hours |
| OpenAI | GPTBot | Crawling content that may be used to train generative AI foundation models | Disallowing it signals the content should not be used in training |
| OpenAI | ChatGPT-User | User-initiated actions in ChatGPT and Custom GPTs | robots.txt rules may not apply; it does not determine appearance in search |
| Anthropic | ClaudeBot | Collecting web content for model training | Honors robots.txt; blocking keeps future material out of training datasets |
| Anthropic | Claude-SearchBot | Navigating the web to improve search result quality | Honors robots.txt; blocking may reduce visibility in search results |
| Anthropic | Claude-User | Fetching pages in response to a user's question | Honors robots.txt; blocking prevents retrieval for a user query |
| Perplexity | PerplexityBot | Surfacing and linking websites in Perplexity search results | Not used to crawl content for AI foundation models |
| Perplexity | Perplexity-User | Visiting a page because a user asked | Generally ignores robots.txt rules, since a user requested the fetch |
Sources: Google AI features, Google crawlers, OpenAI crawlers, Anthropic crawlers, Perplexity crawlers.
Four common misconceptions
1. "Blocking Google-Extended removes me from AI Overviews." It does not. Google's documentation says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. The token manages whether content may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. For AI features inside Search, Google says the control is robots.txt directives for Googlebot.
2. "You need llms.txt to appear in AI Overviews." Google's documentation says the opposite: there are no additional requirements to appear in AI Overviews or AI Mode, no special schema.org structured data is needed, and you do not need to create new machine-readable files or AI text files. The only technical condition is that the page is indexed and eligible to be shown in Google Search with a snippet.
llms.txt itself is a proposal, not a standard. llmstxt.org describes it as a proposal to standardise on using an /llms.txt file to provide information that helps agents use a website. Jeremy Howard published it on 3 September 2024. The file is written in Markdown and its only required section is an H1 with the name of the project or site. There is no harm in having one; just know that it is not a condition for appearing in Google's AI features.
3. "nosnippet switches off only the AI part." It switches off more than that. According to Google's robots meta tag documentation, nosnippet means no text snippet is shown for the page in search results; this applies to all forms of search results, including AI Overviews and AI Mode, and it also prevents the content from being used as a direct input for those features. Your ordinary result loses its description text too. max-snippet limits how much of the content may be used as a direct input.
4. "If it is in robots.txt, no AI will fetch my page." For user-triggered fetchers that may not hold. OpenAI writes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity writes that Perplexity-User generally ignores robots.txt rules because a user requested the fetch. Anthropic states that its bots honor robots.txt and that it supports the non-standard Crawl-delay extension.
Bing and Copilot
Microsoft offers this control through meta tags instead of robots.txt. According to Bing's announcement of 22 September 2023, content with no tag may be included in chat answers and may be used to train generative AI foundation models. Content tagged NOCACHE may be included in answers, but only URLs, titles and snippets may be used in training. Content tagged NOARCHIVE is not included in answers and is not used for training. With either tag the content still appears in Bing search results.
That announcement dates from the Bing Chat period; the product was later renamed Copilot. Check the current Bing Webmaster Guidelines for the latest wording.
Decision table
| You want | Do this | Cost |
|---|---|---|
| To appear in AI search but stay out of training | Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot; disallow GPTBot, ClaudeBot, Google-Extended | Google Search is unaffected |
| To keep your text out of Google's AI summaries | nosnippet or max-snippet | The description in your ordinary result shrinks or disappears as well |
| To stay out of Bing chat answers (per the 2023 announcement) | NOARCHIVE meta tag | You still appear in Bing search results |
| To appear everywhere | Block nothing | Content may be used for training |
What we found in our own file
While writing this guide we compared our own robots.txt with the documentation. Two things came up. The file had a line for anthropic-ai; that name does not appear in Anthropic's current documentation, which describes ClaudeBot, Claude-User and Claude-SearchBot. We also had no separate lines for the OpenAI and Anthropic search crawlers; the general allow rule already covered them, but we could not have separated search from training if we ever wanted to. We updated the file: removed that line and added the search and user bots by name.
Checklist
- Open your robots.txt and match every AI line against the table above. Remove names that the documentation no longer lists.
- Decide training and search separately. Do not close both with one line.
- Allow for the roughly 24 hours OpenAI says a change takes to apply on the search side.
- If you have subdomains, check each one's own robots.txt. Anthropic notes that blocking must be done for every subdomain.
- Remember that none of these controls guarantees visibility. Allowing a crawler does not mean you will be cited, and being cited does not mean you will be clicked. We measured that on our own site: our 60-query AI Overview measurement.
Sources
All accessed on 11 November 2026.
- Google Search Central, AI features and your website.
- Google Search Central, Google's common crawlers (Google-Extended).
- Google Search Central, Robots meta tag specifications (nosnippet, max-snippet).
- OpenAI, Overview of OpenAI crawlers.
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?.
- Perplexity, Perplexity crawlers.
- llmstxt.org, The /llms.txt file.
- Microsoft Bing Blogs, Announcing new options for webmasters to control usage of their content in Bing Chat (22 September 2023).

