Back to Blog
Demokan Özkök
9 dk
SEO

AI Crawler Controls: What robots.txt, Google-Extended, nosnippet and llms.txt Really Do

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended do different jobs. What does blocking each one change? A sourced guide from the companies' own documentation.

GEOrobots.txtllms.txtAI SearchTechnical SEO
Türkçe Oku

"Should we block AI bots?" is usually asked as a single decision. It is at least four, each made with a different token and a different consequence. Blocking the wrong line can switch off visibility you wanted to keep, and the line you thought mattered may change nothing.

Every statement in this guide comes from the company's own documentation, checked on 11 November 2026. These documents change, so re-read the linked sources before acting.

First, three different jobs: training, search index, user request

AI companies run bots that do three different jobs, and their documentation names them separately:

  • Training crawlers collect content that may be used to train models. Blocking them signals that you do not want your content used for training.
  • Search crawlers build the index that lets an AI search product show and link your site. Block them and you are absent from that product.
  • User-triggered fetchers visit a page at the moment a user asks a question. Some companies state plainly that these may not be bound by robots.txt.

Which bot does what

CompanyTokenUsed forWhat to know
GoogleGooglebotGoogle Search, including AI Overviews and AI ModeThis token is the control for Search features
GoogleGoogle-ExtendedTraining of Gemini models, and grounding in Gemini Apps and Vertex AIDoes not affect inclusion or ranking in Search; has no separate HTTP user agent
OpenAIOAI-SearchBotSurfacing websites in ChatGPT's search featuresIf opted out, the site is not shown in ChatGPT search answers; changes take about 24 hours
OpenAIGPTBotCrawling content that may be used to train generative AI foundation modelsDisallowing it signals the content should not be used in training
OpenAIChatGPT-UserUser-initiated actions in ChatGPT and Custom GPTsrobots.txt rules may not apply; it does not determine appearance in search
AnthropicClaudeBotCollecting web content for model trainingHonors robots.txt; blocking keeps future material out of training datasets
AnthropicClaude-SearchBotNavigating the web to improve search result qualityHonors robots.txt; blocking may reduce visibility in search results
AnthropicClaude-UserFetching pages in response to a user's questionHonors robots.txt; blocking prevents retrieval for a user query
PerplexityPerplexityBotSurfacing and linking websites in Perplexity search resultsNot used to crawl content for AI foundation models
PerplexityPerplexity-UserVisiting a page because a user askedGenerally ignores robots.txt rules, since a user requested the fetch

Sources: Google AI features, Google crawlers, OpenAI crawlers, Anthropic crawlers, Perplexity crawlers.

Four common misconceptions

1. "Blocking Google-Extended removes me from AI Overviews." It does not. Google's documentation says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. The token manages whether content may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. For AI features inside Search, Google says the control is robots.txt directives for Googlebot.

2. "You need llms.txt to appear in AI Overviews." Google's documentation says the opposite: there are no additional requirements to appear in AI Overviews or AI Mode, no special schema.org structured data is needed, and you do not need to create new machine-readable files or AI text files. The only technical condition is that the page is indexed and eligible to be shown in Google Search with a snippet.

llms.txt itself is a proposal, not a standard. llmstxt.org describes it as a proposal to standardise on using an /llms.txt file to provide information that helps agents use a website. Jeremy Howard published it on 3 September 2024. The file is written in Markdown and its only required section is an H1 with the name of the project or site. There is no harm in having one; just know that it is not a condition for appearing in Google's AI features.

3. "nosnippet switches off only the AI part." It switches off more than that. According to Google's robots meta tag documentation, nosnippet means no text snippet is shown for the page in search results; this applies to all forms of search results, including AI Overviews and AI Mode, and it also prevents the content from being used as a direct input for those features. Your ordinary result loses its description text too. max-snippet limits how much of the content may be used as a direct input.

4. "If it is in robots.txt, no AI will fetch my page." For user-triggered fetchers that may not hold. OpenAI writes that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. Perplexity writes that Perplexity-User generally ignores robots.txt rules because a user requested the fetch. Anthropic states that its bots honor robots.txt and that it supports the non-standard Crawl-delay extension.

Bing and Copilot

Microsoft offers this control through meta tags instead of robots.txt. According to Bing's announcement of 22 September 2023, content with no tag may be included in chat answers and may be used to train generative AI foundation models. Content tagged NOCACHE may be included in answers, but only URLs, titles and snippets may be used in training. Content tagged NOARCHIVE is not included in answers and is not used for training. With either tag the content still appears in Bing search results.

That announcement dates from the Bing Chat period; the product was later renamed Copilot. Check the current Bing Webmaster Guidelines for the latest wording.

Decision table

You wantDo thisCost
To appear in AI search but stay out of trainingAllow OAI-SearchBot, Claude-SearchBot, PerplexityBot; disallow GPTBot, ClaudeBot, Google-ExtendedGoogle Search is unaffected
To keep your text out of Google's AI summariesnosnippet or max-snippetThe description in your ordinary result shrinks or disappears as well
To stay out of Bing chat answers (per the 2023 announcement)NOARCHIVE meta tagYou still appear in Bing search results
To appear everywhereBlock nothingContent may be used for training

What we found in our own file

While writing this guide we compared our own robots.txt with the documentation. Two things came up. The file had a line for anthropic-ai; that name does not appear in Anthropic's current documentation, which describes ClaudeBot, Claude-User and Claude-SearchBot. We also had no separate lines for the OpenAI and Anthropic search crawlers; the general allow rule already covered them, but we could not have separated search from training if we ever wanted to. We updated the file: removed that line and added the search and user bots by name.

Checklist

  1. Open your robots.txt and match every AI line against the table above. Remove names that the documentation no longer lists.
  2. Decide training and search separately. Do not close both with one line.
  3. Allow for the roughly 24 hours OpenAI says a change takes to apply on the search side.
  4. If you have subdomains, check each one's own robots.txt. Anthropic notes that blocking must be done for every subdomain.
  5. Remember that none of these controls guarantees visibility. Allowing a crawler does not mean you will be cited, and being cited does not mean you will be clicked. We measured that on our own site: our 60-query AI Overview measurement.

Sources

All accessed on 11 November 2026.

Need a broader answer hub?

Browse the English FAQ page for quick answers on digital marketing, Google Ads, SEO, pricing and project scope before you compare services.

Visit English FAQ
Demokan Özkök

Founder & Strategic Director

Demokan Özkök, founder of Avangard Reklam, turns search data into measurable growth strategy for regional and international brands.

Want to Improve Your Digital Marketing Strategy?

Describe your project and we send scope, price and timeline in writing within 1 business day.

Written Quote (WhatsApp)