robots.txt AI crawlers, which tokens to allow and which to block

Allow the crawlers that feed answers citing your site, and make a deliberate call on the ones that feed training. In practice, let OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and bingbot through if you want visibility. Then decide, content by content, about GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.

That split is written into the official documentation, and it holds from one vendor to the next. OpenAI states on its bots page that each setting is independent of the others, so a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training its foundation models. Google describes the same decoupling from the other side, since Google-Extended governs whether your content trains Gemini without affecting your presence in Search. The useful question is therefore never whether to allow or block AI, but which use you accept, from which operator, and at what cost to your visibility.

One point matters before you write a single line. Blocking does not erase the past, and content already crawled and already absorbed into a model stays in that model. A robots.txt file expresses a preference about future visits, nothing more.

• If you want visibility, allow the whole search family and let person-initiated fetches through.
• If you want editorial protection, block the training family token by token while keeping the search family open.
• For confidential content, never rely on robots.txt, a public file that advertises the very paths you want hidden. Use authentication.
• In every case, check the live result on your own domain instead of trusting the file you believe you published.

OpenAI, overview of crawlers | Google, common crawlers

Test your file. AI crawler access checker | AI crawler simulator

Four families of robots share what people now call AI traffic, and mixing them up causes most of the bad decisions we see during audits. A single company usually runs one token per family, each with its own robots.txt behaviour.

The agent family is the newest one. Google now publishes a dedicated IP range file for user-triggered agents, next to the file covering classic crawlers, and Meta describes a fetcher that helps its AI navigate websites to complete tasks for users. Those clients behave more like a visitor than like a search engine, which makes a robots.txt block ineffective and a firewall block risky.

Google, user-triggered fetchers | Meta, web crawlers

Go further. Prepare your website for AI agents

The table below lists tokens exactly as they appear in their owner documentation, together with the real consequence of a block. Recommendations assume a site that wants commercial visibility.

OpenAI, bots and user agents | Anthropic, web crawling | Perplexity, crawlers

At Google the nuance is structural. AI Overviews and AI Mode are served from Search, and the documentation states that AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot are the control site owners have. In other words, no token pulls you out of Google AI surfaces while keeping you in the blue links. Google-Extended, for its part, affects neither your inclusion in Search nor your ranking, it only decides whether your content can train Gemini and ground Gemini Apps. Apple and Microsoft reason differently, with a two-layer control. At Apple, Applebot crawls for Spotlight, Siri and Safari, while Applebot-Extended exists only to opt out of training. At Microsoft, the NOCACHE and NOARCHIVE tags limit conversational use of the content without removing it from the Bing index.

• Microsoft does not rely on robots.txt alone. The NOCACHE and NOARCHIVE tags described on the Bing Webmaster Blog limit how content is used in conversational answers while keeping the site in search results.
• Apple applies the same two-layer logic. Applebot-Extended removes your content from training, and the nosnippet tag removes it from broad world knowledge answers, without pulling you out of Spotlight, Siri and Safari.
• A token missing from a vendor documentation sometimes circulates in ready-made lists. We leave it out here until an official source confirms it.

Google, AI features and your website | Apple, about Applebot | Bing Webmaster Blog, content controls

Blocking produces three very different effects depending on the family you target, and that grid is the only thing that prevents decisions people regret six months later. Blocking a search robot removes you from answers where you could have been cited, immediately and visibly, and OpenAI even notes that its systems need roughly twenty-four hours to reflect a robots.txt change. Blocking a training robot changes nothing about your visibility overnight, it is a decision about the future use of your content, and it does not undo what models already learned. Blocking a person-initiated fetcher breaks an explicit request, since someone pasted your link into a conversation and expects an answer while your server returns an error. That third case is the most expensive one and the least anticipated.

A fourth effect is often forgotten, the signal you send to everyone else. A robots.txt file is public, and it gets read, archived and compared. Closing a whole family while your competitors stay open hands them the cited-source slot, with no compensation for you until a licensing contract exists.

• Blocking GPTBot does not remove you from ChatGPT search, that surface is governed by OAI-SearchBot.
• Blocking Google-Extended removes you from neither AI Overviews nor AI Mode, which depend on Googlebot and on Search.
• Blocking Googlebot removes you from Search and therefore, mechanically, from Google AI surfaces.
• Blocking a training robot does not erase what has already been learned from your content.
• A badly written block can cancel other rules, because a robot only applies the most specific group that names it.

Related guides. Appear in AI Overviews | Get cited by Perplexity

The Robots Exclusion Protocol has been standardised since September 2022 by RFC 9309, a Standards Track document that ended twenty-five years of informal convention. The standard covers user-agent lines, allow and disallow rules, and the special characters used for path matching. It also requires a parsing limit of at least 500 kibibytes and a cache that does not exceed twenty-four hours.

Three rules explain almost every broken file we come across. A robot applies exactly one group, the most specific one naming it, so declaring a group for GPTBot cancels the star group rules for that robot. A file only covers one scheme, host and port combination, so every subdomain needs its own file, and Anthropic states explicitly that you must repeat the exclusion for every subdomain you want to opt out. Finally robots.txt governs crawling, not indexing or display, which are controlled through robots meta tags.

• Serve the file at the domain root, in plain text, with a text/plain content type.
• Write a star group holding your general rules and your Sitemap line.
• Add a per-token group only when you want different treatment, and copy into it the general rules that token still needs.
• Repeat the exercise on every subdomain, including technical subdomains that serve public content.
• Read the file again after deployment, on the public URL rather than in your repository.

RFC 9309, Robots Exclusion Protocol | Google, how robots.txt is interpreted

A robots.txt file returning an error does not behave like an empty file, and the gap between those two situations can stop crawling on your whole site. RFC 9309 separates an unavailable file from an unreachable one, with opposite consequences.

• A 500 error on robots.txt during a maintenance window is enough to suspend crawling across the site.
• A custom HTML page returned with a 200 status instead of the expected file gets parsed as a rules file, with unpredictable results.
• Redirecting robots.txt to a homepage is the surest way to make your rules invisible.

RFC 9309 | Google, how robots.txt is interpreted

The Crawl-delay directive is not part of RFC 9309, it remains a vendor specific extension. Google states plainly that its parser supports user-agent, allow, disallow and sitemap, and that other fields such as crawl-delay are not supported. Anthropic goes the other way and documents support for that non-standard extension to limit crawling activity, with an example setting one second between requests for ClaudeBot.

The practical consequence is simple. If your server struggles with request volume, Crawl-delay will not save it, you need server side rate limiting, a properly formed 429 response, and if necessary a firewall rule targeting the IP ranges published by the operator involved.

Google, supported fields | Anthropic, Crawl-delay

A user-agent string is declarative, and any script can announce itself as GPTBot or Googlebot. Any serious blocking or allowing decision therefore has to rest on the network origin of the request rather than on its label. Major operators publish IP range lists in JSON exactly for that purpose, maintained on their side, and your firewall or CDN can consume them directly. Google adds a two-step verification, a reverse DNS lookup on the accessing address that must resolve to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup on that name that must return the original address. Apple documents the same method with the applebot.apple.com domain. This double check is the only reliable way to tell an official robot from a scraper borrowing its name, and it also protects your AI traffic statistics from a large amount of noise.

Google, verify crawler requests | Apple, identifying Applebot

robots.txt expresses a preference, it blocks nothing technically. For a binding block, the decision belongs to the web application firewall or the CDN, based on verified IP ranges rather than on the announced name. That is also the layer where rate limiting and 429 responses belong.

This side of the ecosystem moves fast. On 1 July 2025 Cloudflare announced that it was changing the default to block AI crawlers unless they pay creators for their content, then on 24 September 2025 it published a Content Signals Policy that adds a usage preference line to robots.txt. Three signals exist, search for building an index and returning results, ai-input for feeding a model at answer time, and ai-train for training. Cloudflare applies Content-Signal search=yes, ai-train=no by default on the robots.txt files it manages, and deliberately leaves ai-input unset.

In parallel, the IETF chartered an AI Preferences working group in January 2026 to standardise a preference vocabulary and the way it attaches to content, with a Content-Usage mechanism in robots.txt and in HTTP headers, updating RFC 9309. Those documents are still drafts as this guide is published, so no operator is bound by them today.

• A usage signal expresses a legal intent, it does not replace a technical block.
• Check what your CDN writes on your behalf, a managed robots.txt can contain lines you never wrote.
• If you license content, keep a record of when each rule changed, because a retroactive block cannot be proven.

Cloudflare, Content Signals Policy | Cloudflare, blocking by default | IETF, AI Preferences working group

The file you think you published and the file your server returns are two different things, especially behind a CDN, a reverse proxy or a framework that generates robots.txt on the fly. The test below was run on 14 August 2026 against tryhikoo.com with our public checker.

The result deserves a careful read. Some tokens show up as not listed while still being allowed, which means they are not named in the file and simply inherit the star group. That is normal behaviour, and it turns into a trap as soon as the star group carries disallow rules, since an unlisted token then inherits every one of them.

• Open your robots.txt in a browser and check both the response code and the content type returned.
• Run the domain and its subdomains through the AI crawler checker to get a token by token status.
• Repeat the test after each deployment, since a CDN cache can serve an older version for hours.
• Then check how a page actually renders with the AI crawler simulator, because allowed access does not guarantee readable content.
• Finish with your server logs, verifying IP addresses rather than announced names.

Free tools. AI crawler checker | AI crawler simulator | llms.txt generator

No single configuration fits everyone, and recommending that you allow every robot would be as lazy as blocking them all. Here is how we reason depending on the business model behind the site.

• No robots.txt configuration guarantees a citation, it only governs access.
• Accessible but generic content will not be picked up, access is necessary and never sufficient.
• If you block a family, document the decision and its date, you will need both in a negotiation.

Measure what comes next. Analyzer | Spotlight | Free AI audit

• Copying a token list from a blog post without checking vendor documentation, since half the lists in circulation still carry retired tokens.
• Declaring a group for an AI token and forgetting to copy the star group disallow rules into it, which accidentally opens sensitive areas.
• Believing a Disallow protects content, when it publicly advertises it to anyone reading the file.
• Forgetting subdomains, since every host needs its own file.
• Blocking the training family thinking it protects traffic, when traffic depends on the search family.
• Letting a CDN rewrite the file without reading what production actually serves.
• Never revisiting the file, while operators add new tokens several times a year.

Useful definitions. llms.txt | AI visibility | AI citations

The real question is not whether to allow or block artificial intelligence, it is which use you accept, operator by operator, knowing the price of each refusal. The search family governs your presence in answers, the training family is an editorial strategy call, and person-initiated fetches escape robots.txt anyway.

Start by checking what your domain actually returns, token by token, then measure what that access produces in citations. Our AI crawler checker gives the first answer in seconds, and the free audit tells you whether assistants already cite your site.