Which AI crawlers can access your site?
Enter your site URL to check AI crawler access.
Free, no account or credit card required
Sample reportExampleexample.com
7allowed
3blocked
ClaudeBot, Google-Extended, CCBot cannot read example.com.
| Bot | Access | ||
|---|---|---|---|
| GPTBotOpenAI | OpenAI | Training | Allowed |
| OAI-SearchBotOpenAI | OpenAI | ChatGPT search | Allowed |
| ChatGPT-UserOpenAI | OpenAI | On-demand browsing | Allowed |
| ClaudeBotAnthropic | Anthropic | Training | Blocked |
| Claude-UserAnthropic | Anthropic | On-demand browsing | Allowed |
| PerplexityBotPerplexity | Perplexity | Search | Allowed |
| Google-ExtendedGoogle | Gemini | Blocked | |
| Applebot-ExtendedApple | Apple | Apple Intelligence | Allowed |
| CCBotCommon Crawl | Common Crawl | Open corpus | Blocked |
| BingbotMicrosoft | Microsoft | Copilot, Bing | Allowed |
User-agent: *Allow: / User-agent: ClaudeBotDisallow: / User-agent: Google-ExtendedDisallow: / User-agent: CCBotDisallow: / Sitemap: https://example.com/sitemap.xmlIn red, the rules that block an AI bot.
Does AI cite your site?
Allowing crawlers is only the first step. Run a free audit to see whether ChatGPT and Gemini actually recommend your site.
Why does it matter?
AI assistants like ChatGPT, Claude and Perplexity use crawlers to read your site. If they are blocked in your robots.txt, your site stays invisible in their answers.
This test reads your robots.txt and shows, crawler by crawler, which ones are allowed or blocked.
Frequently asked questions about AI crawlers
Another question? Write to us
An AI crawler is a bot that AI assistants use to read web pages, such as OpenAI GPTBot, Anthropic ClaudeBot, PerplexityBot and Google-Extended. If a crawler can access your site, the AI behind it can read and cite your content.
Paste your domain into this checker. It reads your robots.txt and shows, crawler by crawler, which AI bots are allowed or blocked, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
If you want to appear in ChatGPT, Claude, Perplexity and Google AI Overviews, allow their crawlers. Blocking them keeps your content out of AI answers. Block only content you do not want used by AI.
Blocking GPTBot stops OpenAI from crawling your site, which lowers your chances of being cited in ChatGPT. To stay visible, allow GPTBot, OAI-SearchBot and ChatGPT-User.
Add a User-agent rule for each AI crawler (or User-agent: *) without a Disallow on your key pages, for example User-agent: GPTBot then Allow: /. Re-run this checker to confirm.