All free tools

Which AI crawlers can read your site?

Your robots.txt might be blocking ChatGPT, Claude or Perplexity without you knowing.

Website

Enter your site URL to check AI crawler access

Why it matters

AI assistants like ChatGPT, Claude and Perplexity use crawlers to read your site. If they are blocked in your robots.txt, your site stays invisible in their answers.

This test reads your robots.txt and reports, crawler by crawler, which ones are allowed or blocked.

Enter a URL to check which AI crawlers can access your site

Are you actually cited by AI?

Allowing crawlers is just step one. Run a free audit to see if ChatGPT and Gemini actually recommend your site.

Run the free AI audit

AI crawler FAQ

What is an AI crawler?

An AI crawler is a bot that AI assistants use to read web pages, such as OpenAI GPTBot, Anthropic ClaudeBot, PerplexityBot and Google-Extended. If a crawler can access your site, the AI behind it can read and cite your content.

How do I check if my robots.txt blocks AI crawlers?

Paste your domain into this checker. It reads your robots.txt and shows, crawler by crawler, which AI bots are allowed or blocked, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

Should I allow or block AI crawlers?

If you want to appear in ChatGPT, Claude, Perplexity and Google AI Overviews, allow their crawlers. Blocking them keeps your content out of AI answers. Block only content you do not want used by AI.

Does blocking GPTBot remove me from ChatGPT?

Blocking GPTBot stops OpenAI from crawling your site, which lowers your chances of being cited in ChatGPT. To stay visible, allow GPTBot, OAI-SearchBot and ChatGPT-User.

How do I allow AI crawlers in robots.txt?

Add a User-agent rule for each AI crawler (or User-agent: *) without a Disallow on your key pages, for example User-agent: GPTBot then Allow: /. Re-run this checker to confirm.

Book a personalized demo

See how Hikoo can boost your visibility on AI search engines.