Can AI crawlers read your site?
Check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are allowed by your robots.txt, plus whether the page itself can be crawled and indexed: status code, redirects, noindex and canonical.
Free · No signup required
About this tool
How AI Crawlability Checker works.
Why it matters
AI assistants build answers from content their crawlers can reach. A single Disallow rule in robots.txt can quietly remove your brand from ChatGPT, Claude or Perplexity answers.
What we check
Two layers. First, robots.txt rules for each major AI crawler: the most specific user-agent group wins, then the longest matching rule. Second, the page itself: HTTP status, redirect chain, meta robots, X-Robots-Tag, noindex, nofollow and canonical.
What we can't see
This reads robots.txt only. A firewall, CDN bot-protection or login wall can still block crawlers even when robots.txt allows them.
Frequently asked questions.
Common questions about the ai crawlability checker.
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot and CCBot (Common Crawl).
If you want to be mentioned and cited in AI answers, yes. Some publishers block training crawlers like GPTBot while allowing search and browsing bots. It's a business decision, and this tool shows you what is currently allowed.
Edit your robots.txt and remove the Disallow rule for that user-agent, or add an Allow rule. Then re-run this check.
No. The tool is free and needs no account.