Check your robots.txt file.
Fetch any site's robots.txt, see the rules for every crawler, and catch the mistakes that quietly hurt crawling: syntax errors, a blanket Disallow, or a missing sitemap line.
Free · No signup required
About this tool
How Robots.txt Checker works.
What it is
robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. It controls crawling, not indexing.
What we flag
Missing files, pages returned instead of a text file, unknown directives, rules before any user-agent, no Sitemap line, and a Disallow: / that blocks everyone.
Read the rules
Each user-agent group is shown with its Allow and Disallow rules, so you can see exactly what each crawler is told.
Frequently asked questions.
Common questions about the robots.txt checker.
At the root of the host, for example https://example.com/robots.txt. Subdomains need their own file.
Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. Use a noindex tag on a crawlable page to keep it out of search.
Yes. A Sitemap: line helps every crawler find it without guessing.
No. It's free and needs no account.