Robots.txt Checker
Check whether robots.txt allows crawlers to access a specific URL.
Free for up to 50 checks per day. No signup required.
Crawl access
Use the Robots.txt Checker before assuming a page has a content problem
robots.txt controls crawling, not ranking, but a bad rule can still keep search engines from seeing your page. This checker helps you confirm whether a URL path appears allowed or blocked before you chase more complicated indexing explanations.
Robots rules are easy to misread because user-agent groups, wildcard paths, and directory-level rules can interact. A single Disallow rule may block an entire section while the homepage and other pages look normal.
Recommended workflow
If robots.txt blocks the URL, update the rule and test again. Then request a fresh crawl in Bing or GSC. If robots.txt allows the URL, move on to noindex, canonical, HTTP status, and content quality checks.
What this check reviews
- robots.txt accessibility for the target site.
- Allow or Disallow behavior for the submitted URL path.
- HTTP status of the URL being checked.
- Other indexability signals that matter after crawl access is fixed.
Robots rules worth checking
- A staging Disallow rule was left in production.
- A folder-level rule blocks all blog, product, or docs pages.
- CSS or JavaScript resources are blocked and rendering suffers.
- The page is blocked in robots.txt but still appears in the sitemap.