1. Home
  2. Free tools
  3. Robots.txt tester

Robots.txt tester and validator

Paste any page address. We read the site's robots.txt, check it for mistakes and tell you which search crawlers may fetch that page, and which rule decided.

No sign-up.Try:

How to use it

  1. Enter the address of a page, for example https://example.com/blog/. The robots.txt of that site is fetched automatically.
  2. Read the verdict and the fix list. High items stop pages being crawled, low items are tidy-ups.
  3. Check the crawler table to see which bot is blocked and by which line, then edit your robots.txt and test again.

What it checks

  • Whether Googlebot, Bingbot and other crawlers may fetch the page you typed
  • The exact rule and line that allowed or blocked it
  • Misspelled or unknown directives and rules before any User-agent line
  • Disallow: / on a live site, and rules that block CSS, JavaScript or images
  • Missing, relative or unreachable Sitemap lines
  • A robots.txt that returns an error or a web page instead of text

Limits

  • Reads the robots.txt at the site root and applies Google-style matching. Other crawlers can interpret rules slightly differently.
  • Only the first 500 KiB is read, the same limit Google applies.
  • Tests whether a crawler may fetch a URL, not whether it will be indexed, which depends on other signals.

Public pages only, one page at one point in time. See the methodology and terms.

Frequently asked questions

What does robots.txt actually do?

It asks crawlers not to fetch certain URLs. It is a request, not a lock: well-behaved crawlers such as Googlebot and Bingbot obey it, but it does not hide a page from people, and a blocked URL can still appear in search results if other sites link to it. To keep a page out of results, use a noindex tag and let crawlers fetch the page.

How does a crawler choose between Allow and Disallow?

It first picks the group that names it most specifically, falling back to User-agent: *. Within that group the longest matching rule wins, and if an Allow and a Disallow match with the same length, Allow wins. This tool applies the same rules.

What happens if robots.txt is missing or broken?

A 404 means no restrictions: everything may be crawled. A server error (5xx) is treated by Google as everything blocked for a while, so a broken robots.txt can make a whole site disappear from crawling. That is why this tool flags error responses as high priority.

Should I block CSS and JavaScript?

No. Google renders pages like a browser. If it cannot load your styles and scripts it may misjudge the layout, mobile friendliness and content.

Does Crawl-delay work?

Google ignores it. Bing and some other crawlers honour it. Google's crawl rate adapts to how fast your server responds.

Where do I check the rest of the domain, such as DNS or SSL?

That belongs to CheckDomainHealth, our sister site. This tool focuses on how crawlers read one site's robots.txt.

↑ ↓ to moveEnter to openEsc to closeAll results