Why it matters
A wrong rule can hide a whole site from search, so it is worth checking after every change. It guides crawlers but does not keep a page private.
Example
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlWhere you will see it
The free Robots.txt tester looks at this on a live page and explains what it finds in plain English: see which crawlers can fetch a URL, spot syntax mistakes and blocked CSS or JavaScript.
Related terms
- noindex: A directive, set in a robots meta tag or an X-Robots-Tag HTTP header, that asks search engines not to include a page in their results.
- XML sitemap: A file that lists the URLs you want search engines to know about, optionally with last-modified dates.
- Crawler: A program that fetches web pages automatically.
Go deeper
- robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.