1. Home
  2. Glossary
  3. robots.txt
SEO

robots.txt

A plain-text file at the root of a site that tells crawlers which paths they may crawl. It controls crawling, not indexing, and it is not a security measure.

Why it matters

A wrong rule can hide a whole site from search, so it is worth checking after every change. It guides crawlers but does not keep a page private.

Example

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Where you will see it

The free Robots.txt tester looks at this on a live page and explains what it finds in plain English: see which crawlers can fetch a URL, spot syntax mistakes and blocked CSS or JavaScript.

Related terms

  • noindex: A directive, set in a robots meta tag or an X-Robots-Tag HTTP header, that asks search engines not to include a page in their results.
  • XML sitemap: A file that lists the URLs you want search engines to know about, optionally with last-modified dates.
  • Crawler: A program that fetches web pages automatically.

Go deeper

  • robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.

← All 66 glossary terms

↑ ↓ to moveEnter to openEsc to closeAll results