1. Home
  2. Guides
  3. robots.txt explained
SEO guide

robots.txt explained: how to control what crawlers fetch

A robots.txt file is a short text file that tells crawlers which parts of your site they may fetch. It is simple, easy to get wrong, and a single bad line can keep a whole site out of search.

Key points

  • Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it.
  • Rules are matched by the most specific user-agent group, then the longest matching path. Allow wins a tie.
  • A missing file (404) means "allow everything". A server error (5xx) can make Google stop crawling. Never block CSS or JavaScript.

What robots.txt is

Robots.txt lives at the root of a host, for example https://example.com/robots.txt. Each host and each protocol has its own file, so blog.example.com needs its own. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and well-behaved crawlers such as Googlebot and Bingbot read it before they fetch anything else.

User-agent: *
Disallow: /cart/
Disallow: /search

User-agent: Googlebot-Image
Disallow: /private-photos/

Sitemap: https://example.com/sitemap.xml

A group starts with one or more User-agent lines and is followed by Allow and Disallow rules. Sitemap lines stand on their own and apply to the whole file.

How rules are matched

  1. Pick the group. A crawler uses the group that names it most specifically. Googlebot uses User-agent: Googlebot if it exists, otherwise User-agent: *. Groups are not combined across different names.
  2. Find matching rules. A rule matches when the URL path starts with its pattern. * stands for any run of characters and a trailing $ means "end of URL". The path includes the query string.
  3. Longest rule wins. Of all matches, the one with the longest pattern decides. If an Allow and a Disallow tie, Allow wins.

So Disallow: /shop/ with Allow: /shop/sale/ blocks the shop but lets the sale pages through.

It does not remove pages from search

Robots.txt stops crawlers fetching a URL. It does not stop the URL being indexed. If other sites link to a blocked page, search engines may still list the address without a description. Worse, a crawler that is blocked cannot see a noindex tag on the page. To keep a page out of results, allow crawling and add <meta name="robots" content="noindex"> or an X-Robots-Tag header. Google stopped supporting a noindex: line inside robots.txt in 2019.

What the response code means

  • 200: the rules are read. Google reads only the first 500 KiB.
  • 404 or other 4xx: no robots.txt, so everything may be crawled.
  • 5xx or no answer: treated as "everything is disallowed" for a while, and a long-lasting error can drop pages from the index. Keep the file reliable.
  • A web page instead of text: a catch-all route returned HTML. Crawlers will find nothing useful. Serve a real text/plain file.

Mistakes that cost traffic

  • Disallow: / left over from staging. It blocks the whole site. Check it after every launch.
  • Blocking CSS, JavaScript or images. Google renders pages like a browser. If it cannot load your assets it may misjudge layout and content.
  • Typos. Dissallow or a missing colon makes a line useless to some crawlers.
  • Rules before any User-agent line. They apply to nobody.
  • Expecting Crawl-delay to work in Google. It is ignored by Google. Bing and some others honour it.
  • Relative Sitemap addresses. Use the full URL.

AI crawlers and robots.txt

AI companies publish crawler names you can allow or block in the same file: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, CCBot (Common Crawl), and the control tokens Google-Extended and Applebot-Extended. Several separate training from search, so you can opt out of training but stay visible in AI search. Two cautions: robots.txt is a request that only honest crawlers obey, and blocking Google-Extended does not affect Google Search.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

How to test it

Before you save a change, check the exact URLs that matter against the new rules: the home page, a product page, your CSS file and a page you meant to block. After you publish, request /robots.txt in a browser and confirm it returns 200 as plain text.

Robots.txt is public. Do not list private folder names in it as if it were a security measure; anyone can read the file. Protect private content with authentication.

See also: Find and fix broken links, Website down: what to check.

FAQ

Do I need a robots.txt file?

No. Without one, crawlers assume everything is allowed. You need one when you want to keep crawlers out of certain paths or point them at your sitemap.

Can I use robots.txt to hide a page?

No. It asks crawlers not to fetch the page. People can still open it, and the URL can still show in search results if it is linked elsewhere. Use a login or noindex.

Does Google follow wildcards?

Yes. Google and Bing support * and a trailing $. Not every small crawler does.

How long until a robots.txt change takes effect?

Google generally refreshes its copy within a day. It can request a refresh sooner after a big change.

WebsiteHealthLabs Editorial
Guides explain audit results in plain English. Facts are checked against public documentation from Google, W3C and the browser vendors, and we update them when guidance changes.

↑ ↓ to moveEnter to openEsc to closeAll results