
Key points
- Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it.
- Rules are matched by the most specific user-agent group, then the longest matching path. Allow wins a tie.
- A missing file (404) means "allow everything". A server error (5xx) can make Google stop crawling. Never block CSS or JavaScript.
What robots.txt is
Robots.txt lives at the root of a host, for example https://example.com/robots.txt. Each host and each protocol has its own file, so blog.example.com needs its own. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and well-behaved crawlers such as Googlebot and Bingbot read it before they fetch anything else.
User-agent: *
Disallow: /cart/
Disallow: /search
User-agent: Googlebot-Image
Disallow: /private-photos/
Sitemap: https://example.com/sitemap.xml
A group starts with one or more User-agent lines and is followed by Allow and Disallow rules. Sitemap lines stand on their own and apply to the whole file.
How rules are matched
- Pick the group. A crawler uses the group that names it most specifically. Googlebot uses
User-agent: Googlebotif it exists, otherwiseUser-agent: *. Groups are not combined across different names. - Find matching rules. A rule matches when the URL path starts with its pattern.
*stands for any run of characters and a trailing$means "end of URL". The path includes the query string. - Longest rule wins. Of all matches, the one with the longest pattern decides. If an Allow and a Disallow tie, Allow wins.
So Disallow: /shop/ with Allow: /shop/sale/ blocks the shop but lets the sale pages through.
It does not remove pages from search
Robots.txt stops crawlers fetching a URL. It does not stop the URL being indexed. If other sites link to a blocked page, search engines may still list the address without a description. Worse, a crawler that is blocked cannot see a noindex tag on the page. To keep a page out of results, allow crawling and add <meta name="robots" content="noindex"> or an X-Robots-Tag header. Google stopped supporting a noindex: line inside robots.txt in 2019.
What the response code means
- 200: the rules are read. Google reads only the first 500 KiB.
- 404 or other 4xx: no robots.txt, so everything may be crawled.
- 5xx or no answer: treated as "everything is disallowed" for a while, and a long-lasting error can drop pages from the index. Keep the file reliable.
- A web page instead of text: a catch-all route returned HTML. Crawlers will find nothing useful. Serve a real
text/plainfile.
Mistakes that cost traffic
Disallow: /left over from staging. It blocks the whole site. Check it after every launch.- Blocking CSS, JavaScript or images. Google renders pages like a browser. If it cannot load your assets it may misjudge layout and content.
- Typos.
Dissallowor a missing colon makes a line useless to some crawlers. - Rules before any
User-agentline. They apply to nobody. - Expecting
Crawl-delayto work in Google. It is ignored by Google. Bing and some others honour it. - Relative
Sitemapaddresses. Use the full URL.
AI crawlers and robots.txt
AI companies publish crawler names you can allow or block in the same file: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, CCBot (Common Crawl), and the control tokens Google-Extended and Applebot-Extended. Several separate training from search, so you can opt out of training but stay visible in AI search. Two cautions: robots.txt is a request that only honest crawlers obey, and blocking Google-Extended does not affect Google Search.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
How to test it
Before you save a change, check the exact URLs that matter against the new rules: the home page, a product page, your CSS file and a page you meant to block. After you publish, request /robots.txt in a browser and confirm it returns 200 as plain text.
Robots.txt is public. Do not list private folder names in it as if it were a security measure; anyone can read the file. Protect private content with authentication.
See also: Find and fix broken links, Website down: what to check.
FAQ
Do I need a robots.txt file?
No. Without one, crawlers assume everything is allowed. You need one when you want to keep crawlers out of certain paths or point them at your sitemap.
Can I use robots.txt to hide a page?
No. It asks crawlers not to fetch the page. People can still open it, and the URL can still show in search results if it is linked elsewhere. Use a login or noindex.
Does Google follow wildcards?
Yes. Google and Bing support * and a trailing $. Not every small crawler does.
How long until a robots.txt change takes effect?
Google generally refreshes its copy within a day. It can request a refresh sooner after a big change.