Why it matters
It is the only way to apply noindex to files that have no HTML head, such as PDFs, and it can be set for whole directories on the server.
Example
X-Robots-Tag: noindexWhere you will see it
The free HTTP header checker looks at this on a live page and explains what it finds in plain English: see status, redirects, every response header and timing, with plain-English notes.
Related terms
- noindex: A directive, set in a robots meta tag or an X-Robots-Tag HTTP header, that asks search engines not to include a page in their results.
- robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
- HTTP header: A name and value pair sent along with a request or a response.
Go deeper
- robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.