1. Home
  2. Glossary
  3. X-Robots-Tag
SEO

X-Robots-Tag

An HTTP response header that carries the same directives as the robots meta tag, such as noindex. It also works for files that have no HTML head, such as PDFs and images.

Why it matters

It is the only way to apply noindex to files that have no HTML head, such as PDFs, and it can be set for whole directories on the server.

Example

X-Robots-Tag: noindex

Where you will see it

The free HTTP header checker looks at this on a live page and explains what it finds in plain English: see status, redirects, every response header and timing, with plain-English notes.

Related terms

  • noindex: A directive, set in a robots meta tag or an X-Robots-Tag HTTP header, that asks search engines not to include a page in their results.
  • robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
  • HTTP header: A name and value pair sent along with a request or a response.

Go deeper

  • robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.

← All 66 glossary terms

↑ ↓ to moveEnter to openEsc to closeAll results