1. Home
  2. Glossary
  3. AI crawler
SEO

AI crawler

A crawler operated by an AI company, such as GPTBot (OpenAI) or ClaudeBot (Anthropic), that fetches pages for model training or AI search. Operators say they follow robots.txt, so you can allow or block each one by name.

Why it matters

Whether AI services may read your pages is controlled by robots.txt rules aimed at their named crawlers. If you have a view either way, it is worth stating it explicitly.

Example

User-agent: GPTBot
Disallow: /

Where you will see it

The free AI crawler access checker looks at this on a live page and explains what it finds in plain English: check which AI crawlers your robots.txt allows and copy rules to block training bots or allow AI search.

Related terms

  • robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
  • Crawler: A program that fetches web pages automatically.
  • llms.txt: A proposed plain-text Markdown file at /llms.txt that gives AI assistants a short, curated guide to a site's most useful pages.

Go deeper

  • robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.

← All 66 glossary terms

↑ ↓ to moveEnter to openEsc to closeAll results