1. Home
  2. Glossary
  3. Crawler
SEO

Crawler (bot)

A program that fetches web pages automatically. Search engines, AI services and our own audit use crawlers. Each one identifies itself with a user-agent string, and robots.txt can allow or disallow it by that name.

Why it matters

Crawlers decide what can be found. Knowing who visits, and which rules they follow, lets you control access through robots.txt and server settings.

Example

User-agent: Mozilla/5.0 (compatible; WebsiteHealthLabsBot/1.0; +https://websitehealthlabs.com/about/#bot)

Where you will see it

The free Robots.txt tester looks at this on a live page and explains what it finds in plain English: see which crawlers can fetch a URL, spot syntax mistakes and blocked CSS or JavaScript.

Related terms

  • robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
  • AI crawler: A crawler operated by an AI company, such as GPTBot (OpenAI) or ClaudeBot (Anthropic), that fetches pages for model training or AI search.
  • Indexing: A search engine storing a page in its index so that it can appear in results.

Go deeper

  • robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.

← All 66 glossary terms

↑ ↓ to moveEnter to openEsc to closeAll results