Why it matters
Crawlers decide what can be found. Knowing who visits, and which rules they follow, lets you control access through robots.txt and server settings.
Example
User-agent: Mozilla/5.0 (compatible; WebsiteHealthLabsBot/1.0; +https://websitehealthlabs.com/about/#bot)Where you will see it
The free Robots.txt tester looks at this on a live page and explains what it finds in plain English: see which crawlers can fetch a URL, spot syntax mistakes and blocked CSS or JavaScript.
Related terms
- robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
- AI crawler: A crawler operated by an AI company, such as GPTBot (OpenAI) or ClaudeBot (Anthropic), that fetches pages for model training or AI search.
- Indexing: A search engine storing a page in its index so that it can appear in results.
Go deeper
- robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.