Why it matters
Whether AI services may read your pages is controlled by robots.txt rules aimed at their named crawlers. If you have a view either way, it is worth stating it explicitly.
Example
User-agent: GPTBot
Disallow: /Where you will see it
The free AI crawler access checker looks at this on a live page and explains what it finds in plain English: check which AI crawlers your robots.txt allows and copy rules to block training bots or allow AI search.
Related terms
- robots.txt: A plain-text file at the root of a site that tells crawlers which paths they may crawl.
- Crawler: A program that fetches web pages automatically.
- llms.txt: A proposed plain-text Markdown file at /llms.txt that gives AI assistants a short, curated guide to a site's most useful pages.
Go deeper
- robots.txt explained: What robots.txt does and does not do, how Allow and Disallow rules are matched, the mistakes that hide sites from search, and how to handle AI crawlers.