- Home
- Free tools
- AI crawler access checker
AI crawler access checker
Check whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot and other AI crawlers are allowed to read your site, and copy the rules to change that.
How to use it
- Enter any page address on your site.
- Read the table: each AI crawler is marked Allowed or Blocked for that page, with its purpose (training, AI search, or a user-requested fetch).
- Pick a policy in the snippet box, add it to robots.txt and test again.
What it checks
- 14 well-known AI-related crawlers and opt-out tokens
- Whether each is allowed for the page you typed, using robots.txt matching rules
- Which crawlers have their own rule and which fall back to User-agent: *
- Training bots versus AI search bots versus user-requested fetches
- Whether an llms.txt file exists
Limits
- Reads robots.txt rules for a fixed list of known AI crawlers. New crawlers and renamed tokens appear over time.
- Robots rules are a request. Crawlers that ignore robots.txt are not stopped, and this tool cannot detect them.
- Does not check firewall or CDN rules that may block bots outside robots.txt.
Public pages only, one page at one point in time. See the methodology and terms.
Frequently asked questions
Does blocking GPTBot or ClaudeBot remove my content from AI answers?
Blocking a training crawler asks the company not to use new pages for training. It does not remove what a model already learned. AI search products use separate crawlers (for example OAI-SearchBot and PerplexityBot), so you can block training and still allow search.
Will blocking Google-Extended hurt my Google ranking?
No. Google-Extended is a control token for Gemini training and grounding. Google Search crawls with Googlebot, which this token does not affect.
Do all AI companies obey robots.txt?
The big providers say they do. Robots.txt is a request, not enforcement, and smaller scrapers may ignore it. For hard blocking you need server-side or CDN rules.
What is llms.txt?
A proposed Markdown file at /llms.txt that summarises a site for AI assistants. It is not a standard and no major provider has promised to use it, so treat it as optional.
Should I block AI crawlers?
It depends on your goals. Publishers who sell content often block training. Businesses that want to be cited in AI answers often allow search and user crawlers. This tool shows your current policy so you can choose on purpose.