AI crawler checker · free
Is your site blocking AI search access?
This checker reads your robots.txt and separates search discovery, user-requested retrieval, and model-training access for documented OpenAI, Anthropic, and Perplexity agents.
Why the purpose of each agent matters
Search agents, user-requested retrieval agents, and training agents do different jobs. Blocking a search agent can limit whether your page content is used in that provider's answers, while blocking a training agent expresses a separate data-use choice.
The tool scores only documented search and retrieval access. It still shows training access, but never tells you to opt into training just to improve the score. Permission removes one access barrier; it does not guarantee crawling, citation, or recommendation.
What this checker reads
- ✓Current documented agents from OpenAI, Anthropic, and Perplexity, including Claude-User and Claude-SearchBot.
- ✓Whether each one is allowed or blocked at your site root, accounting for specific rules that override the wildcard.
- ✓A search-access score that excludes model-training consent.
- ✓An optional llms.txt presence note, clearly separated from crawler policy.
- ✓A robots.txt snippet only for blocked search and user-retrieval agents.
Free here, deeper in the audit
This checks whether AI can reach you. It cannot tell you what the engines actually say once they can. That is the full audit: 36 grounded questions across ChatGPT, Claude, and Gemini, the competitors showing up instead, and the ranked fixes. $39, once.
Questions
What does it mean if an agent is blocked?
Your robots.txt asks that agent not to crawl the matched path. The consequence depends on its documented purpose, so the tool separates search, user retrieval, and training rather than treating every block as the same problem.
Do I have to allow training agents for AI visibility?
No. Providers document separate agents for search or user retrieval and for potential model training. Your training decision should follow your publishing and data policy, not a visibility score.
Will allowing a search crawler get me cited?
No. It removes an access barrier. Relevance, indexability, content quality, evidence, freshness, and provider-specific systems still determine whether a page is selected.
Does this change anything on my site?
No. It only reads your public robots.txt and reports what it finds. You apply the fix yourself by editing your own file.
Keep going
Crawlers can reach you. But do the engines recommend you?
~5 min scan · $39 · refunds if useless
Run the full audit →