AI crawler checker · free
Is your site blocking AI crawlers?
An AI crawler checker reads your robots.txt and tells you, bot by bot, whether ChatGPT, Claude, Perplexity, and the rest are allowed to crawl you, or quietly blocked.
Why a blocked crawler is the quietest way to lose
If your robots.txt disallows an AI crawler, that engine cannot read your site, so it cannot mention you, cite you, or recommend you. There is no error and no missing click to notice. You simply never show up in the answer, and you never find out why.
This is the single most common fix we find in a full audit. A line meant to block one scraper often catches GPTBot or ClaudeBot by accident, and it stays there for months because nobody thinks to look.
What this checker reads
- ✓Every major AI user-agent: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Amazonbot, Applebot-Extended, and more.
- ✓Whether each one is allowed or blocked at your site root, accounting for specific rules that override the wildcard.
- ✓Whether you have an llms.txt at the root, checked in the same pass.
- ✓The exact robots.txt snippet to paste if anything is blocked.
Free here, deeper in the audit
This checks whether AI can reach you. It cannot tell you what the engines actually say once they can. That is the full audit: 36 grounded questions across ChatGPT, Claude, and Gemini, the competitors showing up instead, and the ranked fixes. $39, once.
Questions
What does it mean if a bot is blocked?
It means your robots.txt tells that crawler not to read your site, so the engine behind it cannot use your pages when it answers buyer questions. Blocking a training bot and a search bot are different trade-offs, and the tool labels which is which.
Is allowing AI crawlers safe?
For visibility, yes. If you want models to mention and cite you, they have to be able to read you. Some teams choose to block training crawlers while allowing search ones, which is a reasonable middle ground the tool helps you see clearly.
Do I need to allow every bot?
No. Allow the search and answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended) at minimum, since those feed live answers. Training crawlers are your call.
Does this change anything on my site?
No. It only reads your public robots.txt and reports what it finds. You apply the fix yourself by editing your own file.
Keep going
Crawlers can reach you. But do the engines recommend you?
~5 min scan · $39 · refunds if useless
Run the full audit →