A broad robots.txt rule can block a provider's search or user-retrieval agent while ordinary Google indexing still looks healthy. The error stays invisible unless you inspect the policy, delivery logs, or answer-level visibility.
The important correction is that AI user agents do not all do the same job. A training opt-out is not a search opt-out. The current provider documentation names separate agents, and a responsible audit keeps those choices separate.
The documented agents in this check
Use current provider documentation as the source of truth. This table covers the agents Ron actively checks as of August 2026.
| User-Agent | Vendor | Purpose | Type |
|---|---|---|---|
| GPTBot | OpenAI | Potential use to improve generative models | Training |
| OAI-SearchBot | OpenAI | ChatGPT search discovery and answer links | Search |
| ChatGPT-User | OpenAI | Some actions taken at a user's request | User retrieval |
| ClaudeBot | Anthropic | Model development | Training |
| Claude-SearchBot | Anthropic | Improves search-result quality | Search |
| Claude-User | Anthropic | Access directed by a user | User retrieval |
| PerplexityBot | Perplexity | Builds and refreshes its search index | Search |
| Perplexity-User | Perplexity | Supports user-requested page access | User retrieval |
The one distinction that actually matters: training vs. search-and-answer
Most confusion starts with one blended AI crawler score. There are at least three policy jobs here: search discovery, user-directed retrieval, and potential model training.
Training crawlers
GPTBot and ClaudeBot are documented for potential model improvement or development. Blocking them is a data-use decision. It should not lower a search-access score, and allowing them should not be sold as a direct citation or ranking tactic.
Search and answer crawlers
OAI-SearchBot, Claude-SearchBot, and PerplexityBot are documented for search discovery. ChatGPT-User, Claude-User, and Perplexity-User support forms of user-requested access. Blocking them can limit page-content access in those workflows, but permission still does not guarantee crawling, selection, citation, or recommendation.
Blocking GPTBot is a training choice. Blocking OAI-SearchBot is a search-access choice. One score should never hide that difference.
How to check your robots.txt right now
Your robots.txt lives at yourdomain.com/robots.txt. Open it in a browser. You are looking for two things.
- A wildcard block that catches everything: a User-agent: * line followed by Disallow: / or a broad path. If this exists without explicit Allow rules for specific bots above it, every crawler including AI bots is blocked.
- Named agent blocks: search for all eight documented names above and classify each block by purpose before changing it.
- Group specificity: a matching named user-agent group is evaluated instead of a wildcard group for that agent. Test the effective rule rather than relying on visual order.
- Empty or missing file: a missing robots.txt means no restrictions at all, which is generally fine. The problems come from files with rules.
Common ways sites end up accidentally blocked
- ✓Security plugins like Wordfence or iThemes Security on WordPress sometimes add aggressive Disallow rules for unknown bots by default.
- ✓CDN or WAF configurations (Cloudflare, Sucuri) that block user-agents not on an allowlist. AI crawlers are new enough that many default lists do not include them.
- ✓A template robots.txt copied from a high-traffic site that deliberately blocks all third-party crawlers.
- ✓A developer testing a staging block who published the robots.txt to production by mistake.
- ✓Rate-limiting rules that send 429s to any bot hitting more than a few pages per minute. Robots.txt technically allows the crawl but the server refuses it in practice.
The corrected robots.txt snippet
This starter expresses one common policy: allow documented search and user-retrieval agents while blocking documented training agents. Review current provider docs, legal requirements, and your own publishing policy before use.
robots.txt: separate access by purpose
# Search and user-requested retrieval User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Potential model training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Real private paths still require authentication User-agent: * Disallow: /account/ Disallow: /checkout/
An Allow rule removes one access barrier. It does not guarantee a fetch, index entry, citation, recommendation, or referral visit. robots.txt is also public and is not access control for private information.
The trade-off worth thinking through
The training decision belongs with the organization that owns the content and risk. The search decision belongs with the distribution strategy. Private, licensed, user-generated, and expensive application routes may also need path-specific handling beyond a simple sitewide choice.
For public pages intended to appear in AI search or user-requested answers, unintended access blocks are worth fixing. Preserve the reason, owner, provider source, test URLs, and next review date so the policy stays accountable.
8
current OpenAI, Anthropic, and Perplexity agents included in Ron's documented policy check
One silent failure worth knowing about
A robots.txt block does not produce a universal alert. The most reliable process is to test the effective rules, inspect CDN and origin behavior, and then measure answer citations or recommendation context separately.
That is the core reason to run an audit now rather than assuming your current configuration is fine. The defaults from 2022 or 2023 almost certainly did not account for OAI-SearchBot or PerplexityBot. They did not exist yet.
What to do next
Start by pulling up your robots.txt. Scan it for the user-agents in the table above. If you find a Disallow on any of the search-and-answer bots, fix it using the snippet above. If the file looks clean at the robots.txt level, check your server-side WAF rules and any security plugins for bot filtering logic that might be operating separately.
If you want a faster read without digging through files manually, the free AI Crawler Checker tool linked below fetches your robots.txt and highlights any rules affecting known AI crawlers. It takes about ten seconds and flags training-only blocks separately from search-and-answer blocks so you know exactly what you are dealing with.