The short answer
An AI crawler policy should classify each documented user agent by purpose, decide access at the narrowest practical path, record the owner and reason, and test the live robots.txt file after deployment.
Business outcome
You avoid accidentally blocking AI search while preserving a separate, documented choice about model training and sensitive paths.
The process
Build it in five passes
Inventory documented agents
Build the list from provider documentation, not copied blog lists. Separate search indexing, user-triggered retrieval, and potential model training. OpenAI, Anthropic, and Perplexity each document agents with different purposes, and those names can change.
Choose access by purpose
Decide whether public pages should be available for search and answer retrieval. Make the training decision separately. Keep private, account, checkout, internal search, staging, and expensive parameter routes out of scope regardless of provider.
Write the smallest rules
Prefer clear user-agent groups and specific disallow paths. Do not add an Allow rule for the whole site unless it helps explain a broader Disallow. Remember that robots.txt is public and is not access control for confidential information.
Check page-level controls
Review noindex, nosnippet, max-snippet, authentication, canonical tags, and response headers. A permissive robots.txt cannot make a noindexed or inaccessible page eligible, and a robots block can prevent a crawler from seeing a page-level noindex instruction.
Test and monitor
Fetch the live file from the exact production host, test representative URLs for every rule, inspect server logs where available, and add a review date. Recheck provider documentation before treating a user agent as current.
Before it ships
Quality checklist
- Every listed agent links back to current provider documentation.
- Search, user retrieval, and training are separate policy decisions.
- Sensitive and authenticated content is protected by real access controls.
- Rules are tested against representative allowed and blocked URLs.
- Sitemap locations use absolute production URLs.
- The file has an owner, decision record, and scheduled review date.
Copyable artifact
Search-friendly, training-restricted starter
This example allows documented search and user-retrieval agents while restricting documented training agents. Review current provider docs and your own policy before publishing.
# OpenAI search and user-requested access User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # OpenAI potential model training User-agent: GPTBot Disallow: / # Anthropic search and user-requested access User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # Anthropic potential model training User-agent: ClaudeBot Disallow: / # Perplexity search and user-requested access User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Protect non-public application routes for every crawler User-agent: * Disallow: /api/ Disallow: /account/ Disallow: /checkout/ Sitemap: https://www.example.com/sitemap.xml
Validation
How you know it is ready
- 01The live production file returns HTTP 200 as plain text without a redirect loop.
- 02Each representative URL produces the intended decision for every named agent.
- 03No confidential path relies on robots.txt as its only protection.
Do not overclaim
robots.txt is a voluntary crawling protocol, not a security boundary and not a guarantee of recommendation. Providers may use other sources, and search eligibility still depends on indexing, content, quality, and product-specific systems.
Questions
What teams usually ask
Can I allow AI search and block model training?
Providers document separate agents for these purposes, so a policy can distinguish them. Confirm each current user agent in the provider's official documentation before deployment.
Does an Allow rule improve AI rankings?
No. It only expresses crawl permission. It does not guarantee crawling, indexing, citation, recommendation, or any position.
Should robots.txt hide private URLs?
No. The file is public and may reveal the paths. Protect private content with authentication and authorization, then use crawler rules only as an additional operational control.
Sources reviewed