The short answer
Claude-SearchBot supports search-result quality, Claude-User accesses pages at a user's direction, and ClaudeBot is used for model development; Anthropic says all three honor robots.txt.
Business outcome
Your Claude access policy becomes deliberate, current, and testable instead of relying on outdated user-agent lists.
The process
Build it in five passes
Replace inherited bot lists
Compare your file and crawler tooling with Anthropic's current help documentation. Remove obsolete names only after verifying they are not required for another documented system, and add Claude-SearchBot and Claude-User when they match your policy.
Set the search decision
Decide whether Claude-SearchBot may access public editorial, product, help, and research pages. Keep low-value crawl spaces, private paths, internal search results, and expensive parameter combinations constrained.
Set the user-retrieval decision
Claude-User is used when a person directs Claude to visit a page. Decide whether your public information should be available in that context. Test any consent, geography, bot challenge, or WAF behavior that may differ from robots.txt.
Set the training decision
Govern ClaudeBot separately according to your publishing and data policy. Record whether restrictions apply sitewide or to particular licensed, user-generated, or sensitive public sections.
Add operations and review
Anthropic documents support for Crawl-delay. Use rate controls only when needed and monitor effects. Save the provider source, policy owner, test evidence, and a trigger to review the rules when documentation changes.
Before it ships
Quality checklist
- Claude-SearchBot is classified as search access.
- Claude-User is classified as user-directed retrieval.
- ClaudeBot is classified as model-development access.
- Old copied user-agent names are not treated as current without evidence.
- Rate limits solve an observed capacity issue rather than blocking by accident.
- Live origin and CDN responses match the written policy.
Copyable artifact
Anthropic access policy block
Use the choices, reasons, and test URLs to create your final robots.txt groups.
ANTHROPIC ACCESS REVIEW Date: [YYYY-MM-DD] Owner: [name] Official documentation checked: [URL and date] Claude-SearchBot Decision: allow | restrict | block Allowed public sections: [paths] Restricted sections: [paths] Reason: [business and publishing reason] Claude-User Decision: allow | restrict | block Allowed public sections: [paths] Restricted sections: [paths] Reason: [business and publishing reason] ClaudeBot Decision: allow | restrict | block Allowed public sections: [paths] Restricted sections: [paths] Reason: [training policy reason] Capacity control needed: yes | no Crawl-delay if justified: [seconds] Tested production URLs: [list] Next review trigger: [date or documentation change]
Validation
How you know it is ready
- 01Each documented Anthropic agent returns the intended decision on test URLs.
- 02A public priority page is readable as useful HTML through the full delivery stack.
- 03The decision record cites current Anthropic documentation and a review date.
Do not overclaim
Crawler names, functions, and support can change. Anthropic also notes that third-party search providers may have their own crawlers, so this policy controls Anthropic agents, not the entire information supply chain.
Questions
What teams usually ask
What happened to older Claude crawler names?
Use Anthropic's current documented list as the source of truth. Crawler inventories copied from older posts can become stale and should be reviewed before use.
Does Claude support crawl-delay?
Anthropic's current help documentation says its crawlers support the non-standard Crawl-delay field. Apply it only when rate management is actually needed.
Can Claude search be allowed while training is blocked?
The documented agents are separate, so robots.txt can express different choices for Claude-SearchBot and ClaudeBot.
Sources reviewed