The short answer
OAI-SearchBot supports ChatGPT search discovery and answer links, ChatGPT-User handles some user-requested visits, and GPTBot controls potential model-training access; blocking one does not express the same choice for the others.
Business outcome
You can preserve discoverability in ChatGPT search while making an independent decision about training access.
The process
Build it in five passes
Map each agent to its purpose
Treat OAI-SearchBot as the primary search-discovery control, ChatGPT-User as user-triggered access, and GPTBot as the control for content that may be used to improve generative models. Use the current OpenAI publisher documentation as the source of truth.
Choose the search policy
If you want public pages considered for ChatGPT search summaries and snippets, allow OAI-SearchBot to crawl them. OpenAI says a disallowed page can still appear as a link and title from third-party information, but inclusion of page content in summaries depends on OAI-SearchBot access.
Choose the training policy separately
Decide whether GPTBot may access all public content, selected paths, or none. This choice should come from your data, legal, and publishing policy. Do not assume blocking GPTBot blocks ChatGPT search, and do not assume allowing search opts into training.
Check the full delivery path
Verify robots.txt, noindex or snippet controls, CDN and firewall behavior, HTML response, canonical URL, and server logs. A user-agent rule that looks correct in a file is not enough if the request is challenged or the page returns incomplete content.
Measure actual outcomes
Track ChatGPT referral traffic using raw session sources and monitor answer-level citations with a stable prompt set. OpenAI documents `utm_source=chatgpt.com` on referral URLs, but referral visits do not capture all answer visibility.
Before it ships
Quality checklist
- OAI-SearchBot has an explicit, intentional policy.
- ChatGPT-User has an explicit, intentional policy.
- GPTBot is governed as a separate training decision.
- Public priority pages are not blocked by a broad wildcard rule.
- CDN, WAF, and origin behavior are tested with representative requests.
- Referral measurement retains the raw chatgpt.com source value.
Copyable artifact
OpenAI decision record
Complete this before changing robots.txt so the published rules reflect an accountable business choice.
PROVIDER: OpenAI REVIEW DATE: [YYYY-MM-DD] OWNER: [name and team] OAI-SearchBot Purpose: search discovery and answer links Decision: allow | restrict paths | block Business reason: [reason] Test URLs: [public page, blocked path] ChatGPT-User Purpose: user-requested page access Decision: allow | restrict paths | block Business reason: [reason] Test URLs: [public page, blocked path] GPTBot Purpose: potential model training Decision: allow | restrict paths | block Business reason: [reason] Test URLs: [public page, blocked path] NEXT REVIEW: [date or provider-change trigger]
Validation
How you know it is ready
- 01The live robots.txt behavior matches all three recorded decisions.
- 02Priority pages return useful HTML without a challenge or authentication wall.
- 03Search visibility and referral outcomes are measured separately from crawl access.
Do not overclaim
Provider documentation and product behavior can change. Recheck OpenAI's current instructions before deployment, and do not describe crawler access as a ranking factor or guarantee.
Questions
What teams usually ask
If GPTBot is blocked, can ChatGPT still cite us?
OpenAI documents separate controls for search and training. Search inclusion depends on OAI-SearchBot and other product systems, not GPTBot permission alone.
Is ChatGPT-User the search crawler?
No. It is associated with user-requested access. OAI-SearchBot is the documented agent to consider for ChatGPT search discovery and summaries.
Will allowing OAI-SearchBot guarantee a citation?
No. Permission removes one access barrier. It does not guarantee crawling, selection, citation, recommendation, or referral traffic.
Sources reviewed