By Hridoy Ahmed | Last updated: October 2026 | 9 min read
Short answer: AI companies run separate crawlers for training, search and user requests. You control most of them with robots.txt, one user-agent at a time. If you want to appear in AI search tools, allow their search crawlers, such as OAI-SearchBot and Claude-SearchBot. If you only want to opt out of model training, block the training crawlers, such as GPTBot and ClaudeBot. Blocking is a business choice, and robots.txt is a request, not a lock.
In this guide
- The three jobs AI crawlers do
- Main AI crawlers and what they do
- Google is different
- 3 common robots.txt choices
- What robots.txt cannot do
- FAQ
The three jobs AI crawlers do
- Training: collects public web content that may be used to train AI models.
- Search: builds an index so a site can appear in an AI tool’s search answers.
- User requests: fetches a page when a person asks the AI tool about it.
These are separate bots. Blocking one does not block the others. That is the most common mistake I see.
Main AI crawlers and what they do
These details come from each company’s own documentation. Bot names and rules can change, so check the official pages before you edit robots.txt.
OpenAI
- GPTBot is the control for whether your content may be used to train OpenAI’s models.
- OAI-SearchBot is used to surface sites in ChatGPT’s search features. OpenAI says sites opted out of it will not be shown in ChatGPT search answers.
- ChatGPT-User is used for certain actions that a user starts in ChatGPT. Because a person starts them, OpenAI says robots.txt rules may not apply.
OpenAI says it can take about 24 hours for a robots.txt change to take effect for search. See OpenAI’s crawler documentation.
Anthropic (Claude)
- ClaudeBot collects web content that may contribute to model training.
- Claude-User may access a site when a person asks Claude a question. Anthropic says disabling it may reduce your visibility for user-directed web search.
- Claude-SearchBot helps improve search result quality. Anthropic says disabling it may reduce your visibility and accuracy in user search results.
Anthropic says its bots honor robots.txt, and that blocking IP addresses instead may not work well, because the bot then cannot read your robots.txt. See Anthropic’s page on its bots.
Perplexity
- PerplexityBot is used for search indexing.
- Perplexity-User fetches pages when a person asks a question. Guides based on Perplexity’s documentation say it generally ignores robots.txt for that reason. Check Perplexity’s own bot page for current details.
Google is different
Google’s AI Overviews and AI Mode are part of Google Search. They use Googlebot, so Googlebot’s robots.txt rules are the crawling control. If you block Googlebot, you also leave Google Search.
Google-Extended is a separate robots.txt token. It controls whether Google-crawled content may be used for Gemini training and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search. To limit what Google shows from your pages in its AI features, Google points to the usual snippet controls, such as nosnippet, max-snippet and noindex. See my guide to Google AI Overviews and AI Mode.
3 common robots.txt choices
Choice 1: Allow everything
You do not add any AI rules. Use this if you want the widest reach and you are fine with training use. Make sure no broad rule blocks the AI bots by accident.
Choice 2: Allow search, block training
Use this if you want to appear in AI answers but not contribute to model training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /Choice 3: Block AI bots
Use this only if you are sure you do not want AI visibility. Add a Disallow: / rule for each bot you want to block. You may lose visibility in the AI tools that rely on those bots.
Whichever you pick, test it. Fetch your live robots.txt and check each rule. My guide to creating a robots.txt file for WordPress and my free Robots.txt Generator can help. If you would rather have this reviewed for you, see my AI search optimization service.
What robots.txt cannot do
- It is a request. Well-behaved bots follow it. Unknown scrapers can ignore it. For those, use rate limits or a web application firewall.
- It is not security. Never use robots.txt to hide private pages. Use a login or proper access control.
- User agents can be faked. OpenAI publishes IP ranges for its bots, so you can check a real visit. Verify before you trust a name in your logs.
- It does not remove pages already collected. It controls future crawling.
- It does not guarantee citations. Allowing a search bot only makes a page eligible to be used.
Common mistakes
- Blocking GPTBot and thinking you are hidden from ChatGPT search
- Blocking all unknown bots with a broad rule and losing search crawlers
- Using old names like Claude-Web or Anthropic-AI instead of current bot names
- Copying a robots.txt from a blog without checking what each rule does
- Treating robots.txt as a security tool
FAQ
Should I block AI crawlers?
It depends on your goals. If you want to appear in AI answers, allow the search crawlers. If you do not want your content used for training, block the training crawlers. Make the choice on purpose.
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot and OAI-SearchBot are separate crawlers. OpenAI says OAI-SearchBot is the one that controls appearing in ChatGPT search.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search.
Do all AI bots follow robots.txt?
No. Major companies say their main crawlers follow it, but user-triggered fetchers may not, and unknown scrapers can ignore it.
Not sure what your robots.txt should say? I can review it with you and explain each choice. Work with me on AI search optimization.
About the author: Hridoy Ahmed is an SEO specialist and WordPress security expert based in Dhaka, Bangladesh. He has 8+ years of experience and works with clients worldwide.