GPTBot, OAI-SearchBot, ClaudeBot: Which AI Crawlers to Allow in robots.txt

Many websites now block “AI bots” in their robots.txt file. Often the goal is simple: keep content out of AI training. The side effect is less obvious. The same rules can also remove the site from the answers that ChatGPT, Claude and Perplexity give when people ask for recommendations. This guide explains which AI crawlers exist, what each one does, and how to set up robots.txt so you decide what to allow instead of blocking everything by accident.

Three Jobs, Three Kinds of Bots

AI companies send different bots for different purposes, and robots.txt treats each one separately.

• Training crawlers collect public web content that may be used to train AI models.
• Search crawlers build the index that AI assistants use to find and cite websites in their answers.
• User-initiated fetchers open a specific page because a person asked the assistant to read it.

Blocking a training crawler and blocking a search crawler have very different consequences. The first limits how your content is used for future models. The second can make your site invisible in AI search.

The Bots That Matter

OpenAI

GPTBot collects content that may be used to train OpenAI’s models. OAI-SearchBot is the one that affects visibility: OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. ChatGPT-User handles actions that a user starts in ChatGPT, and OpenAI notes that robots.txt rules may not apply to it.

Anthropic

ClaudeBot collects content that may contribute to model training. Claude-SearchBot indexes content to improve search results, and Anthropic says that disabling it may reduce your visibility in Claude’s answers. Claude-User fetches pages when a person asks Claude a question, and blocking it prevents Claude from reading your site for those requests. Anthropic also supports the non-standard Crawl-delay directive if you want to slow its bots down.

Perplexity

PerplexityBot builds the index behind Perplexity’s answers, and Perplexity recommends allowing it if you want to appear in its results. Perplexity-User fetches pages for user requests, and Perplexity states that it generally ignores robots.txt because a person asked for the page.

Google

Google-Extended is not a separate crawler. It is a robots.txt token that controls whether content Google has already crawled may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google states that it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Google Search, so blocking Google-Extended does not remove your pages from them.

Three robots.txt Setups

1. Maximum Visibility

Allow every crawler. This suits most service businesses that want to be found and recommended.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

2. No Training, Stay Visible in AI Search

Block the training crawlers and keep the search crawlers open. This is the setup most businesses actually want when they say “block AI”.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

3. Block AI Crawlers Completely

Use this only if keeping content away from AI matters more than being recommended by AI assistants. Keep Googlebot open, or your site will disappear from Google Search as well.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Replace example.com with your own domain in every setup.

Common Mistakes

• Blocking everything with one rule. A blanket block on AI bots often removes the search crawlers too, and with them your chance of being cited.
• Expecting Google-Extended to remove you from AI Overviews. It does not. AI Overviews follow Google Search rules, such as noindex and snippet controls.
• Forgetting the firewall. Some hosting providers, CDNs and security plugins block bots on their own. Your robots.txt may allow a crawler while the server still refuses it.
• Treating robots.txt as protection. It is a request that reputable bots follow, not a lock. User-initiated fetchers may not follow it, and content that must stay private needs a login.

How to Check Your Site

1. Open yourdomain.com/robots.txt in a browser and read the rules for each bot listed above.
2. Check whether a security plugin, CDN or hosting firewall has a setting for AI bots.
3. Look at your server logs for visits from OAI-SearchBot, Claude-SearchBot and PerplexityBot. If they never appear, something may be blocking them.
4. Review the setup every few months. AI companies add and rename bots, so check their official documentation.

Where This Fits in AI Visibility

Allowing the right crawlers does not guarantee that an assistant will recommend you. It only makes recommendation possible. What assistants say about your company depends on your content and on the outside sources they trust, which we cover in how AI search engines choose which company to recommend. If you want this checked for your own site, see our AIO and GEO visibility service, or read our guide on choosing an AI search optimization agency in Cyprus.