GPTBot vs OAI-SearchBot: which AI crawlers should you allow?
Updated · 2 min read
Many website owners blocked “AI bots” in 2023 and 2024 to stop their content being used for training. The problem: some of those rules also stop ChatGPT, Perplexity and Claude from showing and linking to your site when customers ask for a recommendation.
AI companies now use separate crawlers for training and for answering questions. Understanding the difference lets you protect your content and still appear in AI answers.
The two kinds of AI crawler
Answer crawlers fetch your pages so an AI assistant can quote, summarise and link to them in a live answer. Blocking them makes you invisible in those answers.
Training crawlers collect content to train future AI models. Blocking them has no effect on whether you appear in today's answers.
The main crawlers to know
- OAI-SearchBot (OpenAI, answers): lets ChatGPT search show and link to your site. Allow it if you want to be recommended in ChatGPT.
- ChatGPT-User (OpenAI, answers): fetches a page when a ChatGPT user asks about it or shares a link.
- GPTBot (OpenAI, training): collects content for training OpenAI models.
- PerplexityBot (Perplexity, answers): indexes your site for Perplexity answers.
- Claude-SearchBot (Anthropic, answers): indexes your site for Claude's search answers.
- ClaudeBot (Anthropic, training): collects content for training Claude models.
- Google-Extended (Google, training and Gemini use): controls whether Google may use your content for Gemini. It doesn't affect Google Search rankings.
- Applebot-Extended (Apple, training): controls use of your content for Apple Intelligence.
Crawler names and purposes change from time to time, so check each company's official documentation when you set up your rules.
Recommended setup for most businesses
If you want customers to find you through AI assistants, but you'd rather not donate your content to model training, use this approach:
- Allow the answer crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot.
- Decide about the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended). Blocking them is a reasonable choice for most small businesses and has no direct effect on AI answers.
- Check your hosting, CDN and security plugin settings too. Some offer one-click “block AI bots” switches that override robots.txt.
Our free AI robots.txt generator builds this for you, and the AI crawler checker shows what your current robots.txt allows.
What robots.txt can and can't do
robots.txt is a public request, not a lock. Reputable AI companies follow it, but it doesn't stop every crawler, and it doesn't remove content that was collected in the past. For sensitive content, use logins or server-level blocking instead.
Questions
If I block GPTBot, will ChatGPT stop recommending me?
Not by itself. GPTBot is OpenAI's training crawler. ChatGPT's live answers rely on OAI-SearchBot and ChatGPT-User. Allow those two to stay visible in ChatGPT answers.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls use of your content for Gemini and related AI. Google Search crawling uses Googlebot, which is separate.