Eudoxia

Is your hosting or Cloudflare blocking AI crawlers?

Updated · 2 min read

robots.txt is a polite request. But before a crawler even gets there, your hosting provider, CDN or security plugin decides whether to let it in at all. Many of them now offer, and some turn on by default, settings that block AI bots.

The result: your robots.txt says “welcome”, while your firewall quietly sends AI assistants away.

Why this happens

Since 2024, many publishers have asked their providers to stop AI companies collecting their content. Providers responded with one-click switches to block AI crawlers. Cloudflare, which sits in front of a large share of websites, has blocked known AI crawlers by default for many new domains since 2025, and now sorts crawlers by purpose (search, agent and training) with separate settings for each.

These settings are reasonable for publishers protecting content. For a business that wants customers to find it through ChatGPT, Perplexity or Claude, blocking their answer crawlers is usually a mistake.

Signs you might be affected

  • Your robots.txt allows AI crawlers, but AI assistants say they can't access your site or describe it wrongly.
  • You use Cloudflare, a security plugin (such as a WordPress firewall) or a hosting plan with “bot protection”.
  • Your site was recently moved, redesigned, or put behind a new security service.

How to check

  1. Check robots.txt first with the free AI crawler checker. If it blocks answer crawlers, fix that before anything else.
  2. Cloudflare: open your domain in the dashboard and look at the bot and AI crawler settings (Cloudflare calls this area AI Crawl Control, and also has Security → Bots settings). See which crawlers are blocked, and whether a managed robots.txt is being added for you.
  3. WordPress: check your security or firewall plugin for “block AI bots”, “bad bots” or rate-limit rules.
  4. Hosting: look for “bot protection” or “AI scraper blocking” options in your control panel, or ask support whether they block OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot.

What to allow and what to block

A sensible setup for most businesses: allow answer and search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot), and decide separately about training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended). Blocking training is a reasonable choice and doesn't stop you appearing in answers. See GPTBot vs OAI-SearchBot for the full list.

In Cloudflare, that usually means allowing the search and agent categories (or the individual answer crawlers) while blocking training. Menus change often, so follow Cloudflare's current documentation for the exact steps.

Keep it from breaking again

Providers change their defaults, plugins update, and new security services get added. Re-check after any change to your hosting, CDN or security setup. Eudoxia Pro checks your robots.txt and AI crawler access every day and emails you if answer crawlers become blocked.

Questions

Will allowing AI crawlers slow down my website?

Answer crawlers fetch pages occasionally and in small numbers for a typical business site. If one becomes a problem, rate-limit it rather than blocking it outright.

Does Cloudflare's AI blocking affect Google Search?

Google Search uses Googlebot, which isn't treated like a training-only AI crawler. Be careful with broad rules, though: Cloudflare notes that some blocking choices can also affect multi-purpose crawlers, so read each setting's description before saving.

More guides