Cloudflare AI bot controls: separate Search, Training, Agent

Cloudflare gives all customers new tools to control how AI bots access their websites, replacing its prior single-click blanket AI crawler block with three distinct categories. Site owners can now set separate rules for Search, Training and Agent bots, and the new options are available to users on the free plan.

What are the new AI bot categories?

Search, Training and Agent are the three categories site owners can choose between, and each maps to a specific crawler purpose. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT. Multi-purpose crawlers that serve more than one role, such as Googlebot, will be treated according to whichever rule is strictest.

Cloudflare first allowed customers to block all AI crawlers with a single click in July 2024. The three-category approach replaces that blanket option with granular controls that let site owners permit search indexing while blocking model training and interactive agents where they prefer.

How will defaults, verification and BotBase work?

Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, while Search crawlers will still be allowed. For enterprise customers Cloudflare is launching BotBase, a searchable database of known bots inside the dashboard that shows how each bot is classified and how it uses content.

Previously, all verified bots were allowed through automatically. Going forward a bot's category determines whether it gets access. The "Verified" label alone will not be enough: bot operators will need to prove they identify themselves honestly and do not abuse the access they are granted. Cloudflare is encouraging bot operators to split crawlers by purpose so site owners can apply different rules to search, training and agent traffic.

Why it matters

Cloudflare's change shifts control from an all-or-nothing block to targeted policies that reflect different business priorities, such as allowing search indexing while preventing data scraping for model training. The timing interacts with a broader trend: in June 2026 Cloudflare CEO Matthew Prince said bot traffic had surpassed human traffic on the internet for the first time, a milestone the company had originally expected to occur in late 2027. That makes granular controls more than a convenience; they determine who can reuse site content and where publisher revenue signals, like ads, influence access.

The free-plan availability widens the impact beyond large customers, and BotBase gives enterprises a single place to inspect classifications and content usage patterns, including whether a bot just links to pages or reproduces them in full.

What to watch

Track whether major bot operators split their crawlers by purpose, as Cloudflare is asking, or continue to run multi-purpose bots that will be subject to the strictest rule. Also watch adoption of BotBase among enterprise customers and whether the verification process tightens access for previously allowed verified bots.