Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers

Cloudflare gives all customers new tools to control how AI bots access their websites.

Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. Now, instead of a blanket block, site owners can choose between three categories. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT.

Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot that both search and train will be treated according to whichever rule is strictest. The new options are available to users on the free plan as well.

Cloudflare is pushing bot operators to split their crawlers by purpose. Cloudflare CEO Matthew Prince has previously called out Google for bundling its search crawler with its AI crawler. In June 2026, Prince noted that bot traffic had surpassed human traffic on the internet for the first time, a milestone he originally didn't expect until late 2027.

New bot database gives enterprises more granular control

For enterprise customers, Cloudflare is also launching BotBase, a searchable database of all known bots built right into the dashboard. It shows how each bot is classified and how it uses content, whether it just links to pages or reproduces them in full.

Cloudflare is also changing the rules for verified bots. Previously, all verified bots were allowed through automatically. Going forward, a bot's category determines whether it gets access. The "Verified" label alone won't be enough anymore: Bot operators will need to prove they identify themselves honestly and don't abuse the access they're granted.