Instead of simply blocking unwanted AI crawlers, Cloudflare has introduced a new defense method that lures them into a maze of AI-generated content, designed to waste their time and resources.
The new feature, called "AI Labyrinth," is now available as an opt-in option for all Cloudflare customers, including those on free plans. The system activates automatically when it detects suspicious bot activity, without requiring customers to set up special rules.
"We have found that blocking malicious bots can alert the attacker that you are on to them, leading to a shift in approach, and a never-ending arms race," Cloudflare explains in its blog post. Rather than continuing this pattern, the company now uses generative AI to misdirect bots.
When the system detects unwanted crawlers, it redirects them to AI-generated pages that look real enough to entice the crawler to follow them. Since these pages aren't part of the actual website, the crawler wastes time and resources exploring fake content.
Rapid growth drives need for new defenses
According to Cloudflare, AI crawlers now generate more than 50 billion requests daily to their network - nearly 1 percent of all web requests the company handles, highlighting the growing need for such defensive measures.
The technical implementation uses Workers AI with an open-source model to create unique HTML pages on various topics. To maintain performance, the system generates and screens content for XSS vulnerabilities in advance, storing it for quick access rather than creating it in real-time.
AI Labyrinth also functions as a honeypot: since no human would deliberately explore deep into a maze of AI-generated nonsense, anyone who does can be identified as a bot with high confidence. This information feeds into Cloudflare's machine learning models, continuously improving bot detection.
Cloudflare says this is just the first step in using generative AI to defend against bots. Future versions will make the fake links harder to detect and integrate them more seamlessly into protected websites' existing structures.
