Cloudflare Overhauls AI Bot Blocking: Granular Controls for Search, Training, and Agent Crawlers

1. Introduction: Cloudflare's New Era in AI Bot Management

Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. It was a simple but blunt approach — all AI bots were treated equally regardless of their purpose. In July 2026, the company introduced a completely new approach that radically changes how AI bots are managed.

The new system allows site owners to differentiate between three categories of AI bots: Search, Training, and Agent. This change is especially important for businesses that want control over their content — who uses it and how.

2. Why Cloudflare Changed Its Approach

The reason is simple — AI bot traffic surpassed human internet traffic for the first time in June 2026. According to Cloudflare CEO Matthew Prince, this milestone was expected by late 2027, but reality exceeded expectations. This growth demonstrated that the old "one-size-fits-all" approach was no longer working.

Today, AI bots serve different purposes. Some index websites for search engines, some collect data for AI model training, and some act on behalf of users. Each of these categories requires different access permissions.

3. Three Categories: Search, Training, Agent

The main innovation of the new system is the division of bots into three categories. The Search category includes bots that index web pages for search engines, like Googlebot. The Training category includes bots that collect data for AI model training. The Agent category includes bots acting on behalf of users, like ChatGPT.

Multi-purpose crawlers like Googlebot, which both searches and collects data for training, will be treated according to whichever rule is strictest. This is Cloudflare's response to criticism that Google bundled its search and AI crawlers together.

4. How It Works in Practice

Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. The new options are available to users on the free plan as well.

Site owners can choose from three options: allow all categories, block specific categories, or completely block AI bots as in the old system. This flexibility is especially important for small and medium businesses.

5. BotBase: Enterprise-Grade Management

For enterprise customers, Cloudflare is launching BotBase — a searchable database of all known bots built right into the dashboard. It shows how each bot is classified and how it uses content, whether it just links to pages or reproduces them in full.

Cloudflare is also changing the rules for verified bots. Previously, all verified bots were allowed through automatically. Going forward, a bot's category determines whether it gets access. The "Verified" label alone won't be enough — bot operators will need to prove they identify themselves honestly.

6. What This Means for Georgian Businesses

In Georgia, where more and more businesses are moving to online platforms, Cloudflare's new controls are especially important. Georgian websites often have limited resources, and aggressive AI bot scraping can create serious server load.

For SiTech Agency clients who use Cloudflare, this change means better control over their content. We recommend that every business review their Cloudflare dashboard and configure AI bot access according to their needs.

7. Conclusion

Cloudflare's new granular control over AI bots is a significant step in the evolution of the internet ecosystem. It balances openness with content protection, which is especially important in the AI era. For Georgian businesses, this is an excellent opportunity to protect their content while still providing access to search engines.