Cloudflare new policy blocks mixed-use AI crawlers from ad pages

Cloudflare will block mixed-use crawlers from pages that host ads by default starting September 15, 2026, and the change will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. The company says the default will prevent crawlers that blend search, agent use and training from crawling ad-hosting pages unless a site owner adjusts settings.

What exactly is changing?

The default behavior will block mixed-use crawlers from any pages that host ads beginning September 15, 2026, unless a site owner changes the setting. Cloudflare defines the target as crawlers that combine traditional search activity with agentic or training uses; those mixed-use bots will be blocked by default on ad pages. Cloudflare says the new defaults apply to new customers, new sites created by existing customers, and all existing free customers, so the change affects both fresh deployments and many current free users.

How will publishers be paid or protected?

Cloudflare is evolving its anti-scraping marketplace and monetization tools into a Pay Per Use model that charges AI companies when publisher content "creates value," not just when it is fetched. The company initially plans to work with two partners, Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic's AI search results or when You.com accesses a piece of their premium content. Cloudflare says other AI companies can customize this model to match how they operate.

What evidence and data is Cloudflare using to justify the change?

Cloudflare cites two concrete points as motivation: its data showing that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, and a competitive imbalance where the "world's largest search engine" has access to about "2x more information" than other AI companies because of how discoverability and usage are tied together. CEO Matthew Prince framed the shift around bot traffic, saying, "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The company also points to its existing Pay Per Crawl marketplace and other tools designed to give publishers control over bot access.

How do big search companies fit into this?

Cloudflare singled out the dominant search provider when describing access asymmetries, but the story notes that Google offers a bot called Google Extended that lets site owners opt out of having their content used for training and AI products and services, and that opting out does not affect a site's inclusion in Google Search. The flagship Googlebot continues to crawl for Search and for Search-related AI features such as AI Overviews and AI Mode.

Why it matters

The policy raises the cost and friction for AI companies that rely on broad, mixed-purpose crawling by forcing a clearer separation between search crawling and agent or training crawling, and by creating a path for publishers to be paid when content is used. That matters for model providers who may have leaned on undifferentiated crawling to build datasets or power agentic features, and for publishers seeking new revenue or reduced bandwidth and compute costs: Cloudflare reports more than half of AI crawler traffic re-fetches unchanged pages, a clear inefficiency the policy aims to cut.

What to watch

Watch whether AI companies separate their crawler fleets into distinct search and agent/training bots, and whether more AI firms adopt or adapt Cloudflare's Pay Per Use approach. Also track whether dominant search providers change how their crawlers signal training intent, since Cloudflare called out an access imbalance it quantified as "2x more information."