Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare is setting a firm deadline for AI companies to distinguish between web crawlers used for traditional search and those used for AI training or agent-based services. The company said that, beginning September 15, 2026, its default settings will automatically block “mixed-use” crawlers from pages that display advertising.

In practice, that means bots that combine search indexing with AI agent functions or model training will no longer be able to access those ad-supported pages by default, unless a site owner chooses to change the setting manually. According to Cloudflare, the policy will apply to new customers, newly created sites from existing customers, and all current users on its free tier.

Why the change matters for AI companies

The update could reshape how AI developers gather online material, both for training models and for powering services that rely on live web access. Cloudflare’s position is that many publishers still want visibility in search engines and may also want to participate in AI-driven experiences, but they do not want their work absorbed into AI systems without meaningful control or compensation.

Cloudflare argues that this issue is especially important because some major platforms may gain broader access to content than standalone AI firms. The company specifically points to the world’s largest search engine, saying it effectively has access to roughly twice as much information as other AI companies because website operators often struggle to remain discoverable in search without also exposing content for AI-related uses.

Google has previously rejected that framing. The company has highlighted Google Extended, a tool that allows publishers to prevent their content from being used in AI training and in products such as Gemini Apps and Vertex API, while still remaining eligible for inclusion in Google Search. At the same time, Google’s main crawler, Googlebot, continues to crawl for Search and also supports search-related AI experiences, including AI Overviews and AI Mode.

Cloudflare’s broader argument

Cloudflare says the new policy reflects a larger shift in how the internet operates. The company recently noted that automated traffic has, for the first time, surpassed human traffic online, a milestone it had not expected to arrive until next year.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” said Cloudflare co-founder and CEO Matthew Prince.

Prince said Cloudflare wants to create clearer rules for bots and stronger protections for site owners, while still supporting AI companies that are transparent about what their crawlers do.

“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training,” Prince said.

More control for publishers

The company has spent the past few years building products aimed at helping publishers manage how AI systems interact with their sites. Although Cloudflare also offers tools that support customers building their own AI applications, it has increasingly emphasized publisher protection as AI scraping expands.

Among those efforts is Pay Per Crawl, a marketplace-style product designed to let websites charge AI bots for scraping their pages. Cloudflare now says that concept is evolving into Pay Per Use, a model intended to charge AI companies when a publisher’s content actually generates value, rather than only when it is fetched.

Cloudflare also presents the shift as a practical way to reduce waste. The company says its data indicates that more than half of AI crawler traffic is spent repeatedly fetching pages that have not changed, consuming bandwidth and compute resources without adding new value for publishers.

Early partners for the new model

To begin putting this approach into operation, Cloudflare is working with Ceramic.ai and You.com. Under the arrangement, publishers who opt in can receive payment when their content appears in Ceramic’s AI search results or when You.com accesses premium material.

Cloudflare says the framework is not limited to those two companies. Other AI providers will be able to adapt the model to fit their own products and workflows, suggesting the company is aiming to establish a broader template for how content access, AI usage, and publisher compensation could work together going forward.