September 15.

That's the deadline Cloudflare has drawn for AI companies across the industry. Starting that day, Cloudflare's default settings will block all "mixed-use" crawlers from accessing any ad-served pages -- unless website owners manually adjust settings, or AI companies clearly separate their search crawlers from AI training/Agent crawlers.

Bot Traffic Has Surpassed Humans, the Content Ecosystem Is Unsustainable

Cloudflare CEO Matthew Prince offered a key statistic in the announcement: over 50% of internet traffic is already non-human, and this inflection point arrived a year ahead of expectations. Over 50% of AI crawler traffic is spent re-scraping pages that haven't been updated at all -- draining publishers' bandwidth and server resources while taking creators' content to train models, with zero compensation in return.

Prince was direct: "Now that the majority of internet traffic is non-human, we must move faster for a sustainable ecosystem to emerge."

Cloudflare's new policy has three main layers. First is default blocking of mixed crawlers: crawlers that don't clearly separate search use from AI training/Agent use will be blocked by default on Cloudflare free plans and new sites after September 15. Second is the Pay Per Use mechanism: upgraded from last year's Pay Per Crawl (pay per crawl) to pay-by-value -- not paying every time an AI crawler hits a page, but paying when AI companies create value from publishers' content. Third is partner pilots: already partnered with Ceramic.ai and You.com, where publishers can opt in and get paid when their content appears in AI search results or is cited by AI assistants.

Cloudflare also criticized (without naming) the "world's largest search engine" (obviously Google): it can access roughly 2x more information than other AI companies because the search giant makes it hard for website owners to maintain search discoverability while opting out of AI training. While Google offers Google Extended for opting out of AI training, the core Googlebot serves both search and AI Overviews/AI Mode simultaneously, making it what Cloudflare defines as a "mixed-use crawler."

AI Content Payment Moves from "Moral Appeal" to "Technical Enforcement"

For the past two years, the debate over whether AI companies should pay for training data has raged on, but mostly stayed at the level of rhetoric and lawsuits. The New York Times suing OpenAI, multiple publishers signing open letters, the EU AI Act requiring training data disclosure -- these moves either progress slowly or carry enormous enforcement costs.

What's different this time is that Cloudflare isn't appealing at the legislative level or fighting in court; it's making the cut directly at the network infrastructure layer. Cloudflare carries a significant share of global website traffic, and changing its default settings will directly impact AI companies' ability to access content. This transforms "you should pay" from a moral appeal into "don't pay and you don't get it" technical enforcement.

This isn't Cloudflare's first move on AI content copyright. In 2024 it launched tools to counter AI crawlers; in 2025 it launched the Pay Per Crawl marketplace. This new policy is a continuation of that path: giving website owners more control, giving AI companies a clear compliance path, and establishing an actionable technical mechanism for content payment.

Industry chain reactions have already begun. The same week, US Senator Elizabeth Warren and Representative Mary Gay Scanlon introduced new legislation banning AI companies from selling users' health and location data to data brokers; music platform Tidal announced it would stop paying royalties on 100% AI-generated music starting July 15, though it won't outright ban it -- just label it. From network infrastructure to legislation to content platforms, the regulatory framework around AI data and copyright is rapidly taking shape.

The September 15 deadline will force AI companies to choose: either separate search crawlers from training/Agent crawlers and go the paid partnership route, or face the risk of being blocked by default across a vast number of websites. The era of freely scraping the entire web to train models may genuinely be coming to an end.