Crawler Crisis Begets CDN Changes

Cloudflare's July 1, 2025 announcement of its pay-per-crawl monetization system plants a flag for publishers who have, for too long already, been facing existential threat from AI and its legion of crawlers. The new offering, which launched on what CEO Matthew Prince has dubbed "Content Independence Day," represents the first infrastructure-level solution with teeth to combat the “strip-mining” of content for use by Generative AI products.   

While web traffic from AI crawlers has more than doubled in the past 18 months, the potential revenue for media owners has decreased by billions of dollars - not just due to loss of traffic, but from ad ecosystem repercussions that penalize publishers for the Invalid Traffic caused by those crawlers. Cloudflare’s micropayment framework offers to reshape (and restore) the economics for publishers in this AI era.

Understanding the Crawler Crisis

AI companies extract massive value from the open internet by harvesting data that populates responses in their chat or search interfaces, without contributing compensation to the publishers who create the source material. On the surface this can be viewed as copyright infringement, though attempts to address it as such in court have been unsuccessful. 

Larger cases are still underway where publishers are challenging the foundational training for Large Language Models, most notably The New York Times vs OpenAI. The distinction here is the corpus of content that brought AI like ChatGPT to life, not just the elimination of search traffic to new content published daily across the internet. 

Crawl-to-referral ratios paint a clear picture of the value extraction, reported on here by Bloomberg. In their findings, independent publishers experienced over 70% traffic declines from Google after AI Mode rolled out. While Google maintained a sustainable 2:1 ratio a decade ago, sending one visitor for every two pages crawled, today's landscape shows catastrophic decreases in those returns for publishers:

• Google: 6:1 ➡️ 18:1
• OpenAI: 250:1 ➡️ 1,500:1 
• Anthropic: 6,000:1 ➡️ 60,000:1 

AI Overviews will continue to grow in scale. 13.14% of all queries triggered AI Overviews in March 2025, up from 6.49% of all queries in January 2025. Informational content is the most likely to trigger AI Overviews: Semrush reports that 88.1% of queries that trigger an AI Overview are informational in nature. 

Sharpening Our Blunt Tactics 

Crawlers that bypass robots.txt “no crawl” orders have previously been routed into an “AI Labyrinth” by Cloudflare. The creation of this “trap” was in no small part due to major AI companies like Perplexity brazenly bypassing the instructions and permissions set by publishers. This behavior is characterized not by taking without asking, but taking when asked not to, which goes far beyond crawlers “not understanding certain instructions” as Google put it in their spec. I highly recommend reading the deep dive on this subject from WIRED. 

Cloudflare's new system authenticates AI crawlers and essentially paywalls them before they can access the contents of a website. Pages will be permissioned to each crawler with three variations: Allowed access, Charge (at rates set in Cloudflare), or Blocked. This will allow publishers to force commercial terms and differentiate between search indexing and content ingestion for most crawlers. 

Major publishers backing Cloudflare's initiative include BuzzFeed, Dotdash Meredith, and Salon.com.

Strategies for Human Traffic

While Cloudflare's pay-per-crawl system addresses the crawler monetization challenge, publishers still need to maximize revenue from their most valuable asset: genuine human readers. Here at Aditude, we won’t lose sight of the valuable audiences who visit our customers’ websites. They continue to be of great interest to advertisers, and the programmatic ecosystem rewards publishers who can make their visitors most addressable and measurable.

"Aditude is committed to working with our partners to properly identify all crawlers and have site monetization respond accordingly," says Erik Omlid, CTO of Aditude. "Our focus remains on helping publishers maximize the value of their human traffic while ensuring proper categorization of automated visitors."

In order to rise to the moment, audience differentiation is called for. Much how search engine crawling will be allowed while AI training bots will be charged or blocked, loyal return audiences who are high-intent readers creating high-attention opportunities should be strategically monetized. 

Customizing layouts and ad footprints by source and referrer, permissioning high value demand partners and pricing those impressions accordingly are all tactics that are established in Aditude’s playbook and well-supported by our software. Aditude has been developing on Cloudflare for many years; Aditude’s technical support extends to helping our publishers to customize their Cloudflare accounts where assistance is needed.

With major publishers rallying behind the pay-per-crawl initiative, and AI crawler traffic already surpassing unsustainable levels, the stage is set for a restructuring of how content value flows through the digital ecosystem. Ironically, users of LLMs are already billed on a usage basis, calculated in tokens. AI accounting departments already have pay-per arrangements in their DNA.

For website owners, a great reduction in internet visitors and a retreat of quality content behind paywalls is a lethal cocktail. Implementing protective measures at the CDN level and being increasingly deliberate about how readership is monetized with advertisements promises a better future for independent journalism. 

AI is well-funded by Venture Capital, and will continue to grow and transform in years to come. The question isn't whether to act, but how quickly publishers can adapt to this new reality. Aditude is here to help you make those changes today.