Media Briefing: Declared ‘good bots,’ mixed-use crawlers, gray scrapers – how AI accesses publisher content

This week’s Media Briefing looks at how AI crawlers now reach publisher content, one year after Cloudflare’s default bot changes. Why the black market scraping economy still wins. Netflix strikes video licensing agreements with Penske Media digital brands including Variety and Rolling Stone. From default blocks to ‘mixed-use’ bots The AI crawling environment for publishers is fragmenting. On one side is a visible layer of declared “good bots” that go through infrastructure gatekeepers like Cloudflare, Fastly and Akamai. On the other is a much larger gray crawling economy that still sits largely outside meaningful oversight. Last week,…

This is a curated summary. The full story was reported by Digiday.