“Stealth crawler” sounds like something from a sci-fi spy story, but the term has gained momentum with recent legislation trying to crack down on these web crawlers. A bill passed in New York last month and legislation introduced in the U.S. House of Representatives in July are pushing to prohibit bots from masking their identity. The timing tracks. Bot traffic is no longer a fringe problem. Cloudflare data shows more than half of all web traffic is now bot based. And a growing share of that is crawlers pulling content off publisher sites without identifying themselves or their purpose. Publishers are left in the dark on both counts: who’s…
This is a curated summary. The full story was reported by Digiday.
Read the full story at Digiday
