GenAI crawler problem highlights a bigger issue: the cloud bandwidth nightmare
It comes as no surprise that online scraper bots are scouring the web looking for content to train generative AI (genAI) models, causing massive bandwidth charges to enterprises who told them not to crawl their sites. What is surprising is the world of large language model (LLM) makers’ use of unattributed browsers and other means of escaping…