Reading all these articles about the high traffic from AI bots, here is my experience from a small German hoster: It's a huge problem, and it is getting worse very fast. There are bots like claudebot from Antrophic which are so aggressive that they block whole systems, outranking actual attacks for example against WordPress. And they change their AWS cloud IPs so often that blocking doesn't help for long and they also ignore robots.txt. Our on-call is regularly alerted for such issues.
@leah I've been supporting a small company with their website issues - their dev has been complaining about the database server for a long time... turns out ~80% of their traffic was Claude, and the remaining ~20% was Facebook's bot. After blocking these two based on their user agent, the CPU usage on the system has gone from "pinned at 100%" to basically idle, and response times have fallen back into the acceptable range (previously there were regular spikes to many minutes)
@leah Been seeing this issue grow too.
@leah Not to mention the attitude of "we ignore robots.txt, sue us if you want" and "Without using copyrighted content for free our business model doesnt work" parasites at its finest.