Gregory's ServerI would go so far to say that for small sites 30-50% of traffic is now artificial bot traffic. 20-75% for bigger sites with much content. It's a huge waste of resources they outsource to all of us in terms of money and environmental damage.
And this is part of the much bigger footprint all this AI shit has beside the irresponsible big amounts of resources they waste officially.
|
Top-level
 32 comments
  @obviousdwest @morihofi @tsia_ there are a lot of providers offering such services and I would prefer to fix the source of the peoblem and not the symptoms. @leah @obviousdwest @tsia_ I just discovered in my Matomo analytics that some traffic came from clients with a referer to https://www.xtraffic.plus. Some sort of traffic generator for SEO optimisation (How even should that work) @morihofi @leah @obviousdwest (knowing some of those SEO people I wouldn’t be surprised if it didn’t work at all)   Maybe: couldn't you reduce the number of hits per minute and block the IP for a certain period of time? In WP, for example, you can do this with Wordfence @werawelt @morihofi @leah nah, typical visitors do bursts (page, CSS, fonts, images, js), so you’d hurt them too much .oO(maybe one could just block all “cloud” services, since they don’t list their customers’ actual ranges in WHOIS and change them too often anyway… if enough people did this, maybe we could get good IP subnet to customer mapping in WHOIS…) @morihofi @leah >50% of my traffic has been non-human for a long time, though that has included search engine spiders etc. I have a target of keeping them to < 50%. See top line in table. https://www.earth.org.uk/note-on-site-technicals.html#Stats The AI bots are a huge slice added to that. In my opinion we should strongly regulate the AI stuff and just forbid most of it, like it's forbidden to run a very dirty coal plant too. For everything else we do it a little like in the pharmaceutical sector. You have to prove that your shiny new product is better/a _real_ benefit compared to the old one. I would bet that only very few very specialized use cases would be the result. But this would require an evaluation that is not based on capitalist logic and this just won't happen.  @leah It ought to be possible to sue people who disregard robots.txt under some kind of computer-related crime law.  @leah I wonder if there wouldn’t be a market for a DDoS protection-like service that detects heavy traffic from a single IP and starts returning garbage data. Ignore robots.txt with your ML scraper, get poisoned data in your ML scraper. Seems like a fair outcome to me. @leah I work for a large software firm with huge amounts of online content - which the AI firms have publicly said they trained on. We've since indicated through our various mechanisms they are not allowed to scrape content... But they very clearly are. We just moved to g4 analytics and started removing bot traffic from our metrics. It was nearly 70% of our traffic, so your numbers are dead on. @leah And here's the thing, they're scraping and surfacing content that's our IP. They're literally stealing from us and redirecting traffic from our business.This is definitely criminal conduct, even if we don't have laws that mark it as such. @T1gerlilly @leah I had a similar experience with my Wordpress site hosted in Canada and my static site hosted in Austria @leah  @leah At minimum, it should follow the European model that says do what thou wilt until there's significant evidence of harm. We have all that and more with AI (also fossil fuels).  @leah i deployed config snippets available for inclusion blocking OpenAI, Claude and some others by useragent as they really misbehave. Pasting random stuff into search boxes etc. Just a single line in the vhost and the get http code for Removed. Ok that went viral. I have two comments regarding the most frequent replies. 1. Yes, cloudflare is a short term solution, but the real solution can't be that we all need to book services at company A to protect against company B. The result would be a huge monopoly battle. 2. Stop suggesting to poison them with huge amounts of fake data. That would only further increase the waste of resources on all sites. @leah the only positive thing in that cloudflare Argument: at least the next cold war would be partially federated... I don't get why people are so harshly opposed to legislature or even say it won't work. Especially in regards to electronic systems. And it's mostly the same people claiming that a 2% subvention cut will drive hundred thousands of workplaces away. 🙄 @leah Idk seems to me like it's a war of attrition, so poisoning their data could very well work? @MayaMitKind and just hoping that something will change because some little sites started poisoning? That's a bold bet. That's like having no car and hoping this will change the car industry. @rappet that's different depending on the case and bot. See traffic as number of requests if it helps. |
@leah For me, an individual who runs a small private website, this is true. 50% - 60% of the traffic on my website comes from bots in an average month.