@nixCraft we blocked them at #nikonians some time ago since they are scraping like mad.

@nixCraft got the same on @osm_fr discourse forum.

They are now blacklisted, with others.

@nixCraft Ah, so not only do they gather fuel by the truckload for their plagiarism bot, but they also actively harm the source of said fuel in the process.

How pleasantly disgusting.

@nixCraft Google did this back to one forum I was a part of years ago - and their stubborn refusal to obey robots.txt's crawl-delay directive still angers me to this day.

It's not just new crawlers that do this. Even the big old search engines have done it for ages.

@nixCraft I wrote my master's thesis on data poisoning LLMs, especially as a self-defense mechanism against bot scraping.

The poisoning tool is pretty basic and shitty for now, I am aiming to release a better version in the summer

@nixCraft if these parasites had any shame they wouldn't be stealing others' work for their AI grift in the first place.

@nixCraft Same on several of the sites I manage. Blocking everywhere.

@nixCraft I know some Claude lovers who are going to be very conflicted about this.

@nixCraft Interesting. I just decided that enough was enough and am since yesterday redirecting (301 permanent) ClaudeBot to large files filled with random bytes elsewhere on the web. 🤣

Didn’t know it was scraping for AI. But I’m sure the “info” they get out of that will be useful to them. 🤭

@nixCraft maybe we should start prefacing ai companies as parasitic when talking about them on reddit et al? Eg ‘parasitic ai company anthropic bla bla bla’

@nixCraft Blocking AI Scraper Bots
https://chriscoyier.net/2023/09/19/blocking-ai-scraper-bots/
By @chriscoyier

Blocking bots
https://ethanmarcotte.com/wrote/blockin-bots/
By @beep

@nixCraft had to mail the press@ address to stop them re-checking every few seconds if I'd changed my mind about blocking them for effectively DoSing my site. Tech bros in a hurry...

@nixCraft fuck ai fuck ai fuck ai

@nixCraft For anyone's reference, I've checked my logs and see user agents "ClaudeBot" and "claudebot" from below IPs, since December 2023 (block at your own risk, don't know what else they may be used for).

PS: they're all AWS IPs.

3.84.110.120
13.59.136.170
18.210.24.192
54.198.157.15
54.81.157.133

@nixCraft This explains a lot -- several sites we manage have been hit by extremely unfriendly ClaudeBot scrapes in the past week. We've been putting in rate limiters that help -- but I like the idea of poisoning 😀

@nixCraft it’s a shame you can’t return a random block of text instead of the page content.

@nixCraft cant wait for there to be a new spinoff of hoarders called "digital horders" lmao. this is the internet version of living off your mom at 30

@nixCraft I'm not so sure I'd call it bad behavior, as far as I can tell their bot respects robots.txt and looking at the one for the LM forums they don't have any crawl delay set or any restrictions on indexing so any new large crawler that finds the site for the first time would likely have the same effect, as anything indexing the site that hasn't before will just continue branching without any delay, google will nuke you like this too if you put a big dataset up and they index it w/o delay

@nixCraft
Yes, saw them as well on my end. Tried contacting them and not much of a response yet. We should feed them digital rubbish, so their products will output 🤡💩
@securingdev

@nixCraft Well done to the @linuxmint team for finding and blocking this Ai scraper 👍

@nixCraft Dang, I actually somewhat enjoyed using Claude, for all its limitations.