Email or username:

Password:

Forgot your password?
All posts nixCraft's posts Post Back to profile
Top-level
Dohány Tamás
6 comments
Matt W
F4GRX Sébastien

@pixelriot @iamdtms @nixCraft @chriscoyier @beep If I were an AI company I would never use any user agent in this list.

27 April at 20:03 | Open on chaos.social
morgan

@iamdtms @nixCraft @chriscoyier @beep please don't block CCBot though, it's extremely well behaved cc @pjox

27 April at 19:12 | Open on lingo.lol
Pedro Ortiz Suarez

@fay @iamdtms @nixCraft @chriscoyier @beep We Crawl very slowly and very politely, always respecting robots.txt. We have been doing so for years, way before LLMs. Yes some companies have used our crawls for AI training, but we’re mainly a research crawl, our goal is to provide resources to researchers, archive and actually increase visibility of underrepresented parts of the web.

27 April at 20:04 | Open on mastodon.social
Pedro Ortiz Suarez

@fay @iamdtms @nixCraft @chriscoyier @beep There are also people who are starting to use our crawls in order to build indexes and alternative open web search engines, which I love, I don’t believe a handful of companies should be deciding the content that people consume on the web.

27 April at 20:07 | Open on mastodon.social
Dohány Tamás

@pjox @fay @nixCraft @chriscoyier @beep Thank you for letting me know. I'll act like this.

27 April at 20:33 | Open on mas.to
Go Up