Gregory's Server@aires apparently you can't crawl the web to index it unless you are google or bing because of cloudflare (the Nazi proxy) https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#fnref:1
|
20 comments
 @aires it not just "a proxy" though, its the proxy that publically regretted ending their business dealing with imprisoned american nazi weev and his nazi organization stormfront  @ionizedgirl @aires "OH FOR FUCK'S SAKE YOU'VE GOTTA BE SHITTING ME" is a phrase I just said out loud while reading your toot :o...  @ionizedgirl @aires Well, er, you can crawl most of CF if you register your bot with them through their verified bot program. Sites that opt into more advanced protection levels may still be inaccessible, and then there’s all the JS apps masquerading as websites that need a headless browser.  @ionizedgirl @aires As someone running a small web hosting company, yes, you can. There are at least three organizations aggressively crawling the web to build an index: TikTok (Bytedance), Amazon, and {not-sure-yet}. They are ignoring rate limits imposed by robots.txt and generating ridiculous server loads. [Except for Bytedance, because after they masked their user agent I've blocked every damn data centre they're running on with a firewall.] Amazon's crawler is pathologically bad, too. @alan @ionizedgirl @aires Are they actually building an index, or are they scraping content for AI training datasets? Given they seem to ignore sitemaps, I’m inclined to think the latter. @liampomfret @ionizedgirl @aires Hard to say but the two are not mutually exclusive, and Google is ripe for being eclipsed at this point. Why not use the data to do both? @alan @ionizedgirl @aires I would've assumed it'd be more efficient for the AI Scraper bots to be working off of a list of URLs they'd already gotten from sitemaps, rather than brute-forcing everything by simultaneously scraping and following links. @liampomfret @ionizedgirl @aires You would think. But some of them are horrible. Amazonbot gets stuck on event pages in some weird way. It gets an error, then appends a fragment of the URL to itself and tries again, from multiple IP addresses. I've had to put in rules that look for the pattern and then return a 403, otherwise it just keeps trying. If this thing was a high school coding assignment, it would fail! @alan @ionizedgirl @aires Oh, you don't have to tell me how horrible they are. At the end of last month, just one of these bots was nuking my own site at a rate of 35+ accesses a second, sustained 24/7. If you could DM me a copy of the rules that you've created for that, I'd love to pass them on to my own technical team, and see if that helps make any difference for some of the bots we hadn't yet been able to nail down.  @liampomfret @alan @ionizedgirl @aires Curious if this is an *Amazon* scraper or if it is someone running on AWS scraping. (can you distinguish between the two? Perhaps Amazon IPs vs AWS IPs) The three party nature of search (indexer needs permission from site), (site needs to want clients of indexer to see it), (user wants an index that contains sites they are interested in) is a really interesting problem. @ChuckMcManis @liampomfret @ionizedgirl @aires The user agent is a current amazonbot, but that doesn't mean someone isn't spoofing it. I've been blocking that one on a per site/URL basis, so if there's some reason for Amazon to be checking in, they can get in. @alan @ionizedgirl @aires Ironically, ChatGPT helped me write some directives for my .htaccess file that help prevent the crawling of some areas of my site by spiders.   @ionizedgirl @aires Whichever of these engines starts filtering to only give results without ad trackers and ad bloat, they will win the internet. (Assuming they’re allowed to spider. I’m not sure I understand how cloudflare prevents them?) |
@ionizedgirl Totally cool and normal that a proxy can choose which search engines get to index which websites :blobfoxfacepalm: