Gregory's Server
@liampomfret @alan @ionizedgirl @aires
Curious if this is an *Amazon* scraper or if it is someone running on AWS scraping. (can you distinguish between the two? Perhaps Amazon IPs vs AWS IPs)
The three party nature of search (indexer needs permission from site), (site needs to want clients of indexer to see it), (user wants an index that contains sites they are interested in) is a really interesting problem.
@ChuckMcManis @liampomfret @ionizedgirl @aires The user agent is a current amazonbot, but that doesn't mean someone isn't spoofing it. I've been blocking that one on a per site/URL basis, so if there's some reason for Amazon to be checking in, they can get in.