Gregory's Server@bagder This is sorta what imgur does for wget as well, to "stop" scraping I guess...
(it has returned 429 "too many requests" every time I've tried, so I assumme it's an ingress rule for the user agent)
|
Top-level
@bagder This is sorta what imgur does for wget as well, to "stop" scraping I guess... (it has returned 429 "too many requests" every time I've tried, so I assumme it's an ingress rule for the user agent) 1 comment
|
@kfh @bagder It makes sense for imgur, because it's a quick way to stop clueless people from trying to spider the site. Usage tracking requires a lot more resources than a simple string comparison on a header, so if you can stop 99% of the spiders before they even hit the usage-tracking code, it's a win.