Gregory's Server@alexskunz @leah some ignore Crawl-Delay in robots.txt, too… grr… I 429 those with .htaccess.
Because, fuck Cloudflare.
|
Top-level
 2 comments
|
|
Top-level
@alexskunz @leah some ignore Crawl-Delay in robots.txt, too… grr… I 429 those with Because, fuck Cloudflare. 2 comments
|
@mirabilos I know that CF has a mixed reputation. From a small website owner's perspective, it's a great help to curb this madness.
(there's a lot more going on than just ignoring crawl delays. 3rd parties that identify themselves as GoogleBot etc. for example are super easy to block with CF — hard to do that with .htaccess)