 Honestly, we are currently out of ideas on how to restore access to Codeberg.org. We are fighting with extreme traffic and high load for several hours now, we have done the typical procedure to identify and block misbehaving AI crawlers. However, we are currently having a hard time figuring out details about the ongoing high traffic situation. 39 comments
@Codeberg hi, I appreciate you guys, your initiative and your effort in this. I'm not bothered with a bit of downtime as I can work offline. Good luck on your issues. I doubt I have useful knowledge in this area that you don't or I would've tried to help.  We were not able to identify the actor who is causing the high load on our systems. We have made the hard decision to temporarily shut off access to a certain project to keep Codeberg available for everyone else. As soon as we allow web access to a certain project, our system resources are used up within seconds.   @Codeberg i had somewhat similar situation this week on my forgejo instance and the only idea I had was to 1. Make one of my public repos private (it did help, but just a little) But this solution is suboptimal in your case, as you care about indexing  @Codeberg I do not know the exact architecture that you have setup in Codeberg and you may have set it up already, but what about reverse proxies that introduce rate limiting based on a window (e.g. https://www.haproxy.com/blog/four-examples-of-haproxy-rate-limiting ). You will need to check which software makes most sense to you here. That could address the scenario you have that a specific repository is affected. Of course, all this does not address all possibly sources of Ddos attacks.  @Codeberg Thank you for your hard work for us, your community members. I hope you will find out what exactly happened here! @Codeberg since you don't use a competent service for freaking 2 day to have better insight on who is the culprit and go back to what you have now after 2 freaking days … Continue to have this kind of issue without knowing the culprit …  @Codeberg good luck dealing with this. Can you say which repo was being requested so much? @Codeberg thanks! that does sound like it could have been part of some poorly designed automated software update process? @edsu Unlikely. It was massive and distributed, and hammering so much that our systems went down as quickly as within one second of re-allowing the access. It calmed down now, though. It seemed to be mostly related to some web operations, so we still think it was crawling and only hammering this massive repo by coincidence.  @Codeberg  @Codeberg the recent pattern of frequent attacks on neutral non-profit sites providing services for public benefit is disgusting, be it accidental or not.   @Codeberg do you want an introduction to the Amazon people at Fastly ? They can probably help you for free through their Fast Forward program, and have tools to fight this |
Gregory's Server
@Codeberg 🥲
Well, I will keep my development offline and upload it when its okay, just hope you get to fix it, if any help just ask it, O don't have much knowledge but I do have will :)