|
Top-level
 Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way. For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end). Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong. 34 comments
In the meantime, all servers had been wiped already so no way to retrieve our data. If you are looking for a bad case of the Mondays, well, that was one. Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server. All in all, it still took about 48 hours to get these 8-ish TB back online. If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra). Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints. /END @kiwix Good golly. I've been using Hetzner for years and even just recently recommended them. Sorry you've had this trouble. I'd be really curious to know what their reason was. (Lost email or no, someone or some system _must_ have a record of why they took such a draconian action.) @tjcrowdertech Yeah we've moved on now and are a bit sad about the whole thing, but it would be nice to know at some point and get closure.  @kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag. @kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.    @kiwix Sorry you had to go through that. Please do keep us updated if they do provide any sort of explanation/apology/recourse. Hetzner is currently set as the first supported hosting provider for the Small Web in Domain and this makes me *very* anxious. Any thoughts/could you possibly look into this and find out what happened, @lenzgr? @SolarDavy @kiwix @lenzgr Not only that but they’re affordable. I haven’t been able to find a VPS service as reliable, with an API that works so well, and such excellent performance and I’ve been looking. But I also can’t risk all the sites we’re going to be hosting disappearing overnight without any explanation (or even with an explanation, to be fair). So this is very worrisome indeed. @aral @kiwix @lenzgr When it's fedi, the issue is usually the anti-porn rule in their TOS. In some cases, it's in good faith. For example, with the tenforward.social, they cited a lot of hardcore porn posted/boosted by the admin, on the admin's main account. In other cases, it seems like they don't really pay attention and are vulnerable to fake reports. For example, a series of small transfem instances were taken down over an anime picture of two clothed girls kissing. @aral @kiwix @lenzgr Here's a Bluesky thread on the latter example. https://bsky.app/profile/makai.chaotic.ninja/post/3kofrm3pcvc2p  @kiwix holy shit... did they finally came up with a real reason ? |
Gregory's Server
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning.
When reached, they could not explain the reason for the cancellation:
Them: - We sent you an email.
Us : -We did not receive it, can you please resend?
Them: - We can't
Us: ಠ_ಠ