Gregory's ServerSo last week (on Sunday 1 December at 00:00), our server host canceled its service without warning.
TL;DR: we do not recommend using @Hetzner_Online 's service
Everyone else: a short 🧵
 50 comments
Turns out that Hetzner has decided to cancel our account and terminate all servers. There was no warning (yes, we checked our spam folder), and nobody could be reached before Monday morning. When reached, they could not explain the reason for the cancellation: In the meantime, all servers had been wiped already so no way to retrieve our data. If you are looking for a bad case of the Mondays, well, that was one. Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server. All in all, it still took about 48 hours to get these 8-ish TB back online. If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra). Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints. /END @kiwix Good golly. I've been using Hetzner for years and even just recently recommended them. Sorry you've had this trouble. I'd be really curious to know what their reason was. (Lost email or no, someone or some system _must_ have a record of why they took such a draconian action.) @tjcrowdertech Yeah we've moved on now and are a bit sad about the whole thing, but it would be nice to know at some point and get closure.  @kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag. @kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting.    @kiwix Sorry you had to go through that. Please do keep us updated if they do provide any sort of explanation/apology/recourse. Hetzner is currently set as the first supported hosting provider for the Small Web in Domain and this makes me *very* anxious. Any thoughts/could you possibly look into this and find out what happened, @lenzgr? @SolarDavy @kiwix @lenzgr Not only that but they’re affordable. I haven’t been able to find a VPS service as reliable, with an API that works so well, and such excellent performance and I’ve been looking. But I also can’t risk all the sites we’re going to be hosting disappearing overnight without any explanation (or even with an explanation, to be fair). So this is very worrisome indeed. @aral @kiwix @lenzgr When it's fedi, the issue is usually the anti-porn rule in their TOS. In some cases, it's in good faith. For example, with the tenforward.social, they cited a lot of hardcore porn posted/boosted by the admin, on the admin's main account. In other cases, it seems like they don't really pay attention and are vulnerable to fake reports. For example, a series of small transfem instances were taken down over an anime picture of two clothed girls kissing. @aral @kiwix @lenzgr Here's a Bluesky thread on the latter example. https://bsky.app/profile/makai.chaotic.ninja/post/3kofrm3pcvc2p  @kiwix holy shit... did they finally came up with a real reason ? @kiwix @Hetzner_Online You don't seem to be the only one having issues with Hetzner. Their solid reputation is nomore. What a shame. @kiwix @Hetzner_Online hmm wonder if it’s related https://mstdn.social/@BrodieOnLinux/113621797090762767 @kiwix we also have our instance on Hetzner, I'm quite scared now...  @kiwix @Hetzner_Online **A Cautionary Tale. ** How many backups are stored only in the cloud? How many buckets are the one and only copy of your entire IP? Looking at you *Entire World* Well done on your planning and recovery btw. Great effort. 👏👏👏 @kiwix yea, they have very bad escalation procedures.. we got bitten by that, too. They send an email to a server and then made it inaccessible... so.. obv no way for us to ever see that email... I don't know when they had reputation for good (customer) service... I basically see them as a discounter these days. @kiwix @Hetzner_Online Yet they have no issue letting cybercriminals use their services  I was considering trying and moving things up to Hetzner - -it turns out there are reasons why it is so cheap: It may occasionally just wipe out your entire business infrastructure, permanently, with no one or nothing to resort for. Won't be touching this unless I hear they offered a multi-hundred-thousand dollar compensation for the error to Kwix in short notice. https://floss.social/@kiwix@mastodon.social/113622097011781641  @kiwix@mastodon.social @Hetzner_Online@social.cologne yahhhh, maybe about 9 months ago, someone falsely reported our instance for CSAM, resulting in hetzner taking it down immediately. it took a community effort and lots of twitter posts at them before they reversed course, despite our appeals  @kiwix @Hetzner_Online From what I've seen over the years is that you're not the first having that experience with Hetzner.. |
Murphy's law states that if things can go wrong, they will. Ideally in the worst possible way.
For us, that meant having our servers disconnected at 00:00 on a Sunday 1st (so likely a scheduled deprecation on their end).
Our main storage backend became entirely unreachable. For the average user that meant not being able to access the library and download files, and for us that meant not being able to connect to it and see what was wrong.