Gregory's ServerIn the meantime, all servers had been wiped already so no way to retrieve our data.
If you are looking for a bad case of the Mondays, well, that was one.
|
8 comments
 If there is any silver lining to this, it is that we could see a few points of vulnerabilities as well as our ability to turn things around in a reasonably quick manner (here be kudos for the two heroes who manage our infra). Learning were made, and we will see in the coming weeks/months how we can implement new safegards within our resource constraints. /END @kiwix Good golly. I've been using Hetzner for years and even just recently recommended them. Sorry you've had this trouble. I'd be really curious to know what their reason was. (Lost email or no, someone or some system _must_ have a record of why they took such a draconian action.) @tjcrowdertech Yeah we've moved on now and are a bit sad about the whole thing, but it would be nice to know at some point and get closure.  @kiwix The poor communication is bad, but this is the worst. If you're terminating service, data should be held for a reasonable time, at least a month, unless it was manually inspected and deemed illegal even to possess (i.e. CSAM), to allow customer to retrieve/migrate it. Immediate deletion is a huge red flag. @kiwix Even if you don't care about customers terminated fir violation of ToS, immediate deletion for them means same could happen to any customer by technical glitch or employee error. That should not be possible in decent professional hosting. |
Luckily we have mirrors and these were not affected. We grabbed a new machine somewhere else (Scaleway ; if we name-and-shame the one we might as well name-and-greet the other) and immediately started re-importing our data to our new Master server.
All in all, it still took about 48 hours to get these 8-ish TB back online.