So, what happened was that one of the web servers (host...

So, what happened was that one of the web servers (host 2) was struggling and I decided to upgrade it like I did yesterday for (host 1): https://mastodon.social/@mastohost/109292313550726464

Yesterday it was really smooth and the upgrade to the cloud instance only caused 90 seconds of downtime.

Today, everything went wrong:
- the network interface for the private IP changed
- when changing the configuration to the new network interface and doing a network restart, it just stopped responding

Like 6 Nov 2022 at 19:22 | Open on mastodon.social

23 comments

Masto.host

So, I was on the phone debugging the problem with OVH and we were able to bring the instance back online.

This caused the service to be partially down (because host 1 was still running for anyone that was using that DNS configuration) for a little over an hour.

Really sorry about the trouble this might have caused.

Traffic grew 10x in one week and it was impossible for me to predict (or afford) to scale for this before it happened.

Hope you understand.

6 Nov 2022 at 19:33 | Open on mastodon.social

Joseph Guhlin

@mastohost All good from my little server! Thanks for fixing it so quickly. Interest in Mastodon grew so rapidly you have no time to prepare. Go well!

6 Nov 2022 at 19:34 | Open on sci.kiwi

Tanaki (pumpkin spice loaf)

@mastohost thanks for the explanation!

6 Nov 2022 at 19:35 | Open on animal.business

Ben Werdmuller

@mastohost You're doing amazing work. Thank you for this!

6 Nov 2022 at 19:35 | Open on werd.social

Brayd :mastodon: :vegan:

@mastohost don't worry, thank you for your work and your awesome service!

6 Nov 2022 at 19:35 | Open on braydmedia.de

Alexis Simon

@mastohost Thanks for the reactivity and report 👍

6 Nov 2022 at 19:35 | Open on ecoevo.social

Adnan 🦙

@mastohost You are a champ! Keep your prices at sustainable level for yourself too!

6 Nov 2022 at 19:35 | Open on 1210.nl

Wesley

@mastohost no worries, you folks are doing amazing in the circumstances. Keep up the good work and get a break when you can too!

6 Nov 2022 at 19:35 | Open on mastodon.ie

Nasher

@mastohost With the #twittermigration, things are getting ouf of hand for anyone implied in administrating servers. Thanks for the heads up !

6 Nov 2022 at 19:36 | Open on ludosphere.fr

Schmutzie 🦉

@mastohost I’m on your waiting list and completely understand. Who could have forecast such an influx? You’re being as responsive and transparent as possible, and that’s impressive :)

6 Nov 2022 at 19:37 | Open on mastodon.social

SalarRahmanian :scala: :nixos:

@mastohost fyi mentioned you in my latest newsletter https://fosstodon.org/@softinio/109297853373790610

6 Nov 2022 at 19:40 | Open on fosstodon.org

oortville

@mastohost Thank you for making it so easy and affordable to start up a small instance. Thanks for your efforts to keep up with the #TwitterMigration.

6 Nov 2022 at 19:40 | Open on oortville.masto.host

Darth ŠČ! ✞

@mastohost thank you 🙏🏼
The real truth is that I have the knowledge to host my own Linux server, but I don’t have time to take good care of it. We all appreciate you very much 🤫