Client call in panic mode. CRM software techs logged...

Client call in panic mode. CRM software techs logged in and filled up the disk.
With 50GB free out of 100GB, I wondered how that's possible. They wanted to copy the entire directory to another for "safety".
Explained I could backup and restore if needed in seconds. But no, they had done as they said. Result? Disk filled, everything halted.
I'm out and about.
Proxmox doesn't let me modify virtual hardware through its Android app or limited mobile web page. Desktop site request doesn't help, layout's all messed up. In emergency mode, I set up an xrdp FreeBSD jail with Mate and Firefox, connected via VPN on my smartphone, accessed the client's Proxmox.
Added a virtual disk, extended the LVM, and the underlying XFS filesystem.
Total time? A blind 10 minutes.
Triumphantly, I call the client (who knew I was away from a computer) to report success.
The naive response: "Okay, but all this time just to expand a virtual disk?"

I decide it's better not to comment 😊

#TechSupport #SysAdmin #Proxmox #ClientStories #ITLife

Like 24 January at 16:08 | Open on mastodon.bsd.cafe

16 comments

Jesper Mårtenson :unverified:

@stefano Hahah, nicely done. :D Ignore the haters! :P

24 January at 16:09 | Open on mammoth.gry.st

Jim

@stefano They don't pay us for what we do, they pay us for what we know.

24 January at 16:11 | Open on sully.site

Stefano Marinelli

@sullybiker they don't pay us to solve someone else's mess 🙂

24 January at 16:15 | Open on mastodon.bsd.cafe

Jim

@stefano Sometimes that's all I do!

24 January at 16:16 | Open on sully.site

Stefano Marinelli

@sullybiker me too. But not in 10 minutes while all I have is my mobile phone.
😆

24 January at 16:18 | Open on mastodon.bsd.cafe

Jim

@stefano Be careful with that. Walk on water enough times, people will start thinking it's normal.

24 January at 16:18 | Open on sully.site

Gonzalo Nemmi :runbsd:

@sullybiker @stefano this. So much. Now you know the next time, solving the same problem will take no less than 4 hours of absolute downtime of all the systems, even those that are not involved.

If you don't do it for you, at the very least do it for them, so they can understand the underlying complexity of the issues at stake.

See .. this happens to every professional: if "you" can't make your work be respected and valued for what it is: "nobody" will.

24 January at 17:14 | Open on mastodon.sdf.org

Stefano Marinelli

@gnemmi @sullybiker I will. In general, I have a trusting relationship with my clients. They know that I give my best, and when I say something is impossible or time-consuming, they know it's true and trust me. It works with some, with others... less :-)
Additionally, interventions of this kind do not incur any costs for them (they are covered by comprehensive agreements I provide), so whether it takes me 1 minute or 10 minutes, it makes no difference to them.

24 January at 17:46 | Open on mastodon.bsd.cafe

Gonzalo Nemmi :runbsd:

@stefano @sullybiker this is pretty subjective, as it derives from personal experience, but in my case I found out that even if the client is on a particular agreement, in the end, this kind of episodes have a negative impact as they tends to diminish the quality of your craft on the eye of the client, ultimately leading them to believe that whatever it is that you do has no value, or at least, is not as valuable as they initially thought it was. Clients have an inherent tendency to undervalue

24 January at 18:02 | Open on mastodon.sdf.org

Jim

@gnemmi @stefano I had a similar issue with someone wrecking their server that I fixed out of hours. I did it because I absolutely despise going to sleep on problems. Unfortunately it created the idea I work around the clock, which I then had to undo.

24 January at 18:04 | Open on sully.site

Stefano Marinelli

@sullybiker True...

24 January at 17:44 | Open on mastodon.bsd.cafe

Jan-Philipp

@stefano Why did you add an additional disk instead of resizing the existing one?

24 January at 16:21 | Open on norden.social

Stefano Marinelli

@jpl Because this way, in an emergency, it was simpler and safer. Otherwise, I would have had to resize that disk, modify the partition table (on an active disk), and extend everything. This way, I carried out a quicker intervention, given that I was in uncomfortable conditions, without touching the original disk. Furthermore, this allows me to revert everything to more appropriate disk sizes and organization (I can't shrink XFS, but I can still work around it by adding a virtual disk, copying everything, removing what I added earlier, and reformatting the partition from the previous setup).

Expand text...

24 January at 17:54 | Open on mastodon.bsd.cafe