Ok, here’s my first IT Horror Story of the Saturday night. It's a bit long.

I was at a client, a healthcare facility, to replace some hard drives. They didn’t want to spend money, so we had to keep the current setup running, which was outdated and unreliable.
Now I can say it: they didn’t want to spend because the general manager’s goal was to give work to a company of his friends, who were already providing support on two VMs, and he wanted to hand everything over to them. The IT manager hadn’t yet understood the financial interests of these people and still believed everything was in good faith, that they really didn’t have the funds. We were holding everything together with duct tape, but it was working and stable. I had set up a "cluster" with OpenNebula and GlusterFS for storage (later replaced with MooseFS and then with Ceph), using all available hardware.

We scheduled an intervention and notified everyone to disconnect and shut down the machines by 12:30. By 13, we had completed the backups, aiming to start the intervention by 14 and get the work up and running by 15:30. The goal was to update the systems and check the disks. We shut down all the VMs, had everyone disconnect. It was lunchtime.
We updated the servers, rebooted them. One of the disks started throwing errors. GlusterFS, for some reason I never really investigated (I have my theories, which I’ll share later, but from that day forward, GlusterFS no longer exists for me), decided to overwrite both that disk and its replica with zeros. I hadn’t changed anything.

Panic – there were backups, but on a USB 2 disk (old servers, no USB 3)! I immediately stopped everything. I was almost fainting. The IT manager didn’t understand what had happened, so I explained it to him. He announced to everyone that we would cancel the rest of the intervention, restore from backup, and have the work back up by 15:30 as planned, prioritizing the most critical VMs.

The "competing" company reached out. They had powered on one of their VMs and started "doing their own interventions." Even though they had been warned not to do anything. And they complained "we" had lost some of their data. Of course, the manager and those guys went "carpe diem": they put me under accusation, saying I had undoubtedly made a mistake that led to "lost data." I wrote a technical report explaining what I had seen, noting various SSH logins from those guys during the intervention. The "history" had been erased. Of course, not by me.

They continued to harass me for a while. The last thing they asked was for me to go to a meeting "to explain in person." They tried to schedule it the day before my wedding. And they knew it.

They threatened to ask me for an unspecified (high) financial compensation for 'the lost data.' What lost data? The ones that, allegedly, the other company would have entered in the meantime.

Final result: no problem for me (I hadn’t done anything wrong), the backups were fine, they only calmed down when I proved (logs in hand) that I wasn’t the only one connected to that machine, and my witnesses (two colleagues and the IT manager) had seen all my actions, confirming I hadn’t done anything wrong.

In the end: I realized they would do it again and I left the client – even the IT manager decided to resign and change jobs. The general manager managed to install, at astronomical figures, the company he wanted to place. After two months, they got their hands on the system and broke it. They asked me for assistance, which I refused. At any price.

After a few years, I found out that the general manager ended up in jail for corruption, bribes, and for favoring his friend companies in many sectors.

That day, I celebrated.

#HorrorStory #ITHorrorStory #ITSupport #TechTroubles #SysAdmin

This window was my view from the room where I worked in the 'old' house, from 2008 to 2015.
Many of the projects I have up and running today were conceived, studied, implemented, and understood from that room, gazing out of that window. It offered me peace, relaxation, and focus, with its bucolic, green backdrop, connecting me with nature and a slower pace of life.

#FensterFreitag #WindowFriday #ThrowbackThursday #WorkSpace #NatureViews #Inspiration #Mindfulness #Tranquility #Italy

I really struggle to understand the direction of today's society.

For years, we've been saying we need to reduce emissions and consumption, removing chargers from smartphone boxes "to pollute less," then the "AI" comes along and we reopen fossil fuel power plants, increasing consumption and emissions for... well, who knows why!

We make people feel guilty for not switching to an electric car (or one with high energy efficiency), yet we fly for pure leisure, just to get "a few more likes" on YouTube.

We create increasingly efficient electronic devices, focus on the consumption and emissions of data centers (local or remote), and then stop optimizing code and dependencies because "there’s autoscaling" and "resources are cheap."

To me, these are ideological short circuits.

#Sustainability

In recent days, I've received a few messages and noticed some posts regarding my articles on the blog. The criticism is mainly about the fact that, in some cases, I don't document "every" step but assume a basic understanding of the topics discussed. For example, if the article is about "how to install Y within a FreeBSD jail," I don't document how to install FreeBSD, what jails are, or how they should be managed, etc. In some cases, I refer to previous articles, but my aim is never to create "for dummies" tutorials. I believe that self-hosting, if done without awareness, creates more problems than it solves.

I’ll probably need to publish an article specifically about this—and maybe link it to a menu at the top of the page to explain it. My approach has usually been to provide tools to understand how I solved a problem, not to hand out "ready-made solutions"—the goal is to help people understand, not to mindlessly copy without comprehension.

After all, the blog is called "IT Notes," and they are my notes, turned into articles, mostly related to direct experiences I’ve just had. It’s not called "IT Course"—those, when necessary, I create in other ways.

#ITNotes #BlogPosts #Blog

This morning I noticed a problem. The Mastodon instance actually has two Jail environments, one of which is on the VPS of the reverse proxy. Two days ago, I stopped it (it's not necessary, but serves as support and failover if the primary one is down), and it remained stopped.
Push notifications via Google FCM stopped arriving, while those via UnifiedPush were still coming through.
After a morning of debugging, I figured it out: the Jail environments on the bigger BSD Cafe VPS route all traffic via Wireguard, and there was an MTU issue.
The push notifications via FCM contain the full notification data in an encrypted form, while those via UnifiedPush simply "wake up" the app, which then fetches the notification.
In other words, the first type had large packets that were being blocked due to the MTU discrepancy.
I suspect this might have affected the functioning of the other Jails as well, so I modified all of them, and now everything should be fine.

#BSDCafe #BSDCafeServices #BSDCafeUpdates