Email or username:

Password:

Forgot your password?
Daniël Franke 🏳️‍🌈

Hey fellow #Linux users, despite the #CrowdStrike problem only affecting #Windows, this is not a windows problem.

This is an "automatic update that got forced onto everyone with insufficient testing while also having way too many permissions" problem.

If you think big corps wouldn't run something similar on Linux, I have a an NFT of a bridge to sell you.

59 comments
xbezdick

@ljrk @ainmosni that is microsoft/uefi/bios issue - not fedora

Eva Winterschön

@ainmosni do you really think that there are no orgs using #Crowdstrike Falcon and other tools on their fleets of developer machines running #Linux?

there are many. and that kernel #rootkit module watches EVERYTHING. sometimes it also locks up the entire machine from resource abuse.

Chris Adams

@ainmosni exactly. Software of this class needs to be seen as giving the vendor unreviewed root on every system you own. If your CIO isn’t comfortable with it in those terms, a different model is needed.

Daniël Franke 🏳️‍🌈

@JdeBP Yeah not surprising at all. This kind of stuff is all too common in certain companies.

Adam Honse

@ainmosni Not a Linux problem, but absolutely a failure of the proprietary and software as a service models. If this were an open, in-tree Linux driver, it would likely go through a much more robust review and test cycle as well as not be simultaneously pushed to every Linux system in existence using it. It would be filtered through multiple distros' testing and release processes which would make discovery of the bug much more likely.

Daniël Franke 🏳️‍🌈

@CalcProgrammer1
If this was part of the OS, sure, but this is third party stuff that the corps installed themselves. Again, they would do exactly the same on Linux, and as replies to my post show, CrowdStrike is common on Linux as well.

Adam Honse

@ainmosni I agree, the root issue here is giving corporations permission to install proprietary code into your kernel that you can't review or fix should it go wrong.

Joe

@ainmosni I don't think it's totally a "Modern Microsoft problem"

But I think there's a case that at least partially the fault of late 90s/early 2000s Microsoft, who were more than happy enough to ignore security until it became too big a problem to ignore, creating the environment for a swathe of slightly dodgy AV vendors to grow into?

Claire

@ainmosni Cant get hit by dodgy updates if you keep forgetting to update 🧠

The DJ Mr P

@ainmosni I spent 40 years in the IT industry including 10 as an industry analyst. I'm well versed in modern platforms and CI/CD. Yet, this suprises me. Given how widespread this is, it seems that cursory testing on the part of CrowdStrike, it's customers, and Microsoft should have caught this.
Much of tbe blame is the fire and forget mentality of cloud services. Customers of these services need to test before rolling out anything new and not rely on the vendors for testing

Daniël Franke 🏳️‍🌈

@DJMRP

I am both amazed and not-surprised at all that this happened.

At this scale, they should have had a pretty advanced multi-tier, multi-tier rollout system.

But instead it seems they committed the YOLO-est of all YOLO deployments.

Felix :thisisfine: Eckhofer

@DJMRP @ainmosni Don't think that is possible with any EDR product. One of the selling points is near-realtime protection against emerging threats and there are dozens of updates every day.

Not saying this is great, just pointing out that "test every update" is not feasible for 99.9% of orgs.

yakkoj 🦊

@ainmosni take off; they're too busy hammering us with useless tenable scans and misinterpreting the results ;o)

Daniël Franke 🏳️‍🌈

@amici @catsalad

Hell if I know, I just like how much more scammy it sounds than "I have a bridge to sell you".

:karkat_glare: ᘛ⁐̤ᕐᐷ :karkat_glare:

@ainmosni@berlin.social this is not an "corpo" thing, the big problem here is that, this shouldn't happen, this means that all the inside infrastructure is not working, and their shadow security thing is not doing well, a company having kernel level access to any computer with your software is not a good security measure, and not having a comprobation pipeline only makes it worst.

if you want to see how linux and companies can work together by doing an open solution. that has all the comprobationes needed to make it secure. just see SUSE, a company making profit by making secure systems, fast, and all the things without this kind of security problems, i already mentioned this case: https://securelist.com/trng-2023/ on my toots, where kaspesky got hacked getting root access to all the company memebers iphone's. something has to change, neither microsoft making a better software or companies reliying on better companies to mantain their infrastructure software. but this is not a "anti-corpo pro linux thingy". this is a serious problem in how microsoft handle their software.

@ainmosni@berlin.social this is not an "corpo" thing, the big problem here is that, this shouldn't happen, this means that all the inside infrastructure is not working, and their shadow security thing is not doing well, a company having kernel level access to any computer with your software is not a good security measure, and not having a comprobation pipeline only makes it worst.

if you want to see how linux and companies can work together by doing an open solution. that has all the comprobationes...

COD

@ainmosni True, but having my personal PC on Linux saved my ass this morning as I had a 830 AM new client pitch/demo, and came into my home office to a BSOD on my work laptop. I was able to log into Office 365 and Teams from Linux and do the meeting with no issues.

Dawid Rejowski

@ainmosni

Problem itself is certainly not the fault of Windows, but I think such a great stoppage in the critical operation is.

On Linux rollback using Btrfs snapshots is something common. You can uninstall faulty program, rollback to older version and pin it to prevent updating without even booting the system from the disk. There is also a world of immutable distros, which propably would just boot to previous (pre-update) slot and show notification.

Meanwhile entering Windows recovery mode for a temporary fix looks like that: 101010.pl/@didek/1128127311371

@ainmosni

Problem itself is certainly not the fault of Windows, but I think such a great stoppage in the critical operation is.

On Linux rollback using Btrfs snapshots is something common. You can uninstall faulty program, rollback to older version and pin it to prevent updating without even booting the system from the disk. There is also a world of immutable distros, which propably would just boot to previous (pre-update) slot and show notification.

Daniël Franke 🏳️‍🌈

@didek And yet, if you give an incompetent corp root/kernel access, all those things won't help depending on the error.

Sure, this exact problem would be easy to fix, but what if it caused data loss? The OS is not worth much if all the user data is gone.

I really prefer linux over windows, but this problem wasn't caused by windows.

Dawid Rejowski

@ainmosni

I see we both agree. Again, I also think the problem is not Windows fault at all.

Just wanted to point out some of Windows related issues showing their relevance alongside all of this.

Magnus Ahltorp

@didek @ainmosni Windows installations are on average more susceptible to people selling bad software. This due to both causation and correlation.

Joe | 🇵🇸 | 🧑‍💻

@ainmosni For data loss, the solution is easy: "backup before update", which always should be done especially with seriously needed devices.

DELETED

@ainmosni This post for some reason reminds me of a scene in the movie Whatever Happened to Baby Jane where Bette Davis exclaims "But you are, Blanche! You are in that chair!"

Charles Randall

@ainmosni truth. I already got screwed on ubuntu 22 lts by updated nvidia drivers that would crash randomly, freezing my display and making it impossible to recover cleanly.

Lizzie

@ainmosni@berlin.social I mean if anyone uses linux as their main, they know how kernel updates can break the entire OSes ability to do certain things. This is an automatic, forced update, to something that has core control of so much stuff inside of the operating system of a lot of critical infrastructure as a security measure. Sure you can leave your headless Debian server not updated for years and it'll be just fine, but with this software, you dont have that choice, on windows or linux or macos or anything. It just does it. And you have no say. It could have easily affected linux computers like this instead of windows. I mean yeah you could have disabled automatic updates, but you'd fail any security audit that came your way.

@ainmosni@berlin.social I mean if anyone uses linux as their main, they know how kernel updates can break the entire OSes ability to do certain things. This is an automatic, forced update, to something that has core control of so much stuff inside of the operating system of a lot of critical infrastructure as a security measure. Sure you can leave your headless Debian server not updated for years and it'll be just fine, but with this software, you dont have that choice, on windows or linux or macos...

AppSec stof

@ainmosni the way tooany permissions problem you mentioned - is a Microsoft problem

It's the way Windows designed it to work, to promote Defender and make it unnecessary more difficult for commercial products, and mostly impossible for open source - to tackle the functionality of Defender on Windows

Perspective of shared responsibility, of course - just to be clear, the permissions problem is not really a choice of CrowdStrike

Daniël Franke 🏳️‍🌈

@stoff

Note that I'm saying this as a big fan of linux with decades of experience.

Run anything on linux as root, and it has too many permissions. This is not a windows problem.

Sure, there's SELinux, but people find that "too difficult", so many turn it off.

I'm all for smack talking windows, but in this case, the problem is CrowdStrike.

Magnus Ahltorp

@ainmosni @stoff Yes, but people making Linux decisions generally have a higher degree of technical knowledge than people making Windows decisions. Many more Windows decisions are made by clueless managers. That happens with Linux as well, but to a lesser degree.

John Carlsen 4 Harris&Walz🇺🇸

@ainmosni

You're completely right.

The problem is with the model of automatic and forced updates, where users have no process of "acceptance testing" (and no practical mechanism to roll back if a version fails the test).

Unfortunately, I see it creeping into the world of Linux users, too.

For example, Ubuntu seemed pretty good about 10 years ago when I started using it, but it migrated to a model of forced updates (also switching to the snap package manager, which also brings with it some other problems). (I ran the same version of Ubuntu for 10 years before switching to LMDE6 this year.)

@ainmosni

You're completely right.

The problem is with the model of automatic and forced updates, where users have no process of "acceptance testing" (and no practical mechanism to roll back if a version fails the test).

Unfortunately, I see it creeping into the world of Linux users, too.

For example, Ubuntu seemed pretty good about 10 years ago when I started using it, but it migrated to a model of forced updates (also switching to the snap package manager, which also brings with it some other problems)....

Kinene

@johnlogic @ainmosni Switched from Ubuntu to Mint, and now using antX, since all my computers are old. Yeah, if I want an update, I will use the terminal.

ticho

@ainmosni I've had this exact thing happen with CrowdStrike Falcon on a couple of Linux servers last year, where a buggy CS update caused their kernel module to randomly corrupt kernel memory, resulting in occasional crashes.

So yes, not an OS problem, but a problem with this particular class of security software that has its tendrils all along the spine of an operating system.

James Mitchell

@ainmosni it's not like Crowdstrike hasn't ruined a couple of my weeks as a guy who is only responsible for supporting Linux installs

🍥SarahBurnout🍥

@ainmosni similar things have happened.

such as when debian pushed an update of their zfs but as the 2.x version on 4.x kernel that can only work with 5.x kernel, which trashed zfs filesystems.

JaxxAI

@ainmosni it partly is a Windows problem. The kernel shouldn't crap itself at boot when it encounters a corrupt unsigned binary excuse for a third party sys file.

Reki

@JaxxAI @ainmosni Exactly, it's round Robin blame. Not enough software diversity, putting all eggs in one basket, giving "keys" to your infrastructure by installing that root kit with automated self update mechanism. Finally windows which cannot detect misbehaving module decides to stay dead. #WCGW

Ikeruga

@ainmosni Also lots of people conveniently forgetting the recent xz fiasco >.>

Wilfried Klaebe

@ikeruga Which was not a fiasco because the source was open and someone took the time to look into it.

@ainmosni

yes, it's me, liza 🇵🇷 🦛 🦦

@ainmosni Ubuntu and their now cultish desire to snap the whole OS. i have to use snap if i want to use LXC/LXD and it gives me a lot of pause using it extensively because i can't control the auto-update of snaps in the background. everytime i think i can, they push something onto the system that undoes my settings and they start auto-updating without my control. i hate it.

Daniël Franke 🏳️‍🌈

@blogdiva yeah, Ubuntu's NIH syndrome made me give up on the distro a decent time ago.

Diane 🕵

@ainmosni

It is also a too much centralization problem.

If there were fewer near monopolies in tech, it would be harder for one bug to cause global outages.

Joe | 🇵🇸 | 🧑‍💻

@ainmosni
When people say this is not likely to happen in Linux, they don't just mean "it's not possible to put faulty code in the Linux kernel as an update" or "it's not possible to force automatic updates in Linux", but they mean instead "if it happened, it won't happen in all the Linux world like Windows", so it would be almost unnoticeable, why? (continued in the next comment)

Joe | 🇵🇸 | 🧑‍💻

@ainmosni
Combine all the following to understand the point:
1. It is the failure of the proprietary and software as a service models. If this were an open, in-tree Linux driver, it would likely go through a much more robust review and test cycle as well as not be simultaneously pushed to every Linux system in existence using it. It would be filtered through multiple distros' testing and release processes which would make discovery of the bug much more likely (mentioned by @CalcProgrammer1 ).

DELETED

@ainmosni think Microsoft made a bit of a mistake canning Windows 10X which would of made recovery from this kind of thing a lot easier (atomic updates).

Joe | 🇵🇸 | 🧑‍💻

@ainmosni
2. The filter mechanism of Linux distros failed and we got the fault update (that no one knows of)? Backup before an update and the update failed, we can still roll back with BTRFS or roll back to the last bootable image when using an immutable distro (Mentioned by @didek ).

(continued in the next comment)

Joe | 🇵🇸 | 🧑‍💻

@ainmosni
Windows does not have all this, all it has is, a non-snapshot and non-immutable OS model that's relying on forced automatic updates that are sent directly to all Windows machines around the globe without the different distros review and testing mechanism that usually their updates goes through.

So It's not just a technical issue, it's also a philosophical issue in Windows.

Boyd Stephen Smith Jr.

@ainmosni My experience is the opposite. In any org that I was allowed a company-owned and issued Linux system, we were exempt from the requirement to have the enterprise remote install/control BS. (Only two orgs allowed this.)

That said, it's certainly _possible_ to have similar procedures and process around a Linux install base -- and also similarly ill-advised.

Personne

@ainmosni
....except, the fix would be scalable and automated, from a compartmentalised part of the system, and more importantly: immediately reversible. Potato's, tomatoes.

Estarriol, Cat owned Dragon

@ainmosni crowdstrike stated it was only rolled out to Windows, not apple or linux.

So could easily have been all 3....

SpaceLifeForm

@ainmosni

The bad actors could try, but it really is a Windows design problem.

It should not automagically install new drivers.

No other OS does this.

#Linux #CrowdStrike #Windows

Go Up