Gregory's Server"Linux would have prevented this!" literally true because my former colleague KP Singh wrote a kernel security module that lets EDR implementations load ebpf into the kernel to monitor and act on security hooks and Crowdstrike now uses that rather than requiring its own kernel module that would otherwise absolutely have allowed this to happen, so everyone please say thank you to him
|
66 comments
 @smolwaffle @mjg59 i mean, security vendor doesn't support current LTS of major OS distribution is a smell in itself... @smolwaffle @growse ebpf doesn't entirely abstract you from kernel internals - you still need to deal with internal API and ABI changes, so there's still some work in supporting newer kernels. It just means you'll fail to work rather than taking the entire thing down with you. @mjg59 @smolwaffle fair, and arguably the important bit. When it comes to an agent that (typically) another group is foisting on a service, "working" is not a primary concern. "Not taking the box down" is much more significant:)  @james my understanding is that that affected people still using the old kernel driver (eg, if your os is too old to have the new ebpf hotness) https://blog.fefe.de/?ts=9864a262 @nilz @TonyYarusso @mjg59 @james The "user mode" mentioned as the workaround in that post is eBPF, as opposed to the old driver in "kernel mode". @mjg59 @james Probably relates to this Redhat issue - https://access.redhat.com/solutions/7068083 Apparently their latest releases still ship with 5.14, which they apparently maintain as their own branch (mainline considered it EOL back in November 2021). Do wonder WTF there rationale is for that one. @StryderNotavi @mjg59 @james That's how Redhat has always worked, they pick a kernel version and stick with it for the entire life cycle of that release, backporting features and bugfixes as needed. It isn't like a vanilla 5.14 kernel at all. EL6 was officially 100% EOL just last July, and that was 2.6.32. @astraleureka @mjg59 @james That makes sense, although it does get me that RHEL 9.0 was released May 2022 with a Kernel that had already been EOL for six months. I can get why you'd avoid upgrading subsequent releases to keep things stable for enterprise customers, but surely you'd want to start off up to date? @StryderNotavi @astraleureka @mjg59 @james A package freeze occurs before GA, and at that point in time CentOS Stream had 5.14. The term backporting is commonly used, but it makes things sound minor. RHEL engineering rebases entire subsystems in the kernel, it's why the version number can't be used for comparisons. It only marks the starting point; the RHEL 9.4 kernel is more like a 6.5/6.6 kernel today than 5.14.  @mjg59 and that is due to the open nature of Linux. Your friend couldn't write that functionality for the windows kernel. @astraleureka @mjg59 yeah, but it can push you into less robust options ;)  @mjg59 If it runs in kernel mode it can still crash the system. The issue here is code that is fed incorrect data and does not validate it and crashes. @mjg59 I you have code that runs in kernel mode (the CPU mode) and there is a fault (invalid op code, memory access, divide by 0 etc), then the whole system crashes because there is no fault handler to just crash a process. @mjg59 @jfmezei this is what happened with redhat. They apparently backported some ebpf code to their older rhel9 kernel and introduced a bug. Kernel bugs are normally a good way of crashing things. I would guess their testing didn't pick it up, but then third party crowdstrike loads it's complex ebpf ruleset and "goodnight nurse" ! We lost a bunch of servers that way. Thankfully grub made recovery vastly easier than the current windows nightmare @jhaar @mjg59 @jfmezei True, though I do think mjg’s point still stands since the crash there was really as a result of bugs in Red Hat’s backported kernel code for BPF handling, and not because of CrowdStrike’s actual BPF code. It’s just extremely unfortunate timing for people making the “eBPF would solve this” argument that this BPF handling bug surfaced recently and of all things it was CrowdStrike that managed to trigger it. :blobcatfacepalm: @alexhaydock Longer term Crowdstrike is probably doing eBPF a favor: giving it a strong workout - discovering bugs/etc. In fact, the same thing happened after Apple stopped allowing kext modules in MacOS. Crowdstrike flipped over to use the new Mac telemetry APIs and hit bugs there too, ...and then Apple fixed them. @mjg59 I think KP Singh deserves to be hailed as an honorary "Linux Chad": @mjg59 looks like crowdstrike could have dope the same for windows if I understand the basics right. https://opensource.microsoft.com/blog/2021/05/10/making-ebpf-work-on-windows/ If they are doing it for Linux and not windows, I think that may be a big enough neglect to destroy the company to lawsuits and maybe even get a couple of people in jail for the financial damage they wrought.  The minus in Company value suggests what the Assets are worth when theyre being sold. Because the Brand is burned now, so its time to be taken over by another corp….  @Netux @mjg59 I was about to say something similar, but with this url: https://github.com/microsoft/ebpf-for-windows @mjg59 Meanwhile in RedHat mailing lists, crowdstrike ebpf kernel panicking the machine: https://access.redhat.com/solutions/7068083 :D    @mjg59 And macOS killed off kernel extensions years ago. https://developer.apple.com/support/kernel-extensions/ We weren’t just lucky. @mjg59 @pluralistic eBPF is a re-implementation of concepts from dtrace to avoid CDDL and not invented here syndrome. Windows has dtrace and AMSI. CrowdStrike implemented its own kernel module using those. What you are saying is that someone demanded a higher quality implementation on Linux or shipping binary kernel extensions for Linux was too much pain for them.    @mjg59 and also, apparently MS did try to force AV vendors out of kernel drivers and through APIs, but they did not cooperate and threatened an antitrust lawsuit. I wonder if Linux could've gotten away with it if it was that big a platform 🤔 @mjg59 I find these recent takes by the Linux “Master Race” / Community extremely toxic and damaging to the community. Open Source Software is not the answer and has shared it’s own number of recent controversies (ie XC, OpenSSL) but these seem to be forgotten about pretty quickly - just because it’s open doesn’t mean it’s secure. As a software developer and user of MacOS, Windows 11 and Debian 12, I find all 3 OS’s have their place, purpose and reason to co-exist @Simbo2k6 I don't think open source solves this problem, but this *specific* problem is absolutely solved by Linux and I am not going to generalise beyond that |
@mjg59 thx kp