 Ah yes, let's ship a kernel driver that parses update files that are pushed globally simultaneously to millions of users without progressive staging, and let's write it in a memory unsafe language so it crashes if an update is malformed, and let's have no automated boot recovery mechanism to disable things after a few failed boots. What could possibly go wrong? 🤦♂️ 108 comments
 I assume you're all joking, but please, use the terms correctly! From the back of my head, "IT security" is "ensuring confidentiality, integrity AND availability", and a bricked computer only ticks - at maximum - two of those boxes.    @gsuberland @marcan @dysfun Lol I knew Steve was nuts and full of snake oil but this is the first I've heard the vacuum cleaner line.  @azonenberg @gsuberland @marcan @dysfun @marcan but you gotta understand antivirus is important and you can't wait even an hour for an update to roll out, it has to happen instantly /s 🦋 @marcan The thing that surprises me the most about this situation: How did something like this not happen waaaayyy sooner? This seems so incredibly fragile, how did it hold up for so long.  @l_prod @marcan it happened years ago to McAfee https://www.zdnet.com/article/defective-mcafee-update-causes-worldwide-meltdown-of-xp-pcs/ Guess who was CTO of McAfee at the time @marcan I have no idea what any of that means, but I'm glad you and others understand it! @javierg The language matters because segfaulting on invalid input only happens in memory unsafe languages. On a memory safe language you generally have to make a conscious decision about how to handle errors and unexpected situations. @javierg At least with a memory-safe language someone had to make an *active decision* to reliably crash (making this something solvable by policy, e.g. ban such constructs in the linter), as opposed to no decision at all (which is impossible to protect against or have processes that forbid, once you're using a memory unsafe language).  re: > At least with a memory-safe language someone had to make an *active decision* to reliably crash (making this something solvable by policy, e.g. ban such constructs in the linter), I've seen entire teams making concise active decision to break things for the sake of save their ass or the corporate reputation - if any. I agree, it's way better if the programming language has all the controls and tries their best to avoid unconscious bad decisions.  @marcan@social.treehouse.systems ... ship a kernel driver that parses update files that ...Fainted on-site, someone call an ambulance? :nkocampfiredrink:    I disagree. They are still plugged in. Intel vPro and the AMD equivalent have a KVM over IP built-in.  @marcan Imagine if someone write such kernel driver in memory-safe language. But malformed update arrives anyway. So this memory-safe driver crashes, but may be with other error code, leaving the system unbootable anyway. So, it really require complex changes, which will not happen. @koteisaev A memory-safe language would force you to make an active choice to crash, which at least gives you a chance to, you know, not do that and instead just bypass the update or fail gracefully. Yes, it is still possible to write crap code for memory-safe languages, but you generally have to do that much more on purpose. With a memory-unsafe language you just don't think about it and the default option is to crash (or worse). @marcan @koteisaev Wouldn't you get a panic by default for an out-of-bounds access in Rust? How would the end result be different? Also see Ariane 5 for an example how memory safety can fail rather spectacularly.  @uecker@mastodon.social @marcan@social.treehouse.systems @koteisaev@mastodon.online The programmer has to make sure the code does not panic by default in the kernel. @uecker @koteisaev It is trivial to set up a Rust build to make panic a compile time error, forcing you to use primitives that don't do that and to handle the errors gracefully, such as `.get()` instead of indexing with []. You cannot do that in a memory unsafe language, fundamentally. @marcan @koteisaev You can not do what? Forcing people to use at instead of [] in C++? Would certainly be possible. But the discussion what "you could do" misses the point completely. What would happen in reality in a poor code base written on limited budget? Most likely would simply panic, or? @marcan @koteisaev The argument that a memory safe language would have prevented the problem is incorrect, because terminating the program (kernel) on invalid operation *is* something that could also happen in a memory safe language and seems even the default in Rust for many things. Whether you could have done it differently (certainly!) and whether Rust makes this easier or not is a different discussion. @uecker @koteisaev The argument is that using a memory safe language would be a *requirement* to be *able* to avoid this class of problems, as evidenced by decades of memory safety bugs. Yes you can write crap code in any language, but it's plainly obvious to everyone who isn't in denial about the state of software engineering that approximately nobody can write correct and memory-safe complex code in memory-unsafe languages. @marcan I agree that memory-safe languages are necessary. And many others here would agree on this. @marcan @koteisaev . My preferred solution is to use a subset of C and compile to eBPF which is then verified at run-time. @marcan Here broken piece of input comes, and your memory safe code dies with exit code 9000. I mean, this whole situaiton when faulty driver can cause crash whole system instead of markng this driver as faulty and to not not use it at next boot, and if necessary, reboot with new settings, or use special fallback driver whose purpose would be to report about problem on next reboot, and then it would be quite short outage and systems would be alive after few reboots, read - few minutes max.  @marcan but they're all so clever - I simply cannot comprehend how it could all have gone so wrong. Give them some more money.  @marcan  @marcan somewhere in Crowdstrike's bugtracker, 3 bugs titled "move ruleset eval out of process", "always canary dynamic update files" and "add fuzzing to the update files parser", filed by SREs 5+ years ago, marked as P4, get resurrected and set as P0. (Remember go/outage-2013 ? :p) @delroth @marcan I wonder if Microsoft are again scowling at third-parties making their system "unreliable" and weighing up auto-disabling additional kernel drivers after repeated failed boots again. (Consumer Windows does so much attempted autorepair I'm kind of surprised it *can't* dig itself out of this hole if safe mode works. But maybe "trusted boot" and disk encryption built atop that screws all this up in an enterprise world.) @LionsPhil @marcan they issue the signing certs, they can set any policy they want - but they don't. @LionsPhil safe mode does fix it, but I think you can only enter safe mode using the Bitlocker recovery key  @marcan isn't that the definition of av/endpoint security software? Other than the non memory safe language bit, it has to work that way, with the incentives all around keeping it that way. (I'm still amused (?) that ms gets to sell security for software it sells, separately, by monthly subscription) @marcan there is a process as crowdstrike has explained: just boot into safemode and delete the driver file  @tamtararam @marcan alternatively, just reboot the machine again and again until the driver update finishes before the BSOD happens. @marcan ppl will be like "haha windows bad" but they have not seen the shit companies install on enterprise servers. you dont even need ebpf nor kernel modules to hang a system. just setup fanotify to block all file open requests and have the process that is supposed to approve those die and stop responding to those. and there you go: McAffee for Linux achieved.  @marcan they _could_ have done all of what you said, but who cares about boring shit like that? Certainly not investors or share-holders, that's for sure!   @marcan And one that cannot be changed outside of our company because it isn't FOSS! What can POSSIBLY go wrong? 😂   @etchedpixels Current rumors is it was broken in a post-processing step, so it might have been signed internally after that, they just never tested the actual final blob that was distributed.  @marcan and lets make sure FBI cybercrimes folks work at the provider of this service, and of course make sure this critical infrastructure defense service is also: <checks notes> front and center in many of the most significant cold war 2.0 nation state hacking allegations on the planet lmaaaaaooo now THAT is what I call safety and security. Its like riding on top of a fully armed drone fighter with your core systems. I wonder if this makes you more or less a target? lmaaaao  @marcan is it bad that part of me wonders, like
what if it had been even worse what if the systems were just completely unrecoverable   @marcan Well it's a nice stunt. I mean nobody would ever use "endpoint security" software on an important system. That would be ridiculous and a clear breach of, for example, the contract rules of Crowdstrike.  @marcan rust in kernel wasn't available when this thing was written. And you just don't go rewrite stuff for the fun of it. Not to defend them for whatever errors happened in the qa proces, but hindsight 20/20 here. |
Gregory's Server
@marcan they are about security, not reliability