 So, if you ask me what my takeaway from the Crowdstrike issue is, I'd say: boot counting/boot assessment/automatic fallback should really be a MUST for today's systems. *Before* you invoke your first kernel you need have tracking of boot attempts and a logic for falling back to older versions automatically. It's a major shortcoming that this is not default behaviour of today's distros, in particular commercial ones. Of course systemd has supported this for a long time: 40 comments
@pid_eins Is there any distro that has implemented automatic boot assessment, as you suggested? @karlggestd @david not really true. The ones that use systemd-boot migth, but boot counting in grub is pretty useless and manual if you ask me. @pid_eins @karlggestd @david You can do it, but it is involved. But basically, using BootNext entries in EFI and only setting the fixed boot order after the system is booted is what I implemented for an immutable digital signage project.   @poleguy the way automatic boot assessment with systemd works is that on each boot we make one of three assesments: "good", "bad", "dontknow". If we make the "bad" assessment we'll count down the entry's counter (and if it ist zero we give up on it in the future). if we make the "good" assessment we'll drop the counter entirely from the entry, marking it as good for basically all eternity. If we do "dontknow" we don't do a thing @poleguy this means that a bad actor can play games with us until the point we managed to do one boot that worked correctly, but from that point on, we'll never regress anymore. I like to believe that that's quite a sensible and simple policy that should work for most cases. It balances robustness against chance for attackers to hold off updates indefinitely. @pid_eins thanks. That does seem reasonable and for remotely managed systems and better than the alternative, which is manual intervention. I worry a smidge about added complexity. I can't shake the feeling that we keep adding layers of complexity to our systems. It feels okay to add complexity that is proportional to the complexity of the problem being solved. In this case it seems sane. However these remotely managed systems all tend to have out of band methods to recover already, no?  @pid_eins but would this really prevent it, when the configuration of a kernel driver goes bad? If I understand things correctly here (big if), only if you store that config in a volume that can be reverted it would be possible to fix the issue. Otherwise you boot into the emergency shell and you are non the wiser than Windows systems are right now. And given it's an endpoint protection that is supposed to react pretty instant to changes, I don't see how you would get theses in the A/B update. @sheogorath on linux drivers dont really have a "configuration" per se. At least not much you pass into the early, risky parts of the boot process. Subsystems might have some config. In a systemd world you wrap the im authenticated/signed PE addons or confext images, and those you drop next to a specific kernel image, thus you can revert them together as one or update as one and so on. Or in other words: the way we parameterize kernels in modern ways also makes it easy to do assessment/fallback. @pid_eins how exactly is a successful boot defined though? Boots to init? What happens if the system boots successfully, runs for ~60 seconds, and then the kernel panics when the first cron job/timer runs?  @JustinAzoff @pid_eins see the link at the start of the thread, flexible strategies available @JustinAzoff depends on the usecase. Different systems/OSes want different stuff there. Some might just check if system manages to reach some point in the boot process, others might want to also require network pings to work, other stuff might instead just want to check that some services stay up for some minimum amount of time and so on. systemd gives you the basic infra for this and some super basic tests in this sense, but individual OS images might want to fill in more tests/conditions.   Well, the lesson for me (aside for other obvious ones) is that for the industrial systems it should be absolutely mandatory to be something like #SUSE #SLEMicro (or its Red Hat equivalent): snapshot based, with R/O system, where the system would automatically boot from an older snapshot if the current one fails. The fact that airline computers are not something like this, is just mind-blowing. Yes, preaching the same gospel @sysrich preached for years.  @pid_eins for a system like Crowdstrike, you'd want to extend that to cover data files the kernel loads. I wonder how well that'd work with the rate of updates they were pushing out? @jamesh i think everyone agrees you have to cover the kernel itself and the initrd with these assesment/fallback schemes. I personally would also then cover the rootfs you boot into with that, but people have different opinions how far the coverage should reach, and how much you "pin" through a boot attempt. @vurpo nope, of course boot assessment would catch this. Key is just that you "pin" enough as part of an attempt, and thus can revert sufficient parts to get things working. On Linux you'd pin kernel *and* initrd at the very leas, and in the model i propose even the entire /usr for each attempt, to maximize coverage of the assesment logic.  @bse @pid_eins @vurpo If you deliberately bypass the system integrity and safety features, then they won't save you. It doesn't matter what those features are. If you blindly sign initrds, checking won't help you. If you blindly mark a boot config as good, or if you replace your rollback image, or whatever... The system won't save you from yourself infallibly. You'll still want staging, and monitoring, and disaster recovery. @pid_eins @vurpo Since both entries would be using the same kernel and hence use the same /lib/modules/$(uname -r)/, you need a mechanism to have multiple versions of your modules folder. If you're serious about preventing older boot entries from breaking retroactively, i think full system snapshots are the only option. Short of that, there might be some compromises like bundling a kernel and all modules, which of course does not protect userspace, but might be easier for commonly used distros. @pid_eins The shocking thing is that this was a requirement for Carrier Grade Linux two decades ago already.  |
Gregory's Server
And it's a shame that commercial distros do not hook into that, and the boot stack of them hasn't changed in more than a decade, is laughably bad at security (unsigned initrds, ffs!) and robustness, and the if you have boot assessment enabled at all turn it into a fantastic DoS (by showing you a boot menu instead of reverting to a working boot choice).