 HT to @wdormann here - somebody has backdoored the open source project XZ which has downstream impacts. For example, although OpenSSH doesn’t use XZ, Debian patch OpenSSH and introduced a dependency which translates as the XZ changes introducing a sshd authentication bypass backdoor it appears. One dude bothered to investigate in his free time about why ssh was running slow, so it was caught fairly early - i.e. hopefully before distros started bundling it. 56 comments
 @GossiTheDog We're incredibly lucky that it wasn't more sophisticated, or it may have gone unnoticed for a long time.  @emaste @GossiTheDog Pretty wild that the implication is that this particular backdoor was only caught because it was coded badly enough to have a significant performance impact. What if it’d been better written, I wonder? I suspect distros probably want to roll XZ back to around January 2024, stop bundling updates until the developer is removed in GitHub or a logical explanation can be given, and somebody needs to fund a code review of it. As I said, the impact here will be very limited due to how quick it was caught. Everybody owes the finder a beer. @GossiTheDog I'm sure the FBI will thank him in their traditional fashion for cyber security saviours @GossiTheDog indeed we do. This exploit was carefully aimed at Linux using systemd launched SSH. We non Linux folks can breathe a hair easier but who knows what else is going on. Don't panic but definitely be cautious.   @GossiTheDog That specific dev contributed to wasmtime etc too. So more projects need audits. Postgre developer @AndresFreundTec saving Linux security from backdoors as a side of desk activity  "Oops, I just stopped a massive operation that could have brought down everything under the sun, accidentally." *flexes* CISA advisory: Reported Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094 https://www.cisa.gov/news-events/alerts/2024/03/29/reported-supply-chain-compromise-affecting-xz-utils-data-compression-library-cve-2024-3094 The person/account on XZ repo also altered the security disclosure policy on that and other repos they author in months prior.   @GossiTheDog That's interesting. So basically removed all of the requests for specific information, and left reporting as: "send us an email or something"? @jbaggs yeah. So they changed one to “If you discover a security vulnerability in this project please report it privately. Do not disclose it as a public issue.” as an example. Looks like a long game attack. @GossiTheDog @jbaggs bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417 this is also interesting, they signed up yesterday on ubuntu How long before the numpties begin combining this with their Baltimore bridge cyber conspiracy theories? Interesting find by @fuomag9 - the XZ repo person tried getting Ubuntu to update yesterday by filing a bug report https://bugs.launchpad.net/bugs/2059417 The Twilight zone time - a bug from 2015 comes back around in XZ incident, it appears https://github.com/google/sanitizers/issues/342 @GossiTheDog wait, are we at "implementing features to get plausible deniability for evading vuln scanning?" Because if so this feels quite novel. At the very least I can't remember a comparable supply chain attack of this sophistication from the top of my head  @uoxc @GossiTheDog Doesn't need to be that sophisticated.. Either (a) developer continues submitting other work whilst compromising at least one project. or (b) the account is in use by two people - the real developer and a hacker. So the question becomes compromised account or compromised developer. In the first case it's not that unbelievable that someone who was up to no good would try to cover their tracks by doing other unrelated things.   @GossiTheDog @fuomag9 Interesting that Debian and other distros already knew about the issue (on embargo) and were already reverting, but Ubuntu didn’t seem to know about it yet from these comments Also, reading these comments apparently 5.6.0 was already on ubuntu? @GossiTheDog either that or whatever build infra they're using for the release tarballs has been compromised. @spad @GossiTheDog No, because one part of the backdoor lies in the version controlled code. Only the trigger is missing from git and present in the tarballs.  @SamantazFox @spad @GossiTheDog could be, but I’m not certain. I haven’t dug into this besides some quick reading, but something feels odd to me about the nature of the attack. It is almost like someone took benign code and used it like “gadgets”. Caveat: Just a hunch at this time. @GossiTheDog That's my concern as well. It's super easy to contribute code and make a good impression, leading to the core maintainers trusting you, and in turn getting enrolled in the core team because everyone is tired. I wonder how many other projects have bad actors in their core team. @GossiTheDog@cyberplace.social Even wilder, the original developer may have been a target of a pressure campaign to get them to pass maintenance on to the second individual: https://mastodon.social/@vegard/112181070803627404 @GossiTheDog @wdormann it was included in debian testing and unstable, they've released a fix now  @GossiTheDog @wdormann @GossiTheDog ... or slows down user facing interactions Really just incredible luck that this was found. One has to wonder how many malicious changes are never detected. @GossiTheDog  @GossiTheDog @wdormann OpenSSH, OpenSSL...  @leeloo @GossiTheDog @wdormann Maybe read the announcements before spouting stupid hot takes? @sjuvonen @GossiTheDog @wdormann As I understand it, Debian took a heavily audited piece of code (OpenSSH), added a patch to use a library from an untrusted source, who then (allegedly deliberately) added a back door. What am I missing? @leeloo You’re the one throwing accusations. Explain how Fedora and openSUSE among others using the same libs and being similarly affected is Debian’s fault? Everybody’s doing it because xz is a dependency of systemd. Clout chasing by pissing on foundational open source projects would probably fly better on LinkedIn. @sjuvonen I'm also not the one claiming Debian added a patch, that was from the post I replied to. And if systemd really requires non-systemd daemons to link to random compression libraries, systemd is a lot worse than I have ever claimed.  @emberquill @leeloo @sjuvonen @wdormann yep, good take I think based on what we know so far, disclaimer that I am an idiot In mid 2023 the presumed attacker got OSS Fuzzer to decrease detection on XZ, in 2022 it looks like significant pressure from multiple accounts was used against the then maintainer and creator to hand over the project. To me it looks like a lot of work went into this over an extended period, and it all got rumbled due to one person being bored and looking into performance.  @GossiTheDog @emberquill I am just learning about this same as every one else, but these kinds of attacks seem to be high effort, high risk since its public with a good audit trail and all it takes is one curious soul to say "hmm that's weird" to blow the whole thing up. I don't know if there are other better hidden ops out there, but the ones that are confirmed seem to get detected within days and weeks, rarely months and years, or at least that's the layman's impression I have. Is it worth it?  @raven667 @GossiTheDog Debian-unstable was vulnerable for a whole month, and the only reason why this exploit was noticed at all was because one person thought their ssh authentication was too slow and started investigating. It could have easily remained unnoticed for another month or two.  @GossiTheDog @emberquill @leeloo @sjuvonen @wdormann once again, some weird nerd’s obsession saves the day. Weird nerds are the heroes we need, but definitely don’t deserve. @emberquill @leeloo @sjuvonen @GossiTheDog @wdormann The question that occurs is why does compatibility with systemd-notify require any more than a write to a well known pipe? xz should not be in the picture here. *especially* with something as critical as ssh. @GossiTheDog @wdormann Several linux distros have already investigated how they're impacted by this (thanks @mgorny and @VoidLinux). Any takes on this from @almalinux and @alpinelinux?  @GossiTheDog Debian Sid's and testing's liblzma has the backdoor, although it looks like it was reverted already. I don't think any official releases of Debian or Ubuntu had the compromised packaging. https://metadata.ftp-master.debian.org/changelogs//main/x/xz-utils/xz-utils_5.6.1+really5.4.5-1_changelog |
Gregory's Server
Worryingly it looks like the backdoor comes via one of the two main devs and dates back over a month from their GitHub account, with legit commits too - XZ is used in systemd so this one might play out for a while.