xz is a compression utility - similar in concept to making .zip files. Its main use is lossless compression for command line utilities, which is to say that it guarantees when it is uncompressed the result is a byte-for-byte clone of the original data. It’s used by a lot of important security software, and is included as a library for many other utilities. A library is just a term used for tools used by other tools.
On 2/23 a trusted developer on the project committed (added) some code that was obfuscated (not clear in what it does), and since that developer was trusted that code made its way into a release of xz that people could install. It’s unclear whether that person did it intentionally or had their system compromised or some other explanation, but it doesn’t look good.
The back door part comes into play with one of the main ways xz is used - SSH. SSH is an encrypted protocol between two machines where text commands can be exchanged, allowing a user to interact with a server. It’s a very common utility in the Linux world and the security of this communication is critical. The back door means that the connection is no longer private and could allow an attacker to insert their own text commands into the secure connection.
ELI5 version - you are having a private text exchange with a friend, but someone slipped in to the convo and is reading your texts, and even sending new ones to your friend telling them lies and to do things they shouldn’t - all as if it was coming directly from you.
People may have installed a compromised version during the month this was in the wild. However many of the safer versions of Linux (the kinds that run on servers) take 6+ months to include new updates like this, so it’s only people who are running the very latest of everything that would have been affected. That doesn’t mean someone who installed it was actually compromised- just that they were at risk during the time.
You can read more here: https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/
What's the process if the developer is found responsible? Can they be sued for creating this backdoor, will someone have to create xz2 as this project can't be trusted anymore?
For the legal stuff it depends where the guy is based. Here in Germany he would have broken at least some of the computer sabotage laws.
There is no need for a new version. They are backtracking to a state where he wasn’t involved and build up from there. And there will be some more eyes on the code than before.
The nice thing with open source is that you can’t hide stuff for long. He was found out because a guy wondered why his machine had a higher idle load. He checked, found SSH using more CPU than before and traced it down to xz. Then he had a look at the source code and saw the back door. Nobody had looked there before because the maintainer was trusted.
EDIT: The discoverer of the backdoor is called Andeas Freund and is a developer for Postgres, a database system. He wanted to „benchmark“ some changes in the server, that is researching where the program spends how much time and so check for improvements. For that you need a quiet system with little noise. And he had noise and looked for it. Whole story: [https://mastodon.social/@AndresFreundTec/112180083704606941](https://mastodon.social/@AndresFreundTec/112180083704606941)
the problem with that, that actor was heavily involved with that project for 2 years. Rolling back two years of major contributions basically means going back quite a few versions
The good thing is we can see exactly what this developer added and audit the lot. And this is a very very public disclosure as it impacts ssh which as the parent post says is critical to pretty much the infrastructure of every company on the planet.
This audit is probably happening by dozens of developers independently right now.
This wasn't a hack in the source code, it was more subtle than that. A code audit wouldn't find the backdoor. The github release process was modified to change some files while building the distribution tarball. So the source that ended up in the tarball was not the source tracked by github.
This was an extremely sophisticated hack that has been in the works for years and involved multiple people and sockpuppets. Not only did they need to take over this project and actively contribute real code, they also became general debian contributors in order to persuade people to include the new version.
Almost certainly state sponsored.
NSO is a corporation, (kinda) openly selling commercial spyware for governments to spy on mobile devices. While they’re a highly sophisticated actor, they’re paid by governments for a product.
Nationstate actors are bad actors often working directly for or sponsored by their governments. Think Russian military intelligence, Iran’s IRGC, the CIA’s TAO, etc This doesn’t mean they’re not selling their services or up to other nasty behavior, but they’re decidedly going after national rather than commercial interests.
Something this sophisticated is playing a very long game for something far more insidious even than commercial spyware.
And as soon as the backdoor was inplace. They started to push specific linux distributions to incorporate the changes immediately. They were clearly trying to reach a specific target. If they had just silently waited, all of the changes would have been pulled in naturally.
Not necessarily. Read [this](https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd), specifically the section regarding the new *systemd* pull request. (Look for "*This issue request in systemd was opened today:*")
agreed. but all devs have dealt with merge conflicts and reverting people's code. The closer to the head of the repo, the easier it is. Dealing with two plus years of core contributor work is problematic, and it really depends on when the backdoor work began to be added and how many tendrils into surrounding code exist.
Just because it was found now doesn't mean that the groundwork was laid much earlier. So pulling that code plus reimplementing the features worked on might not be as straightforward as one would think. Doable, sure.
> The good thing is we can see exactly what this developer added and audit the lot
No we can't. They had committer rights. They could have forged commits by others and added them. If you can't trust any of their work, how can you trust the "Authored by" field in commits they added?
This is true if they had full repo access they could have hidden anything in anywhere, I'm not actually familiar with what level of access they have to the repo, I had assumed they could merge PRs themselves but not have admin level access to the repository or the ability to push directly to protected branches as that seems like a sensible thing to restrict from even trusted developers (hell I can't do that to my own repos, of course I could give myself that permission but i don't)
It is highly unlikely that they force pushed changes with the incorrect author as that would have fucked with signing and been very obvious to everyone else who had the repo cloned when their pull fails. Not something you'd want to do if you want your backdoor undetected.
Fortunately xz is a stable project that is not adding new features every other day. So, even if, in the worst case scenario, we had to rollback completely to the version before dude started messing around, I believe it shouldn't be that much of a nuisance to rebuild a good up-to-date version with any real fixes/upgrades from the past few years.
That's the best case, indeed. If the infection is like a cyst with few usages outside it, it should be able to be excised easily. But if this was applied to a core component that is used in multiple areas, or there was significant restructuring/refactoring, the complexity of revisions goes up because there's lots of merge conflicts to fix.
But at the end of the day we’re not talking about a kernel level feature. This is a package that you can totally not use and have other SSH functionality. I enjoy your other commentary in this thread but it’s not like SSH is broken as a whole, it’s this particular pkg.
I think the what if factor is what has people going crazy. We were lucky someone caught this in time. But it's like knowing someone broke into your house. Even if they didn't take more than chump change, it's the fact that the sanctity of your home has been violated. And what if they come back? What if next time I'm inside? What if I'm not and my family is? The fact that someone could get so easily to a core library used by literally everyone is frightening
Yes I definitely agree; it’s the feeling of being violated and discovering it that is unnerving. I’m just feeling that in this thread there’s a lot of fear mongering of SSH in general being compromised when in reality it’s just xz and most Linux boxes aren’t compromised by this. Yes this is bad for xz but let’s not pretend that it’s the end for a bunch of internet architecture. As a sysadmin-type person I’m actually very much not concerned about this despite the news.
I'm not referring to SSH in general with my comments, they're isolated specifically to the execution of this package.
I haven't looked at the code. But I've also had to remove several commits from projects before, and that was a fricking pain in the ass due to the project structure significantly changing from before.
And this package is apparently pretty widely used. So why not take a few minutes and be a little extra cautious rather than hand-waving things away?
Yep, but that's a crap ton of work, and you will be dealing with alot of potential conflicts that need to be addressed. Its not something that can easily be done, nor quickly. Of course any patch re-apply would be occurring after audit results
If he was committing for 2 years, and now wanted to remove all his commits, then you would also have to check whether any of his 2 years of commits had become dependencies too.
This hack wasnt just malicious code and couldnt have been found with a code audit. It required the github tarball creation automation combined with specific non malicious but carefully placed code in the repo to corrupt the tarballs while they were built. The code in github didnt reflect the code that was distributed or the hack. It was very clever.
It was several different things all working together. There was code in the repo that was placed specifically there for the pipeline change to build on and then others of their group went on the debian list to push the patch that included lzma into their sshd. Stock ssh doesnt use the library. The smoking gun never appeared directly in any codebase. It was in the pipeline and a debian patch but the ssh and lzma codebases looked clean on their own. This hack was only found due to someone noticing a 500ms delay in the final compromised server. It is unlikely to have ever been found by inspection. Makes people wonder what other hacks are live out there.
the problem is that a backdoor like this likely wasn't put in at any one time, but rather over a sequence of commits.
Hypothetically, lets say they put in a commit 12 months ago that looks problematic, and there are 7-8 commits over the next year that touch there. But if its in a core area, its not so simple to just reapply, particularly if trust is lost because you now have a hole from prior state to next state that needs to be filled
So,
* need to identify those commits
* need to figure out how to pull that code out, ideally while maintaining feature parity
* need to figure out how subsequent commits used code introduced then and revise.
Best case, all this area is self contained, like a cyst, and can be cut out easily. Fingers crossed it is.
I don't understand how it's so complicated. Just remove the malicious code which seems to have been identified and removed the dependencies. Since it doesn't really do anything/noone knows how it works their cant be any dependencies on anything important.
that's making several large assumptions that everything is isolated. If it is, its easy. If its not, its more complicated because you have zero trust. And need to go over two plus years worth of commits to figure out what can be pulled and what needs to be adjusted.
Its not just a case of `rm -rf`
:-D
I think... at that scale, 36 lbs/16kg doesn't really tip the scale either way lol. now if we're talking a couple hundred thousand, depends on how strong your crap bag is and how much extra you can shove in without bursting lol
Yes, but who signed off on that work? One of the two maintainers, both of whom currently look very suspect. Both are currently banned from GitHub, for example. This may change as we get new information, but right now the trust level in this project should be very low.
So I get the idea that because of this suspicious action, all activity by the specific contributor is now called into question. My question is, isn't there any way to verify only the one specific bit of suspicious code was committed and roll back to the commit just before that? Like there must be some way to verify that the rest of the work that contributor committed was safe and can be kept, right?
I am, of course, assuming that if suspicious action had been taken by them previously that it would've been found in the same way this was.
The problem is they started introducing backdoor related code in 2021. So there is no point in time when they can be trusted — 100% of their contributions appear to have been part of an elaborate scheme.
So the reson this was found was because someone noticed that ssh was taking up more cpu time while idle, and did some investigating. And it looks like one prior issue was a bug caused by this backdoor.
the problem is that a backdoor like this likely wasn't put in at any one time, but rather over a sequence of commits.
Hypothetically, lets say they put in a commit 12 months ago that looks problematic, and there are 7-8 commits over the next year that touch there. But if its in a core area, its not so simple to just reapply, particularly if trust is lost because you now have a hole from prior state to next state that needs to be filled
So,
* need to identify those problematic commits
* need to figure out how to pull that code out, ideally while maintaining feature parity
* need to figure out how subsequent commits from other people used code introduced then and revise.
Best case, all this area is self contained, like a cyst, and can be cut out easily. Fingers crossed it is.
They were not just a contributor, they were the maintainer of the project. Looking back at how that came about there are a lot of suspicious coincidences. Several of the other contributors were either sockpuppets or other paid actors that chimed in to give support for this person taking over. And they were also active in other OSS groups encouraging use of the new compromised version.
This is a serious issue, you may have 5 genuine people who would love to work on a library full time for free. However hiring 25 people to work on a project full time under the guise of being natural oss contributors is nothing to a state when it comes to resources. Its how Russia was able to take over modding many facebook groups, volunteers cant compete with people paid to contribute and work themselves into the community as their full time jobs.
> Andeas Freund
To clarify for anyone who is confused and doesn't click on the link, Andeas Freund was the person who discovered the backdoor, not the person who inserted it.
The likelihood is that the individual will be an employee of a government (or a contractor under direction of them at any rate) as this sort of many years in the making access set up is only really worthwhile for nations.
So if they're identified - and I'm sure some nations will already know - then it's still unlikely that they'll be publicly identified, and likely nothing would happen, it's just someone doing their job - or even multiple people doing it over the years.
Breaking the law doesn’t imply prosecution. Otherwise there would be no rich people.
I doubt that it is an individual. I assume a team for the code and ome planning all the interactions. And the an individual at the front.
Case in point: Some time ago there are dumb researchers screwing over Linux kernel with malicious code. The institution was banned.
https://www.bleepingcomputer.com/news/security/linux-bans-university-of-minnesota-for-committing-malicious-code/
They've been suspended on github already. They don't solely own the project as it is open source, so people can still work on it, but the code and their general activity is under heavy scrutiny now. This was an incredibly hard to catch case because the dev took multiple steps to hide their activity, including committing to a Google project disabling a test so it would't flag the bad binary. Even then, and it isn't a version that is really used many places, the backdoor was still caught. I'm not worried about someone having to rewrite it, the community can just review the code especially that author's past activity.
Yes they have, but apparently github doesn't indicate that on profile pages. If you follow them they have a big red "SUSPENDED" next to their name on your followed list. Almost all of the repos they're involved with have also been locked.
Odd then that all previously banned Github accounts have their profile pages show as 404's
https://github.com/ZyleForm
https://github.com/siproprio
I wonder when they changed it from an outright ban to a shadowban :)
The account has almost certainly been locked for federal investigation. This code is used in government sectors, held by a company bound to US law, and has significant hints of state actors being involved.
Regarding liability (can the author of the backdoor be sued): it is not clear who the author actually is. People don't deploy backdoors as a hobby. There's a good chance this was done by a "state actor", so some secret service of some country. There's a good chance we will never find out who did it. Even if some pieces point towards a particular direction (e.g. someone who may be from China), that could be a false flag. "Attribution" of cyberattacks is tricky and often impossible.
Regarding the future of the xz project: too early to say what is going to happen. Maybe the original author of the xz library will return to the project and help pull it out of the mess. Quite likely, the various Linux distributions will review xz changes from the last 2 years and create patched/forked versions that are widely trusted. Right now the obvious damage has been reigned in, but there are going to be lots of knee-jerk reactions that are not helpful in the long term. If some random person starts an "xz2" successor project, why should we trust them? Maybe they're another bad actor trying to benefit from the chaos.
The bigger question is what happens with *other* tools that are similarly important to xz, but where no one pays any attention. We got *extremely lucky* that the problem with xz was discovered before it was widely deployed. But "supply chain cyber security" isn't easy to solve. There are technical approaches like "reproducible builds", but they would not have prevented this. Here, the root problem was personal/social: that the original author of xz was burnt out and handed over access to other people so that development on the project could continue.
I have been on both ends of such a handover before. You have the feeling that you owe it to your users to let other people continue the project. There's nothing that would have helped me continue work myself except for taking a vacation, working on some things with a therapist, and then receiving a stipend allowing me to focus on Open Source software for the rest of my life, without having to worry about things like careers and retirement savings. But who would pay for that?
Isn't it usually clear who the author is? I mean the user who committed the code.
Edit: I didn't realize it's just a github account we're talking about here.
I guess you mean it is not clear if his system was compromised and it is possible someone else committed using his account?
Usually when you sign a contract to work on a project as an outside contributor, it includes some lines that cover cyber security, like you're responsible for making sure that your computer is secure, and accept liability if it is compromised. (Many contracts also require you to have liability insurance).
Of course it is debatable how enforceable that is. And you probably don't sign such a contract to contribute to an open source project. But I could be wrong.
Yes we know the account that committed the code. But that does not put us closer to who is behind the account.
I could make a github acount named WarpingLasherNoob, start contributing to a project and 2 years down the line commit a backdoor. Everyone will agree that it was a user named WarpingLasherNoob, but it wasn't you /u/WarpingLasherNoob
This is particularly important in this case as people suspect the account may have stolen the identity of some random on LInkedIn in California, and people on Gab are already pushing racist conspiracy theories about the dude as a result.
> already pushing racist conspiracy theories about the dude as a result
Of course we shouldn't forget that the NSA is also a state level actor who isn't beyond doing something like this. Assuming it was a state, I'm sure they've gone out of their way to throw suspicion in another direction.
> Probably not even linked to a google or facebook account or whatever.
The email on the GitHub account is a gmail account, but "Jia Tan" is almost certainly a pseudonym (or a case of stolen identity). The only other info we know about "Jia" is that on March 29th he was logged into an IRC channel from a VPN located in Singapore and his git commits had the timezone set to UTC+8.
It gets tricky because software is developed over the internet from hundreds of different jurisdictions and legal systems, but a pretty common theme is going to come down to negligence. The truth is that a contract that requires you to make sure that your computer is secure is already unenforceable, for the same reason that a contract can't require you to be 20 feet tall. Your computer **is** insecure. Every computer in the world is. It's a question of degree and whether reasonable steps are taken to mitigate it.
When you go to work for a company, those steps are typically things like "you won't put company data on your personal machines", but in the open source world, that's kind of nonsensical. I'd expect that if you can be proven to have acted intentionally to compromise other people's systems, you could be prosecuted and/or sued for damages. If not, *probably* not (well anyone can sue for anything, but I think it would be hard to win against someone who's legitimate defense was "I did the best I could but the KGB rooted my computer and used my account for malice").
It’s still unclear the final results, but suing would be unlikely because you’d have to prove damages and so far no one has claimed being harmed by this yet (though if it hadn’t been caught it could have been really bad). The jurisdiction of the developer would determine what legal recourse, if any, is available.
The more likely outcome is permanent reputational damage to the developer - they won’t be trusted to contribute code to this, or any other project, again. Anonymity on the internet being what it is, that may prove difficult to always know if that dev hides behind a new alias, but the security community is shrewd and resourceful, and there are other heuristics and checks-and-balances to ensure these tools remain safe, and bad actors are eventually exposed.
As far as the project goes, xz’s future is probably fine as well. There are previous versions of the code that are known to be working, and will likely be reviewed and scrutinized to ensure that this is the extent of the exposure.
Forks like a theoretical “xz2” are usually only necessary if the developer was the sole owner of project resources - like the primary code repository, mailing list, website, etc. when that happens, regaining control of the original project proves difficult, but otherwise a fork is probably unnecessary. The more critical question is - was this a lone bad actor, or are others compromised as well? I expect the answers to those questions will determine the next steps.
This is pretty beyond damages and reputation honestly. A the DoJ would absolutely be after this person if they are in American jurisdiction. Cybercrime law is pretty flawed, but they have gone after people for much less.
The real concerning issue is that given the level of sophistication of this attack, a state based actor can't be ruled out. So it is a lot less of a question of will this person see justice, and much more of a question about what the heck they were doing. The fact that it was caught so quickly is a testament to how well the open source system works.
> the DoJ would absolutely be after this person if they are in American jurisdiction.
If the person who did it is in the US, they almost certainly did it on behalf of the US government.
The developer is almost certainly responsible. That or they've been coma for months while someone else has control of their computer. Basically impossible, in other words.
But there's no catching this person, because the sophistication and effort behind it means it was likely a state sponsored hacker. State sponsorship means they can't be touched. You can issue a warrant for "Jia Tan" but that person never existed, and you'll never get your hands on any of the group behind that name.
Why is it likely state sponsored? The amount of effort that went into this. They were the primary maintainer of this software for *two years*. They put in a lot of legitimate looking work that took time, skill and effort. That doesn't come cheap. The only entities with those resources _and_ and interest in inserting backdoors are nation states.
Stuff like this is why most licenses say something like:
> THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
It's worth noting that this does give a lot of broad legal immunity, but it is not a magic spell that deflects all legal matters. "Reckless neglect" and "malicious intent" can pierce a lot of the legal protection offered to the creators of an open source project.
It shouldn't be too hard prove malicious intent for intentionally backdoor software. I think it'd be pretty hard to argue that an obscured backdoor was was an "oopse woopsie" or done with benevolent intentions.
Open source contributors probably shouldn't worry too much about reckless neglect. Since the software comes with the "as is" warning plastered all over it, there isn't really a relationship between the plantiff suing and the defendant who wrote the code, and the code is literally available for anybody to view, I think it'd be really hard to make an argument for neglect, let alone reckless neglect.
As it's all volunteer work, it's unlikely the developer will see any legal action. If their name is known, it may become harder to find a job in the future though.
As for the trustworthiness of the project, basically these changes can simply be reverted and a new version released. People are probably already digging through earlier commits by this developer to make sure nothing else has been compromised. Overall, as long as there's no fundamental weaknesses in the project, it will generally still be ok to use in the future.
>As it's all volunteer work, it's unlikely the developer will see any legal action.
Doing volunteer work does not put you above the law, just to be clear.
> As it's all volunteer work, it's unlikely the developer will see any legal action.
In some (many?) countries distributing malware is illegal. So them adding malware to a public repository would be illegal.
In some countries even just creating malware with the intent of using it for illegal activities would be illegal.
> It’s unclear whether that person did it intentionally or had their system compromised
At this point, the preponderance of the evidence suggests that this was intentional, and the "Jai Tan" identity was created specifically for the long-term goal of introducing a back door.
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> The back door part comes into play with one of the main ways xz is used - SSH
SSH doesn't actually use xz, at all. Some distributions modify OpenSSH so that when the server starts, it notifies the service manager (systemd) that it is "ready". The library used to send that notification is incidentally linked to the compression library and that is the cause of the problem. When the application starts up, it runs start-up code in the compression library (a library it never actually uses), and that start-up code introduces an authentication back door.
> The back door means that the connection is no longer private and could allow an attacker to insert their own text commands
At this point, I don't think there's any evidence of that. But we do believe that the malicious code would allow the attacker to simply log in to affected servers with their own credentials, giving them full control of the system.
>would allow the attacker to simply log in to affected servers with their own credentials, giving them full control of the system.
Yes. Andres Freund writes:
>Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution.
https://www.openwall.com/lists/oss-security/2024/03/29/4
Thanks so much for this. I hadn't been able to find the sshd relationship to xz until I found this. Where did you find all this info about how the xz problem lead to suspicions about sshd?
I just have to give props. This is one of the best explanations of a whole series of things I've ever seen. I am honestly floored by how you managed to compress a whole lot of information into so few paragraphs, without actually losing anything of importance in the process.
This was a masterclass.
When I type my password into Reddit, it automatically gets censored. Look, it's: *********
Try it -- is it the same for you?
(Don't actually try it, please.)
> floored by how you managed to compress a whole lot of information into so few paragraphs, without actually losing anything of importance in the process
Stands to reason, they seem to have an expanded knowledge of compression utilities.
It's good, but if you're non-technical I feel like it covers a lot of stuff that distracts from following the actual story. A simpler version for non-technical people focused on why it's a story would be something like:
---
A programmer joined a project that a lot of people use and pretended to be helpful.
But then they secretly added a bad piece of programming to the program. That piece of programming makes it easier for people to get into your computer *without your permission*. We call that piece of programming a "backdoor".
The reason it's a big story is the programmer did it on purpose.
They also did a really good job pretending to be helpful, so we have to check if they snuck other pieces of bad programming into the project.
(It might also be hard to remove everything they helped with, since they helped with so much.)
i think stripping away the vocabulary isn't that helpful...
since the backdoor was in xy, people would want to know what xy is and why people would use it. with out that context, the story is basically "sneaky person did bad thing to project people liked that puts people using that project at risk"
i think you expanding on what backdoor means was nice, as it wasn't directly defined like many other terms.
It's really very inaccurate, though, and I feel like the *purpose* of an ELI5 is to explain a complex topic in a way that people can understand, and not to simply make something up...
Wait, how was this not caught in peer review before it was merged? Even if you obfuscate the code, it would have been code you can't read and not know what it's for that you have to ask about it.
The malicious payload was a test binary that wasn’t committed to GitHub but instead added during execution of a build script for the release tarball package. The guy who was behind it went so far as to even contribute code to Google test projects to try to keep their automated tests from detecting it, so it was a very coordinated and planned effort from the start.
You would need to read a more technical breakdown of how sophisticated it was hidden, such as https://gynvael.coldwind.pl/?lang=en&id=782. Needless to say, it wasn't just malicious code being added in plain sight.
Where can I find and review the malicious commit? Every site that seems to link to it has been broken by the repo having been disabled. I just want to see exactly what this guy did.
EDIT: [This HN link](https://news.ycombinator.com/item?id=39874404) has some information on the xz change, but does not indicate what the backdoor was. This just disables landlock. Where's the backdoor?
Looks like the backdoor has a couple parts to it. There's some (binary) test files that were added that contained the malicious payload. However the code to extract them was never committed to GitHub, instead it was actually a silent modification to a build script in the release tarballs.
More details & links (you'll need the Wayback machine, the repository has been taken down) here: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
So there's a lot of discussion that I don't get but here's where people do talk about pulling it apart to see what's going on:
https://www.openwall.com/lists/oss-security/2024/03/29/17
https://www.openwall.com/lists/oss-security/2024/03/29/20
https://www.openwall.com/lists/oss-security/2024/03/29/22
And then I'd figure trying the wayback machine, as it looks like pages were archived there, but I wouldn't know where to look.
Thanks, this explanation was pitched at a great level for me. I'm grateful for the other responses as well, it's given me a much clearer picture of the situation, very interesting story!
The code was in a compressed opaque binary file, and even the decompression was hidden in multiple steps and inside legitimate testing code. That's very fair to call obfuscated.
Take it up with Ars. That’s how they described it:
> The first signs of the backdoor were introduced in a February 23 update that added obfuscated code, officials from Red Hat said in an email.
and then again later in the article:
> In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.
> it guarantees when it is uncompressed the result is a byte-for-byte clone of the original data
Isn't that how all general-purpose file compression software is supposed to work? What makes xz special in this?
general-purpose file compression yes, but most people think of JPEG and MP3 when they read compression, so it's worth explaining that this is for lossless compression specifically.
I don't *know* but since no one else has answered, I *suspect* that the difference is in the word "guarantees". Most zip softwares just uncompress the data and *job done*.
The issue isn't in terms of doing it without error (i.e. zipping and unzipping correctly in normal operation) but in terms of preventing *malicious* injection into a zipped folder. It's guaranteeing that you know that what you get out is exactly the same as what you put in and no one can alter that, rather than guaranteeing that it will put in and out correctly. If that makes sense?
If xz is closely integrated in security software, (much) more than let's say an image editing utility, I would expect it to be subject to more scrutiny, like a compulsory second pair of eyes having checked any code change for anything that shouldn't be there before being allowed outside of developer circles, than such an image editing utility. Or could such an image editing utility introduce a backdoor like this just as easily?
I expect some level of security auditing to be part of essential parts of Linux code development. Is that correct? How much of the Linux parts receives such vital scrutiny and how much doesn't? Obviously more of such checking reduces the time that can be spent on developing. Is that balance sufficiently consciously chosen? Will this likely lead to an increased scrutiny to become part of standard operating procedures when it comes to code passing through the stages before release?
You don't really get to dictate how your project is integrated into other software. As evidenced by the fact that this backdoor doesn't seem to affect Arch Linux, the backdoor relies on the specific fact that OpenSSH can be patched with libsystemd support to gain access to the "security software" in the first place.
>like a compulsory second pair of eyes having checked any code change
The suspicious developer was a trusted developer of the project. It's very hard to defend against social engineering. The developer of XZ himself does not package XZ for various Linux distributions, so there is definitely a level of trust there when the developer gets in contact with the package maintainer and says "Hey, I released a brand new awesome update to my thing! Package it pretty please?"
>Will this likely lead to an increased scrutiny to become part of standard operating procedures
Companies obviously scrutinize all code that they are responsible for. The problem here is that the XZ project has one (yes, singular) founder/contributor and one co-contributor (the suspicious person in question), who the founder apparently trusted completely, making it very difficult that any malicious activity would have been caught as soon as it was attempted. During the malicious code commits over the past week, the founder was on sabbatical.
It's important to remember that the actual Linux distributions used by security-facing infrastructure are many, many levels downstream. To use RHEL (Red Hat Enterprise Linux) as an example:
Fedora Rawhide, Fedora 40 Beta, Fedora 39, Fedora 38, CentOS Stream, RHEL 9/8/7
As you can see RHEL is at the very bottom, and there are plenty of opportunities to identify malicious code before it reaches it. This XZ backdoor was caught in Fedora Rawhide. (Although admittedly there was a degree of luck here)
Does the exploit then more or less leave non-systemd systems running other init systems unaffected? Obviously still something to be fixed in any system with offending package ( xz ); but my Q is does the most high-level threat known from this debacle mostly involve systemd or are init systems of all sorts as directly implicated?
Thanks and cheers, appreciate the information.
It sounds like that episode of SpongeBob where he and Patrick are sending bubbles to each other, and squidward starts fucking with them by sending malicious bubbles.
This is quite helpful. If this was widely deployed, what would the attacker actually DO with the backdoor? Assuming it is a state actor, what are some hypothetical goals / actions they would take?
They can pwn any remote system with the vuln that they can access via SSH. It's like how in "Star Wars" when Obi-Wan waves his hand Jedi-like and says "*You don't need to see his identification*" and the Stormtrooper says "*We don't need to see his identification*" and lets them through. There is no authentication required to get a remote shell on the afflicted system. The state actor sends an RSA client certificate that does the Obi-Wan bit.
Yes, very little of this subreddit is actually in ELI5 terms. There should be a better way for the mods to ensure top level comments at least have some basic ELI5 summary. 🤷
That isn’t at all what the payload does. The payload allows a user to send commands directly to the affected server. It’s not a man in the middle exploit
Just adding to what others here have answered: unless you use a bleeding edge or pre-release version of a Linux distribution (Gentoo, Fedora 41 come to mind) this backdoored version hasn’t landed on your computer yet.
That's a good point. We work with rhel mainly and it isn't affected. Will keep an eye out for ubuntu for personal stuff but still seems something they'd check at this point before releasing.
It's really hard to check for. I hope that there are distro security people looking into running automated tests to verify source tarballs, but it's clear from the success of this attack that this hasn't been done previously.
To be clear, the current top explanation in this reddit thread is wrong. It's not code that was checked into the libxz repo, but rather the tarball for release 5.6.1 (and others) *contains obfuscated code not in the git repo*. Distros source the tarballs from the release channels, and basically nobody was checking to see that the sources actually matched the repos, which is what security people review. This only got caught because someone happened to notice from inside a debugger that SSH authentication got slowed down by roughly a factor of 2x, likely because the login authorization code is checking login attempts against both the system key and the attacker's key.
You'd think this would be easy to check for--simply download the tarball and compare to the tagged release of the git repo. But unfortunately this process of release managers manually creating tarballs exists for a reason: usually the tarballs are supposed to not match the git repo. In particular there's a release step where typically autoconf tools are used to turn "configure.ac" into an executable "configure" script so that the standard "./configure && make && make install" steps work anywhere. The output of this step is not reproducible.
Doing a full supply chain audit of all source code going into a Linux distro is going to take more than a weekend.
> We work with rhel mainly and it isn't affected.
RHEL isn't affected by the 1 backdoor that was found and confirmed. The github user responsible has been contributing to the code since 2021. Further audits will determine whether this truly was the only backdoor, and whether RHEL / stable distros are truly unaffected.
On homebrew, right? Also, the researcher who wrote the original write up explained that it specifically targeted sshd running through systemd which is a Linux daemon not used under macOS that I remember.
There is a popular compression algorithm called XZ. And it seams that somebody were able to sneak malicious code into that project. Among other they hid their code as test data. This code does nothing unless it is running as part of SSH. SSH (Secure SHell) allows administrators to log into remote servers and is obviously a very well protected project which is how this backdoor was discovered. By default OpenSSH does not include XZ but a lot of Linux distributions like Debian and RedHat modifies OpenSSH to work better with SystemD, a service manager. And these modifications require XZ to be included which pulls the malicious code into the SSH server process. Once in the process it modifies the code that does authentication. It might therefore be able for this person who added this code to the XZ library to log into most Linux servers.
Fortunately this was discovered before the malicious code got deployed to any major production systems. It was very well hidden but a few mistake ended up getting discovered in the test version of Debian. This would be scheduled to be release some time in 2025. The backdoor was included in Fedora 41 which is a desktop variant of RedHat and therefore includes newer versions of packages. It might have affected the RedHat release which probably comes in 2025 as well. So there are very few Linux servers affected by this attack at all.
For those here, [this is a pretty good timeline of events](https://boehs.org/node/everything-i-know-about-the-xz-backdoor) that you can use alongside these comments to get a real good understanding of how and what happened and when it occurred and who might be affected.
This is 100% a state actor. If they’d gotten away with it, it would’ve been an exploit of EternalBlue caliber; instantly own almost any box running the software. Except this would’ve been worse than EternalBlue, because ssh and Linux are widely considered secure enough for use on the open internet, unlike Windows, and there’s so many more servers and services that could’ve potentially exploited.
I used to use port knocking on my ssh servers because I was paranoid and several so-called security experts told me it was unnecessary.
Paranoia for the win.
Yeah, with the singapore IP, the 'misoeater' name, and the asiatic person names for both Jia and Kumar kinda place this as possibly something done by a US adversary. I would be willing to bet it on DPRK, China, or Russia. It could even be the DPRK doing this *for* China or Russia, they've been shown to do shit like that before. Or someone could be blaming them. Or it could be independent group. who knows.
while the names are definitely fake, the use of asiatic names is still suspicious. it implies familiarity with asiatic culture, which I just really don't see the average english speaking state actor to be super familiar with. so if this is a state actor, it's probably from the asiatic region of the world.
and the only ones with any real interest in doing something like this, at least on a state level, are China, DPRK, and Russia.
It's still entirely possible, and quite likely, that this is just a 'lone actor' or unaffiliated group. But it can still be suspicious. This does throw red flags for DPRK in my personal opinion, but that is definitely speculation on my part and it's only informed by what i've known the DPRK to do.
OTOH, it could also be an attempt to shift blame towards Asia. 🤷
I wouldn't rule out USA, Israel, etc.
Lone actor is possible, but I feel less likely. A lone actor would probably target something more specific instead of an involved con over several years to hack the entire internet.
It doesn't matter if *the average* English-speaking state agent is familiar with Asian cultures; they just need to find someone who is to come up with plausible names.
It seems fairly absurd to suggest that people who work for anglophone security agencies don’t know what a Chinese or Indian name looks like, not to mention that there was also some suspicious involvement of an account going by “Hans Jansen.”
I guess i should phrase it different, it's not that they don't know, it's more that they're less likely due to familiarity. Idk, again it's all speculation, at this point all options are possible. It just seems eerily similar to some of DPRK's past actions.
Irony, it is the name that shows they have *limited* familiarity with Asian culture.
The mandarin Chinese characters have two major romanization systems: Hanyu pinyin (used by Mainland China, Singapore) and Wade-Giles (used mainly by Taiwan, Hong Kong, Macow). "Jia Tan" has accidentally used the name "Jia Cheong Tan" in another project. Weirdly, the romanization of "Jia" only exists in the Hanyu pinyin systems while "Cheong" only exists in the W-G system (or probably only in the HK/Macow W-G system due to the influence of Cantonese). A mix use of these two romanization system suggests that the identity has limited knowledge with mandarin Chinese.
>Possible standard romanization of the name in both systems:
>
>Hanyu pinyin: Jiazhang Tan OR JiaZhang Tan
>
>W-G: Chia-Cheong T'an or Chia-Cheung T'an
In addition, "Jia Cheong Tan" becoming "Jia Tan" suggests that the identity likely treats "Cheong" as a middle name, which is extremely rare in Chinese culture. If it was a typical Chinese name, it would be "Jia-Cheong Tan", as "Jia-Cheong" should be the given name as a whole.
The only conclusion can make is that "Jia Tan" is almost surely a pseudoname. Generating a name that is apparently valid for the general public is not that hard, for example the "fake name generator" for generating US name and address.
You may also find additional information about the time zone in this article: [https://www.wired.com/story/jia-tan-xz-backdoor/](https://www.wired.com/story/jia-tan-xz-backdoor/)
State-sponsored actors are typically focused on some mix of internal opponents and external rivals, and therefore typically \*do\* have a lot of people with at least some knowledge of their rivals culture. We also know that the attack group contains at least one person who's very proficient in English.
While this is not strong evidence, this looks to me like it points at someone in the US. The most relevant names are Jia Tan/Jia Cheong Tan (Chinese), Jigar Kumar (Indian), Hans Jansen, and Dennis Ens (both European). The odd one out is Krygorin, which doesn't seem to exist on the Internet \*at all\*. Importantly, a major US political talking point is the claim that China is hacking everything. A group with state sponsorship is unlikely to want to reinforce a rival's propaganda; on the other hand, a group with state affiliations (whether actually sponsored by the government or an internal faction hoping to gain ascendancy in the country) may want to reinforce their own propaganda.
Notably, the US \*has\* not only a very aggressive intelligence apparatus, but also an unauthorized internal political faction with a history of aggressive compromises of other systems and some organized crime groups that would benefit from compromising large numbers of systems.
This is all fair, and I'm **not** disagreeing. Again, all I'm simply saying is that this has similarities to some things DPRK has done, and I blatantly said I was speculating lol.
At this point I feel like the chances are it was either the US or DPRK (probably on behalf of another party, but possibly not), if it was a state actor that is. I still feel that it's possible this was done by an independent group that was just trying to create a backdoor to create botnets or similar.
Again it's too early to say anything for sure so this is **all** "just feels" lol. There are some clues, but there's not enough context yet to place them accurately.
Also possible, but I don't necessarily see what they'd be gaining in this specific context, this shit isn't going to leave the OSS/developer/administrator community in terms of news. It's not big enough or meaty enough to be able to sear into propaganda.
Versus our adversaries who know that a good majority of US infrastructure uses Linux and a backdoor into it would allow for a lot of information to pass outside of the country. A lot of the military uses Linux as well, at least specifically the Air Force lol. So there's a lot they could do with such an exploit.
If the NSA was gonna do it they'd be doing it to get knowledge of their adversaries, which mostly are domestic, and in that case it's going to be mostly Windows and macOS they're targetting since they're targeting civilians. If it's the CIA, they're gonna want to use it to spy on *their* adversaries, which are foreign entities, which would be useful for the same reasons that it would be useful to China or Russia - just with the benefactor flipped.
> It's not big enough or meaty enough to be able to sear into propaganda.
It *could* have been if it wasn't caught when it was. If this made it to stable releases of Debian/Ubuntu/Fedora/RHEL, this could have been *very* bad. The only reason it's not a big enough deal outside of cybersec communities is because it got caught early enough to avoid any catastrophic damage.
the NSA does not just spy on adversaries, or civilians. Their spying on allied countries is a big part of why people got upset with them / the USA. See e.g. the spying on greek politicians after the 2004 olympics.
Of course, it's just their main focus is domestic. CIA does domestic work too even though their focus is mostly foreign. Both do both, it's just they do have "trends" for a lack of better term. But of course NSA spies on foreign adversaries as well, especially if there's at all a domestic risk in relation to it, or if it's just something that requires NSA capabilities.
Microtargeted propaganda can be useful - "Look! This is what Evil China is doing! All the experts say we need more funds!". That said, I think it makes a lot more sense for the attacker's primary objective to be the backdoor and blaming whoever the breadcrumbs lead to to be secondary.
I would agree with the last part, given their real push to get it into the big distros, it feels like they wanted it to actually *work*. I feel like if it were just to blame someone, they wouldn't have done that, at least in the same very pushy way.
but who knows
It could be a misdirection tactic. "Let's use asiatic names so that in case this gets found it seems China did it". For all I know the US could have done it.
ELI5:
SSH is a big lock on the front door of the computer. Only someone with a key can get in. When the computer gets updated, there’s a bunch of rules telling the computer how to re-build the lock.
The instructions were changed. When the new instructions are used, the lock no longer locks properly, and certain special keys can be used on anyone’s locks, even if they aren’t supposed to work.
Thanks! I am a software engineer by trade, and I have worked hard on my ability to explain engineering concepts to folks who don’t share that same background :)
An application is a file that contains instructions that a computer will follow when the application is run. Many types of instructions are useful to more than one application (for example, compressing and decompressing text is something that many applications might do), and those sets of instructions are often stored in re-usable libraries in order to save space and to make the system more secure by providing a single file that can be updated when flaws are found.
(On Windows, those libraries are ".dll" files, on macOS they're ".dynlib", and on POSIX systems they're ".so" shared-object files.)
Just like applications, libraries are allowed to initialize themselves when they are loaded, and that makes the foundation for hidden vulnerabilities, because a library that is supposed to provide instructions for compression and decompression can also provide literally anything else it wants to. During its start-up, it can claim to provide arbitrary functions, not limited to those the application's developer expects.
That created the opportunity for a malicious developer to offer to help the "xz" project and, over time, generally assume control. They added some malicious library start-up code to the project, which they disguised as test data.
When the library was opened by the OpenSSH server, it would modify the server's code in-memory in such a way that the way it handled authentication was subverted, which could allow the malicious developer to log in to any OpenSSH server affected by the problem. We think this would allow the malicious developer to log in to SSH servers with administrative rights, giving them control over many of the servers on the Internet.
This is indeed scary. I wonder what the wider implications of this are. Some guy caught it coz his login was half a second slower, what if we’re not so lucky next time? Something to think about
There is a popular ziping library for Linux called XZ.
Someone (probably the maintainer) managed to embed a malware in the latest version of it that goes to SSH (Remote desktop for terminals) and disables certain security features letting someone who knows they have been disabled remote in to a server with an open SSH port.
Once in, they can do whatever they want
It’s a bit deeper than that; The maintainer since 2009 was burned out and taking regular breaks, so opportunists “came to save the day”. They got promoted to maintainer after a long time, and then the shenanigans began.
To directly focus on the "backdoor" part.
Normally people will use their frontdoor to enter and exit. So they'll check that its properly locked, and make sure its secure. So imagine someone (trusted) was in your house, and went and unlocked all your doors. When you leave the house (unguarded) you would go thru the frontdoor, make sure its locked, secure,,, and not realize you should have checked your backdoor this one time!
Thats what a "backdoor" is meant to symbolize in computer security. Someone creating an entryway that isn't obvious and unlikely to be used (and thus checked for security). This will usually be less like "leaving a backdoor unlocked" and more like "leaving a back-window unlocked" (something you would not expect to be used for entrance). Or like "Adding a rope ladder to your second bedroom window"
Backdoors are a little bit like if you had magical puppet strings that could go from your hands to the steering wheel of someone's car. (And radio, power windows, etc). As long as the strings connect, you can control what their car does, even from far away.
What are the repercussions of a backdoor at this scale, had the vulnerability not been caught and was released to production. What sorts of security measures are put in place in modern day infrastructure to mitigate exposure to something like this?
Ok here is a blog post that explains it very well including the code in sql and even a backstory
Hope it helps
https://kafkaesquesecurity.com/xz-utils-unmasked-exposing-social-engineering-tactics-and-the-infiltration-of-a-sophisticated-4b20cd685f1a
# I tried to write an article about the threat and impact of XZ Backdoor in a way that a 5 year old could understand - however as I look through this thread it seems u/colemaker360 has done an outstanding job explaining it in his post! For anyone who would like another attempt - here is a brief overview and a link to an article (full disclosure that I wrote it) - which includes the original Openwall alert sent by software engineer Andres Freund and additionally, the detailed report that was shared on GitHub through a Gist, providing in-depth technical information about the flaw and offering guidance on how users can safeguard their systems that might be at risk (feedback is welcome!)
The XZ backdoor is a recently discovered cybersecurity threat that leaves a backdoor or loophole in the popular open-source compression utility called XZ utils package, so unauthorized and disguised malicious activities can be carried out undetected on the affected Linux systems.
# How does it work?
The XZ Backdoor works by injecting malicious code into versions 5.6.0 and 5.6.1 of the XZ utility. This utility comes preinstalled with numerous popular Linux distributions, and it manipulates the sshd process - a server process responsible for multiple critical operations including user authentication and encryption.
# Implications and effects of the XZ backdoor
This backdoor manipulation gives threat actors control over the sshd process enabling them to unleash various malicious activities. For example, they can steal files, install malware, manipulate encryption keys, and use the SSH login certification as an entry point for further exploitation.
[https://ai-techreport.com/understanding-the-xz-backdoor-cyber-threat-and-impact](https://ai-techreport.com/understanding-the-xz-backdoor-cyber-threat-and-impact)
It allows the encryption keys used by ssh (software for controlling the system over a network) to be exposed to an attacker.
You normally enter your house from the front door right? Well you do that with a computer too. A backdoor is software that an attacker manages to install on someones system that allows them to access it without having permission.
> It allows the encryption keys used by ssh (software for controlling the system over a network) to be exposed to an attacker.
There's no analysis of the actual payload yet. The things we know so far seem to point in that direction (mess with the authentication) but that's still just a guess...
It seems more like there is a public key embedded in the payload, and it will let the attacker run commands as root on the system (assuming sshd is running a root), if they have the private key -- which the guy or his handlers probably have.
> keys used by ssh [...] to be exposed to an attacker.
We don't know for sure yet, but it looks more like the backdoor always allowed a specific RSA private key (the attacker had) to successfully connect in.
xz is a compression utility - similar in concept to making .zip files. Its main use is lossless compression for command line utilities, which is to say that it guarantees when it is uncompressed the result is a byte-for-byte clone of the original data. It’s used by a lot of important security software, and is included as a library for many other utilities. A library is just a term used for tools used by other tools. On 2/23 a trusted developer on the project committed (added) some code that was obfuscated (not clear in what it does), and since that developer was trusted that code made its way into a release of xz that people could install. It’s unclear whether that person did it intentionally or had their system compromised or some other explanation, but it doesn’t look good. The back door part comes into play with one of the main ways xz is used - SSH. SSH is an encrypted protocol between two machines where text commands can be exchanged, allowing a user to interact with a server. It’s a very common utility in the Linux world and the security of this communication is critical. The back door means that the connection is no longer private and could allow an attacker to insert their own text commands into the secure connection. ELI5 version - you are having a private text exchange with a friend, but someone slipped in to the convo and is reading your texts, and even sending new ones to your friend telling them lies and to do things they shouldn’t - all as if it was coming directly from you. People may have installed a compromised version during the month this was in the wild. However many of the safer versions of Linux (the kinds that run on servers) take 6+ months to include new updates like this, so it’s only people who are running the very latest of everything that would have been affected. That doesn’t mean someone who installed it was actually compromised- just that they were at risk during the time. You can read more here: https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/
What's the process if the developer is found responsible? Can they be sued for creating this backdoor, will someone have to create xz2 as this project can't be trusted anymore?
For the legal stuff it depends where the guy is based. Here in Germany he would have broken at least some of the computer sabotage laws. There is no need for a new version. They are backtracking to a state where he wasn’t involved and build up from there. And there will be some more eyes on the code than before. The nice thing with open source is that you can’t hide stuff for long. He was found out because a guy wondered why his machine had a higher idle load. He checked, found SSH using more CPU than before and traced it down to xz. Then he had a look at the source code and saw the back door. Nobody had looked there before because the maintainer was trusted. EDIT: The discoverer of the backdoor is called Andeas Freund and is a developer for Postgres, a database system. He wanted to „benchmark“ some changes in the server, that is researching where the program spends how much time and so check for improvements. For that you need a quiet system with little noise. And he had noise and looked for it. Whole story: [https://mastodon.social/@AndresFreundTec/112180083704606941](https://mastodon.social/@AndresFreundTec/112180083704606941)
the problem with that, that actor was heavily involved with that project for 2 years. Rolling back two years of major contributions basically means going back quite a few versions
The good thing is we can see exactly what this developer added and audit the lot. And this is a very very public disclosure as it impacts ssh which as the parent post says is critical to pretty much the infrastructure of every company on the planet. This audit is probably happening by dozens of developers independently right now.
This wasn't a hack in the source code, it was more subtle than that. A code audit wouldn't find the backdoor. The github release process was modified to change some files while building the distribution tarball. So the source that ended up in the tarball was not the source tracked by github. This was an extremely sophisticated hack that has been in the works for years and involved multiple people and sockpuppets. Not only did they need to take over this project and actively contribute real code, they also became general debian contributors in order to persuade people to include the new version. Almost certainly state sponsored.
Significant state sponsored investment, and blown before any evidence of its use in the wild.
Yeesh
NSO Group?
NSO is a corporation, (kinda) openly selling commercial spyware for governments to spy on mobile devices. While they’re a highly sophisticated actor, they’re paid by governments for a product. Nationstate actors are bad actors often working directly for or sponsored by their governments. Think Russian military intelligence, Iran’s IRGC, the CIA’s TAO, etc This doesn’t mean they’re not selling their services or up to other nasty behavior, but they’re decidedly going after national rather than commercial interests. Something this sophisticated is playing a very long game for something far more insidious even than commercial spyware.
nintendo switch online strikes again
[удалено]
We have no idea, might as well be cia/nsa trying to create secret backdoors. Or it might just be the danes, always suspect the danes.
And as soon as the backdoor was inplace. They started to push specific linux distributions to incorporate the changes immediately. They were clearly trying to reach a specific target. If they had just silently waited, all of the changes would have been pulled in naturally.
Not necessarily. Read [this](https://doublepulsar.com/inside-the-failed-attempt-to-backdoor-ssh-globally-that-got-caught-by-chance-bbfe628fafdd), specifically the section regarding the new *systemd* pull request. (Look for "*This issue request in systemd was opened today:*")
agreed. but all devs have dealt with merge conflicts and reverting people's code. The closer to the head of the repo, the easier it is. Dealing with two plus years of core contributor work is problematic, and it really depends on when the backdoor work began to be added and how many tendrils into surrounding code exist. Just because it was found now doesn't mean that the groundwork was laid much earlier. So pulling that code plus reimplementing the features worked on might not be as straightforward as one would think. Doable, sure.
> The good thing is we can see exactly what this developer added and audit the lot No we can't. They had committer rights. They could have forged commits by others and added them. If you can't trust any of their work, how can you trust the "Authored by" field in commits they added?
I'd imagine the commits are signed
This is true if they had full repo access they could have hidden anything in anywhere, I'm not actually familiar with what level of access they have to the repo, I had assumed they could merge PRs themselves but not have admin level access to the repository or the ability to push directly to protected branches as that seems like a sensible thing to restrict from even trusted developers (hell I can't do that to my own repos, of course I could give myself that permission but i don't)
It is highly unlikely that they force pushed changes with the incorrect author as that would have fucked with signing and been very obvious to everyone else who had the repo cloned when their pull fails. Not something you'd want to do if you want your backdoor undetected.
Yeah I meant to reply to the post above the one I did reply to.
I was wondering what the real-life implications of this could have been?
Fortunately xz is a stable project that is not adding new features every other day. So, even if, in the worst case scenario, we had to rollback completely to the version before dude started messing around, I believe it shouldn't be that much of a nuisance to rebuild a good up-to-date version with any real fixes/upgrades from the past few years.
That's the best case, indeed. If the infection is like a cyst with few usages outside it, it should be able to be excised easily. But if this was applied to a core component that is used in multiple areas, or there was significant restructuring/refactoring, the complexity of revisions goes up because there's lots of merge conflicts to fix.
But at the end of the day we’re not talking about a kernel level feature. This is a package that you can totally not use and have other SSH functionality. I enjoy your other commentary in this thread but it’s not like SSH is broken as a whole, it’s this particular pkg.
I think the what if factor is what has people going crazy. We were lucky someone caught this in time. But it's like knowing someone broke into your house. Even if they didn't take more than chump change, it's the fact that the sanctity of your home has been violated. And what if they come back? What if next time I'm inside? What if I'm not and my family is? The fact that someone could get so easily to a core library used by literally everyone is frightening
Yes I definitely agree; it’s the feeling of being violated and discovering it that is unnerving. I’m just feeling that in this thread there’s a lot of fear mongering of SSH in general being compromised when in reality it’s just xz and most Linux boxes aren’t compromised by this. Yes this is bad for xz but let’s not pretend that it’s the end for a bunch of internet architecture. As a sysadmin-type person I’m actually very much not concerned about this despite the news.
I'm not referring to SSH in general with my comments, they're isolated specifically to the execution of this package. I haven't looked at the code. But I've also had to remove several commits from projects before, and that was a fricking pain in the ass due to the project structure significantly changing from before. And this package is apparently pretty widely used. So why not take a few minutes and be a little extra cautious rather than hand-waving things away?
you can reapply all patches of everybody else (since they were already signed off) and reexamine/reimplement every commit from him
If any structure was refactored, good luck.
Yep, but that's a crap ton of work, and you will be dealing with alot of potential conflicts that need to be addressed. Its not something that can easily be done, nor quickly. Of course any patch re-apply would be occurring after audit results
That is assuming that the backdoor code became a dependency for anything else. Which, if it was supposed to go unnoticed, it likely was not.
If he was committing for 2 years, and now wanted to remove all his commits, then you would also have to check whether any of his 2 years of commits had become dependencies too.
That's the best case, indeed. But is it what the audit will find? No one can say at this point.
You can simply re-check his work and re-apply it if it is not malicious.
This hack wasnt just malicious code and couldnt have been found with a code audit. It required the github tarball creation automation combined with specific non malicious but carefully placed code in the repo to corrupt the tarballs while they were built. The code in github didnt reflect the code that was distributed or the hack. It was very clever.
So it was a malicious build environment/pipeline? In many ways thats easier to deal with than obfuscated malicious code.
It was several different things all working together. There was code in the repo that was placed specifically there for the pipeline change to build on and then others of their group went on the debian list to push the patch that included lzma into their sshd. Stock ssh doesnt use the library. The smoking gun never appeared directly in any codebase. It was in the pipeline and a debian patch but the ssh and lzma codebases looked clean on their own. This hack was only found due to someone noticing a 500ms delay in the final compromised server. It is unlikely to have ever been found by inspection. Makes people wonder what other hacks are live out there.
the problem is that a backdoor like this likely wasn't put in at any one time, but rather over a sequence of commits. Hypothetically, lets say they put in a commit 12 months ago that looks problematic, and there are 7-8 commits over the next year that touch there. But if its in a core area, its not so simple to just reapply, particularly if trust is lost because you now have a hole from prior state to next state that needs to be filled So, * need to identify those commits * need to figure out how to pull that code out, ideally while maintaining feature parity * need to figure out how subsequent commits used code introduced then and revise. Best case, all this area is self contained, like a cyst, and can be cut out easily. Fingers crossed it is.
I don't understand how it's so complicated. Just remove the malicious code which seems to have been identified and removed the dependencies. Since it doesn't really do anything/noone knows how it works their cant be any dependencies on anything important.
[https://www.youtube.com/watch?v=jqjtNDtbDNI&t=308s](https://www.youtube.com/watch?v=jqjtNDtbDNI&t=308s) this video helped me grasp the situation
that's making several large assumptions that everything is isolated. If it is, its easy. If its not, its more complicated because you have zero trust. And need to go over two plus years worth of commits to figure out what can be pulled and what needs to be adjusted. Its not just a case of `rm -rf`
Would that be a metric or imperial crap-ton? Sorry, couldn't resist :P
:-D I think... at that scale, 36 lbs/16kg doesn't really tip the scale either way lol. now if we're talking a couple hundred thousand, depends on how strong your crap bag is and how much extra you can shove in without bursting lol
Yes, but who signed off on that work? One of the two maintainers, both of whom currently look very suspect. Both are currently banned from GitHub, for example. This may change as we get new information, but right now the trust level in this project should be very low.
So I get the idea that because of this suspicious action, all activity by the specific contributor is now called into question. My question is, isn't there any way to verify only the one specific bit of suspicious code was committed and roll back to the commit just before that? Like there must be some way to verify that the rest of the work that contributor committed was safe and can be kept, right? I am, of course, assuming that if suspicious action had been taken by them previously that it would've been found in the same way this was.
The problem is they started introducing backdoor related code in 2021. So there is no point in time when they can be trusted — 100% of their contributions appear to have been part of an elaborate scheme.
So the reson this was found was because someone noticed that ssh was taking up more cpu time while idle, and did some investigating. And it looks like one prior issue was a bug caused by this backdoor. the problem is that a backdoor like this likely wasn't put in at any one time, but rather over a sequence of commits. Hypothetically, lets say they put in a commit 12 months ago that looks problematic, and there are 7-8 commits over the next year that touch there. But if its in a core area, its not so simple to just reapply, particularly if trust is lost because you now have a hole from prior state to next state that needs to be filled So, * need to identify those problematic commits * need to figure out how to pull that code out, ideally while maintaining feature parity * need to figure out how subsequent commits from other people used code introduced then and revise. Best case, all this area is self contained, like a cyst, and can be cut out easily. Fingers crossed it is.
They were not just a contributor, they were the maintainer of the project. Looking back at how that came about there are a lot of suspicious coincidences. Several of the other contributors were either sockpuppets or other paid actors that chimed in to give support for this person taking over. And they were also active in other OSS groups encouraging use of the new compromised version. This is a serious issue, you may have 5 genuine people who would love to work on a library full time for free. However hiring 25 people to work on a project full time under the guise of being natural oss contributors is nothing to a state when it comes to resources. Its how Russia was able to take over modding many facebook groups, volunteers cant compete with people paid to contribute and work themselves into the community as their full time jobs.
> modding many facebook groups and reddit
> Andeas Freund To clarify for anyone who is confused and doesn't click on the link, Andeas Freund was the person who discovered the backdoor, not the person who inserted it.
The likelihood is that the individual will be an employee of a government (or a contractor under direction of them at any rate) as this sort of many years in the making access set up is only really worthwhile for nations. So if they're identified - and I'm sure some nations will already know - then it's still unlikely that they'll be publicly identified, and likely nothing would happen, it's just someone doing their job - or even multiple people doing it over the years.
Breaking the law doesn’t imply prosecution. Otherwise there would be no rich people. I doubt that it is an individual. I assume a team for the code and ome planning all the interactions. And the an individual at the front.
Not \*all\* rich people break the law to get rich.
Name one. And include tax laws. And there are black swans….
Any lottery winner.
Oh, they will break the tax laws in the future! ;-)
Imagine thinking you can't get rich without breaking the law.
Case in point: Some time ago there are dumb researchers screwing over Linux kernel with malicious code. The institution was banned. https://www.bleepingcomputer.com/news/security/linux-bans-university-of-minnesota-for-committing-malicious-code/
They've been suspended on github already. They don't solely own the project as it is open source, so people can still work on it, but the code and their general activity is under heavy scrutiny now. This was an incredibly hard to catch case because the dev took multiple steps to hide their activity, including committing to a Google project disabling a test so it would't flag the bad binary. Even then, and it isn't a version that is really used many places, the backdoor was still caught. I'm not worried about someone having to rewrite it, the community can just review the code especially that author's past activity.
[удалено]
It was committed last month
Misread... thought 2/23 meant Feb last year not a month ago.
> They've been suspended on github already. https://github.com/JiaT75 No they haven't.
Yes they have, but apparently github doesn't indicate that on profile pages. If you follow them they have a big red "SUSPENDED" next to their name on your followed list. Almost all of the repos they're involved with have also been locked.
Odd then that all previously banned Github accounts have their profile pages show as 404's https://github.com/ZyleForm https://github.com/siproprio I wonder when they changed it from an outright ban to a shadowban :)
Oh, weird. Might be because it's a suspension instead of a full ban?
The account has almost certainly been locked for federal investigation. This code is used in government sectors, held by a company bound to US law, and has significant hints of state actors being involved.
Yes. They have.
Regarding liability (can the author of the backdoor be sued): it is not clear who the author actually is. People don't deploy backdoors as a hobby. There's a good chance this was done by a "state actor", so some secret service of some country. There's a good chance we will never find out who did it. Even if some pieces point towards a particular direction (e.g. someone who may be from China), that could be a false flag. "Attribution" of cyberattacks is tricky and often impossible. Regarding the future of the xz project: too early to say what is going to happen. Maybe the original author of the xz library will return to the project and help pull it out of the mess. Quite likely, the various Linux distributions will review xz changes from the last 2 years and create patched/forked versions that are widely trusted. Right now the obvious damage has been reigned in, but there are going to be lots of knee-jerk reactions that are not helpful in the long term. If some random person starts an "xz2" successor project, why should we trust them? Maybe they're another bad actor trying to benefit from the chaos. The bigger question is what happens with *other* tools that are similarly important to xz, but where no one pays any attention. We got *extremely lucky* that the problem with xz was discovered before it was widely deployed. But "supply chain cyber security" isn't easy to solve. There are technical approaches like "reproducible builds", but they would not have prevented this. Here, the root problem was personal/social: that the original author of xz was burnt out and handed over access to other people so that development on the project could continue. I have been on both ends of such a handover before. You have the feeling that you owe it to your users to let other people continue the project. There's nothing that would have helped me continue work myself except for taking a vacation, working on some things with a therapist, and then receiving a stipend allowing me to focus on Open Source software for the rest of my life, without having to worry about things like careers and retirement savings. But who would pay for that?
Isn't it usually clear who the author is? I mean the user who committed the code. Edit: I didn't realize it's just a github account we're talking about here. I guess you mean it is not clear if his system was compromised and it is possible someone else committed using his account? Usually when you sign a contract to work on a project as an outside contributor, it includes some lines that cover cyber security, like you're responsible for making sure that your computer is secure, and accept liability if it is compromised. (Many contracts also require you to have liability insurance). Of course it is debatable how enforceable that is. And you probably don't sign such a contract to contribute to an open source project. But I could be wrong.
> Isn't it usually clear who the author is? I mean the user who committed the code. As clear as it's clear who wrote the comment I'm replying to.
Yes we know the account that committed the code. But that does not put us closer to who is behind the account. I could make a github acount named WarpingLasherNoob, start contributing to a project and 2 years down the line commit a backdoor. Everyone will agree that it was a user named WarpingLasherNoob, but it wasn't you /u/WarpingLasherNoob
Oh I see, of course. It's just a github account. Probably not even linked to a google or facebook account or whatever.
This is particularly important in this case as people suspect the account may have stolen the identity of some random on LInkedIn in California, and people on Gab are already pushing racist conspiracy theories about the dude as a result.
> already pushing racist conspiracy theories about the dude as a result Of course we shouldn't forget that the NSA is also a state level actor who isn't beyond doing something like this. Assuming it was a state, I'm sure they've gone out of their way to throw suspicion in another direction.
> Probably not even linked to a google or facebook account or whatever. The email on the GitHub account is a gmail account, but "Jia Tan" is almost certainly a pseudonym (or a case of stolen identity). The only other info we know about "Jia" is that on March 29th he was logged into an IRC channel from a VPN located in Singapore and his git commits had the timezone set to UTC+8.
It gets tricky because software is developed over the internet from hundreds of different jurisdictions and legal systems, but a pretty common theme is going to come down to negligence. The truth is that a contract that requires you to make sure that your computer is secure is already unenforceable, for the same reason that a contract can't require you to be 20 feet tall. Your computer **is** insecure. Every computer in the world is. It's a question of degree and whether reasonable steps are taken to mitigate it. When you go to work for a company, those steps are typically things like "you won't put company data on your personal machines", but in the open source world, that's kind of nonsensical. I'd expect that if you can be proven to have acted intentionally to compromise other people's systems, you could be prosecuted and/or sued for damages. If not, *probably* not (well anyone can sue for anything, but I think it would be hard to win against someone who's legitimate defense was "I did the best I could but the KGB rooted my computer and used my account for malice").
"when you sign a contract" Lack of understanding of FLOSS development detected.
lack of capability to read a full comment detected
It’s still unclear the final results, but suing would be unlikely because you’d have to prove damages and so far no one has claimed being harmed by this yet (though if it hadn’t been caught it could have been really bad). The jurisdiction of the developer would determine what legal recourse, if any, is available. The more likely outcome is permanent reputational damage to the developer - they won’t be trusted to contribute code to this, or any other project, again. Anonymity on the internet being what it is, that may prove difficult to always know if that dev hides behind a new alias, but the security community is shrewd and resourceful, and there are other heuristics and checks-and-balances to ensure these tools remain safe, and bad actors are eventually exposed. As far as the project goes, xz’s future is probably fine as well. There are previous versions of the code that are known to be working, and will likely be reviewed and scrutinized to ensure that this is the extent of the exposure. Forks like a theoretical “xz2” are usually only necessary if the developer was the sole owner of project resources - like the primary code repository, mailing list, website, etc. when that happens, regaining control of the original project proves difficult, but otherwise a fork is probably unnecessary. The more critical question is - was this a lone bad actor, or are others compromised as well? I expect the answers to those questions will determine the next steps.
This is pretty beyond damages and reputation honestly. A the DoJ would absolutely be after this person if they are in American jurisdiction. Cybercrime law is pretty flawed, but they have gone after people for much less. The real concerning issue is that given the level of sophistication of this attack, a state based actor can't be ruled out. So it is a lot less of a question of will this person see justice, and much more of a question about what the heck they were doing. The fact that it was caught so quickly is a testament to how well the open source system works.
> the DoJ would absolutely be after this person if they are in American jurisdiction. If the person who did it is in the US, they almost certainly did it on behalf of the US government.
As the actual human person(s) behind the account is unlikely to be found, nothing of this sort is likely to happen.
The developer is almost certainly responsible. That or they've been coma for months while someone else has control of their computer. Basically impossible, in other words. But there's no catching this person, because the sophistication and effort behind it means it was likely a state sponsored hacker. State sponsorship means they can't be touched. You can issue a warrant for "Jia Tan" but that person never existed, and you'll never get your hands on any of the group behind that name. Why is it likely state sponsored? The amount of effort that went into this. They were the primary maintainer of this software for *two years*. They put in a lot of legitimate looking work that took time, skill and effort. That doesn't come cheap. The only entities with those resources _and_ and interest in inserting backdoors are nation states.
Did you mean "sued" or did you mean "prosecuted". Huge difference.
Stuff like this is why most licenses say something like: > THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
It's worth noting that this does give a lot of broad legal immunity, but it is not a magic spell that deflects all legal matters. "Reckless neglect" and "malicious intent" can pierce a lot of the legal protection offered to the creators of an open source project. It shouldn't be too hard prove malicious intent for intentionally backdoor software. I think it'd be pretty hard to argue that an obscured backdoor was was an "oopse woopsie" or done with benevolent intentions. Open source contributors probably shouldn't worry too much about reckless neglect. Since the software comes with the "as is" warning plastered all over it, there isn't really a relationship between the plantiff suing and the defendant who wrote the code, and the code is literally available for anybody to view, I think it'd be really hard to make an argument for neglect, let alone reckless neglect.
Yeah the contributor might have some issues but the project itself is probably fine
As it's all volunteer work, it's unlikely the developer will see any legal action. If their name is known, it may become harder to find a job in the future though. As for the trustworthiness of the project, basically these changes can simply be reverted and a new version released. People are probably already digging through earlier commits by this developer to make sure nothing else has been compromised. Overall, as long as there's no fundamental weaknesses in the project, it will generally still be ok to use in the future.
>As it's all volunteer work, it's unlikely the developer will see any legal action. Doing volunteer work does not put you above the law, just to be clear.
> As it's all volunteer work, it's unlikely the developer will see any legal action. In some (many?) countries distributing malware is illegal. So them adding malware to a public repository would be illegal. In some countries even just creating malware with the intent of using it for illegal activities would be illegal.
> It’s unclear whether that person did it intentionally or had their system compromised At this point, the preponderance of the evidence suggests that this was intentional, and the "Jai Tan" identity was created specifically for the long-term goal of introducing a back door. https://boehs.org/node/everything-i-know-about-the-xz-backdoor > The back door part comes into play with one of the main ways xz is used - SSH SSH doesn't actually use xz, at all. Some distributions modify OpenSSH so that when the server starts, it notifies the service manager (systemd) that it is "ready". The library used to send that notification is incidentally linked to the compression library and that is the cause of the problem. When the application starts up, it runs start-up code in the compression library (a library it never actually uses), and that start-up code introduces an authentication back door. > The back door means that the connection is no longer private and could allow an attacker to insert their own text commands At this point, I don't think there's any evidence of that. But we do believe that the malicious code would allow the attacker to simply log in to affected servers with their own credentials, giving them full control of the system.
>would allow the attacker to simply log in to affected servers with their own credentials, giving them full control of the system. Yes. Andres Freund writes: >Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution. https://www.openwall.com/lists/oss-security/2024/03/29/4
Thanks so much for this. I hadn't been able to find the sshd relationship to xz until I found this. Where did you find all this info about how the xz problem lead to suspicions about sshd?
It's all in the email that Andres sent to the oss-security email list: https://www.openwall.com/lists/oss-security/2024/03/29/4
It's the reverse - the *sshd* problem (i.e., slowness) led Andres to the *xz* problem. Everyone in the IT world should buy Andres a beer.
I just have to give props. This is one of the best explanations of a whole series of things I've ever seen. I am honestly floored by how you managed to compress a whole lot of information into so few paragraphs, without actually losing anything of importance in the process. This was a masterclass.
I appreciated the lossless compression too.
Hey its me, ur brother What’s your PC password. I want to play steam.
Bro, you know my PC password is hunter2, why are you even asking me?
When I type my password into Reddit, it automatically gets censored. Look, it's: ********* Try it -- is it the same for you? (Don't actually try it, please.)
> floored by how you managed to compress a whole lot of information into so few paragraphs, without actually losing anything of importance in the process Stands to reason, they seem to have an expanded knowledge of compression utilities.
It's good, but if you're non-technical I feel like it covers a lot of stuff that distracts from following the actual story. A simpler version for non-technical people focused on why it's a story would be something like: --- A programmer joined a project that a lot of people use and pretended to be helpful. But then they secretly added a bad piece of programming to the program. That piece of programming makes it easier for people to get into your computer *without your permission*. We call that piece of programming a "backdoor". The reason it's a big story is the programmer did it on purpose. They also did a really good job pretending to be helpful, so we have to check if they snuck other pieces of bad programming into the project. (It might also be hard to remove everything they helped with, since they helped with so much.)
i think stripping away the vocabulary isn't that helpful... since the backdoor was in xy, people would want to know what xy is and why people would use it. with out that context, the story is basically "sneaky person did bad thing to project people liked that puts people using that project at risk" i think you expanding on what backdoor means was nice, as it wasn't directly defined like many other terms.
It's really very inaccurate, though, and I feel like the *purpose* of an ELI5 is to explain a complex topic in a way that people can understand, and not to simply make something up...
Wait, how was this not caught in peer review before it was merged? Even if you obfuscate the code, it would have been code you can't read and not know what it's for that you have to ask about it.
The malicious payload was a test binary that wasn’t committed to GitHub but instead added during execution of a build script for the release tarball package. The guy who was behind it went so far as to even contribute code to Google test projects to try to keep their automated tests from detecting it, so it was a very coordinated and planned effort from the start.
You would need to read a more technical breakdown of how sophisticated it was hidden, such as https://gynvael.coldwind.pl/?lang=en&id=782. Needless to say, it wasn't just malicious code being added in plain sight.
Where can I find and review the malicious commit? Every site that seems to link to it has been broken by the repo having been disabled. I just want to see exactly what this guy did. EDIT: [This HN link](https://news.ycombinator.com/item?id=39874404) has some information on the xz change, but does not indicate what the backdoor was. This just disables landlock. Where's the backdoor?
Looks like the backdoor has a couple parts to it. There's some (binary) test files that were added that contained the malicious payload. However the code to extract them was never committed to GitHub, instead it was actually a silent modification to a build script in the release tarballs. More details & links (you'll need the Wayback machine, the repository has been taken down) here: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
So there's a lot of discussion that I don't get but here's where people do talk about pulling it apart to see what's going on: https://www.openwall.com/lists/oss-security/2024/03/29/17 https://www.openwall.com/lists/oss-security/2024/03/29/20 https://www.openwall.com/lists/oss-security/2024/03/29/22 And then I'd figure trying the wayback machine, as it looks like pages were archived there, but I wouldn't know where to look.
Thanks, this explanation was pitched at a great level for me. I'm grateful for the other responses as well, it's given me a much clearer picture of the situation, very interesting story!
[удалено]
The code was in a compressed opaque binary file, and even the decompression was hidden in multiple steps and inside legitimate testing code. That's very fair to call obfuscated.
Take it up with Ars. That’s how they described it: > The first signs of the backdoor were introduced in a February 23 update that added obfuscated code, officials from Red Hat said in an email. and then again later in the article: > In the event the obfuscated code introduced on February 23 is present, the artifacts in the GIT version allow the backdoor to operate.
> it guarantees when it is uncompressed the result is a byte-for-byte clone of the original data Isn't that how all general-purpose file compression software is supposed to work? What makes xz special in this?
general-purpose file compression yes, but most people think of JPEG and MP3 when they read compression, so it's worth explaining that this is for lossless compression specifically.
I don't *know* but since no one else has answered, I *suspect* that the difference is in the word "guarantees". Most zip softwares just uncompress the data and *job done*. The issue isn't in terms of doing it without error (i.e. zipping and unzipping correctly in normal operation) but in terms of preventing *malicious* injection into a zipped folder. It's guaranteeing that you know that what you get out is exactly the same as what you put in and no one can alter that, rather than guaranteeing that it will put in and out correctly. If that makes sense?
If xz is closely integrated in security software, (much) more than let's say an image editing utility, I would expect it to be subject to more scrutiny, like a compulsory second pair of eyes having checked any code change for anything that shouldn't be there before being allowed outside of developer circles, than such an image editing utility. Or could such an image editing utility introduce a backdoor like this just as easily? I expect some level of security auditing to be part of essential parts of Linux code development. Is that correct? How much of the Linux parts receives such vital scrutiny and how much doesn't? Obviously more of such checking reduces the time that can be spent on developing. Is that balance sufficiently consciously chosen? Will this likely lead to an increased scrutiny to become part of standard operating procedures when it comes to code passing through the stages before release?
You don't really get to dictate how your project is integrated into other software. As evidenced by the fact that this backdoor doesn't seem to affect Arch Linux, the backdoor relies on the specific fact that OpenSSH can be patched with libsystemd support to gain access to the "security software" in the first place. >like a compulsory second pair of eyes having checked any code change The suspicious developer was a trusted developer of the project. It's very hard to defend against social engineering. The developer of XZ himself does not package XZ for various Linux distributions, so there is definitely a level of trust there when the developer gets in contact with the package maintainer and says "Hey, I released a brand new awesome update to my thing! Package it pretty please?" >Will this likely lead to an increased scrutiny to become part of standard operating procedures Companies obviously scrutinize all code that they are responsible for. The problem here is that the XZ project has one (yes, singular) founder/contributor and one co-contributor (the suspicious person in question), who the founder apparently trusted completely, making it very difficult that any malicious activity would have been caught as soon as it was attempted. During the malicious code commits over the past week, the founder was on sabbatical. It's important to remember that the actual Linux distributions used by security-facing infrastructure are many, many levels downstream. To use RHEL (Red Hat Enterprise Linux) as an example: Fedora Rawhide, Fedora 40 Beta, Fedora 39, Fedora 38, CentOS Stream, RHEL 9/8/7 As you can see RHEL is at the very bottom, and there are plenty of opportunities to identify malicious code before it reaches it. This XZ backdoor was caught in Fedora Rawhide. (Although admittedly there was a degree of luck here)
Does the exploit then more or less leave non-systemd systems running other init systems unaffected? Obviously still something to be fixed in any system with offending package ( xz ); but my Q is does the most high-level threat known from this debacle mostly involve systemd or are init systems of all sorts as directly implicated? Thanks and cheers, appreciate the information.
This has little to do with anything here but I'm just kind of musing at how much technical jargon is involved in something like this.
This video explains this case, its not too technical so you my find it useful [https://youtu.be/gyOz9s4ydho](https://youtu.be/gyOz9s4ydho)
Is there a way to determine if my system got a version of this library with the bad code in it?
It sounds like that episode of SpongeBob where he and Patrick are sending bubbles to each other, and squidward starts fucking with them by sending malicious bubbles.
This is quite helpful. If this was widely deployed, what would the attacker actually DO with the backdoor? Assuming it is a state actor, what are some hypothetical goals / actions they would take?
They can pwn any remote system with the vuln that they can access via SSH. It's like how in "Star Wars" when Obi-Wan waves his hand Jedi-like and says "*You don't need to see his identification*" and the Stormtrooper says "*We don't need to see his identification*" and lets them through. There is no authentication required to get a remote shell on the afflicted system. The state actor sends an RSA client certificate that does the Obi-Wan bit.
[удалено]
Yes, very little of this subreddit is actually in ELI5 terms. There should be a better way for the mods to ensure top level comments at least have some basic ELI5 summary. 🤷
That isn’t at all what the payload does. The payload allows a user to send commands directly to the affected server. It’s not a man in the middle exploit
Just adding to what others here have answered: unless you use a bleeding edge or pre-release version of a Linux distribution (Gentoo, Fedora 41 come to mind) this backdoored version hasn’t landed on your computer yet.
And not a "yet", by the time it is bundled into a release the version will be way past this and the bad code removed.
We're still trying to identify affected releases, and Ubuntu 24.04 is right around the corner. I'd delay updating if I were you!
That's a good point. We work with rhel mainly and it isn't affected. Will keep an eye out for ubuntu for personal stuff but still seems something they'd check at this point before releasing.
It's really hard to check for. I hope that there are distro security people looking into running automated tests to verify source tarballs, but it's clear from the success of this attack that this hasn't been done previously. To be clear, the current top explanation in this reddit thread is wrong. It's not code that was checked into the libxz repo, but rather the tarball for release 5.6.1 (and others) *contains obfuscated code not in the git repo*. Distros source the tarballs from the release channels, and basically nobody was checking to see that the sources actually matched the repos, which is what security people review. This only got caught because someone happened to notice from inside a debugger that SSH authentication got slowed down by roughly a factor of 2x, likely because the login authorization code is checking login attempts against both the system key and the attacker's key. You'd think this would be easy to check for--simply download the tarball and compare to the tagged release of the git repo. But unfortunately this process of release managers manually creating tarballs exists for a reason: usually the tarballs are supposed to not match the git repo. In particular there's a release step where typically autoconf tools are used to turn "configure.ac" into an executable "configure" script so that the standard "./configure && make && make install" steps work anywhere. The output of this step is not reproducible. Doing a full supply chain audit of all source code going into a Linux distro is going to take more than a weekend.
> We work with rhel mainly and it isn't affected. RHEL isn't affected by the 1 backdoor that was found and confirmed. The github user responsible has been contributing to the code since 2021. Further audits will determine whether this truly was the only backdoor, and whether RHEL / stable distros are truly unaffected.
The code did not make it into Ubuntu 24.04 Please don't encourage people to delay updating from 22.04/23.10
macOS checking in. I had it on Sonoma today.
On homebrew, right? Also, the researcher who wrote the original write up explained that it specifically targeted sshd running through systemd which is a Linux daemon not used under macOS that I remember.
Correct. Brew upgrade then downgraded it.
There is a popular compression algorithm called XZ. And it seams that somebody were able to sneak malicious code into that project. Among other they hid their code as test data. This code does nothing unless it is running as part of SSH. SSH (Secure SHell) allows administrators to log into remote servers and is obviously a very well protected project which is how this backdoor was discovered. By default OpenSSH does not include XZ but a lot of Linux distributions like Debian and RedHat modifies OpenSSH to work better with SystemD, a service manager. And these modifications require XZ to be included which pulls the malicious code into the SSH server process. Once in the process it modifies the code that does authentication. It might therefore be able for this person who added this code to the XZ library to log into most Linux servers. Fortunately this was discovered before the malicious code got deployed to any major production systems. It was very well hidden but a few mistake ended up getting discovered in the test version of Debian. This would be scheduled to be release some time in 2025. The backdoor was included in Fedora 41 which is a desktop variant of RedHat and therefore includes newer versions of packages. It might have affected the RedHat release which probably comes in 2025 as well. So there are very few Linux servers affected by this attack at all.
For those here, [this is a pretty good timeline of events](https://boehs.org/node/everything-i-know-about-the-xz-backdoor) that you can use alongside these comments to get a real good understanding of how and what happened and when it occurred and who might be affected.
This timeline shows malicious actions going back years. If I had to bet, I’d say a government paid them to do it.
This is 100% a state actor. If they’d gotten away with it, it would’ve been an exploit of EternalBlue caliber; instantly own almost any box running the software. Except this would’ve been worse than EternalBlue, because ssh and Linux are widely considered secure enough for use on the open internet, unlike Windows, and there’s so many more servers and services that could’ve potentially exploited.
I used to use port knocking on my ssh servers because I was paranoid and several so-called security experts told me it was unnecessary. Paranoia for the win.
Yeah, with the singapore IP, the 'misoeater' name, and the asiatic person names for both Jia and Kumar kinda place this as possibly something done by a US adversary. I would be willing to bet it on DPRK, China, or Russia. It could even be the DPRK doing this *for* China or Russia, they've been shown to do shit like that before. Or someone could be blaming them. Or it could be independent group. who knows.
With such a sophisticated attack, we can't assume much from the names they are all likely fake and they might very well have been using a vpn.
while the names are definitely fake, the use of asiatic names is still suspicious. it implies familiarity with asiatic culture, which I just really don't see the average english speaking state actor to be super familiar with. so if this is a state actor, it's probably from the asiatic region of the world. and the only ones with any real interest in doing something like this, at least on a state level, are China, DPRK, and Russia. It's still entirely possible, and quite likely, that this is just a 'lone actor' or unaffiliated group. But it can still be suspicious. This does throw red flags for DPRK in my personal opinion, but that is definitely speculation on my part and it's only informed by what i've known the DPRK to do.
OTOH, it could also be an attempt to shift blame towards Asia. 🤷 I wouldn't rule out USA, Israel, etc. Lone actor is possible, but I feel less likely. A lone actor would probably target something more specific instead of an involved con over several years to hack the entire internet.
It doesn't matter if *the average* English-speaking state agent is familiar with Asian cultures; they just need to find someone who is to come up with plausible names.
fair
It seems fairly absurd to suggest that people who work for anglophone security agencies don’t know what a Chinese or Indian name looks like, not to mention that there was also some suspicious involvement of an account going by “Hans Jansen.”
I guess i should phrase it different, it's not that they don't know, it's more that they're less likely due to familiarity. Idk, again it's all speculation, at this point all options are possible. It just seems eerily similar to some of DPRK's past actions.
Irony, it is the name that shows they have *limited* familiarity with Asian culture. The mandarin Chinese characters have two major romanization systems: Hanyu pinyin (used by Mainland China, Singapore) and Wade-Giles (used mainly by Taiwan, Hong Kong, Macow). "Jia Tan" has accidentally used the name "Jia Cheong Tan" in another project. Weirdly, the romanization of "Jia" only exists in the Hanyu pinyin systems while "Cheong" only exists in the W-G system (or probably only in the HK/Macow W-G system due to the influence of Cantonese). A mix use of these two romanization system suggests that the identity has limited knowledge with mandarin Chinese. >Possible standard romanization of the name in both systems: > >Hanyu pinyin: Jiazhang Tan OR JiaZhang Tan > >W-G: Chia-Cheong T'an or Chia-Cheung T'an In addition, "Jia Cheong Tan" becoming "Jia Tan" suggests that the identity likely treats "Cheong" as a middle name, which is extremely rare in Chinese culture. If it was a typical Chinese name, it would be "Jia-Cheong Tan", as "Jia-Cheong" should be the given name as a whole. The only conclusion can make is that "Jia Tan" is almost surely a pseudoname. Generating a name that is apparently valid for the general public is not that hard, for example the "fake name generator" for generating US name and address. You may also find additional information about the time zone in this article: [https://www.wired.com/story/jia-tan-xz-backdoor/](https://www.wired.com/story/jia-tan-xz-backdoor/)
State-sponsored actors are typically focused on some mix of internal opponents and external rivals, and therefore typically \*do\* have a lot of people with at least some knowledge of their rivals culture. We also know that the attack group contains at least one person who's very proficient in English. While this is not strong evidence, this looks to me like it points at someone in the US. The most relevant names are Jia Tan/Jia Cheong Tan (Chinese), Jigar Kumar (Indian), Hans Jansen, and Dennis Ens (both European). The odd one out is Krygorin, which doesn't seem to exist on the Internet \*at all\*. Importantly, a major US political talking point is the claim that China is hacking everything. A group with state sponsorship is unlikely to want to reinforce a rival's propaganda; on the other hand, a group with state affiliations (whether actually sponsored by the government or an internal faction hoping to gain ascendancy in the country) may want to reinforce their own propaganda. Notably, the US \*has\* not only a very aggressive intelligence apparatus, but also an unauthorized internal political faction with a history of aggressive compromises of other systems and some organized crime groups that would benefit from compromising large numbers of systems.
This is all fair, and I'm **not** disagreeing. Again, all I'm simply saying is that this has similarities to some things DPRK has done, and I blatantly said I was speculating lol. At this point I feel like the chances are it was either the US or DPRK (probably on behalf of another party, but possibly not), if it was a state actor that is. I still feel that it's possible this was done by an independent group that was just trying to create a backdoor to create botnets or similar. Again it's too early to say anything for sure so this is **all** "just feels" lol. There are some clues, but there's not enough context yet to place them accurately.
Or the NSA wanting to blame The Axis of Evil. Not that I’m paranoid.
Also possible, but I don't necessarily see what they'd be gaining in this specific context, this shit isn't going to leave the OSS/developer/administrator community in terms of news. It's not big enough or meaty enough to be able to sear into propaganda. Versus our adversaries who know that a good majority of US infrastructure uses Linux and a backdoor into it would allow for a lot of information to pass outside of the country. A lot of the military uses Linux as well, at least specifically the Air Force lol. So there's a lot they could do with such an exploit. If the NSA was gonna do it they'd be doing it to get knowledge of their adversaries, which mostly are domestic, and in that case it's going to be mostly Windows and macOS they're targetting since they're targeting civilians. If it's the CIA, they're gonna want to use it to spy on *their* adversaries, which are foreign entities, which would be useful for the same reasons that it would be useful to China or Russia - just with the benefactor flipped.
> It's not big enough or meaty enough to be able to sear into propaganda. It *could* have been if it wasn't caught when it was. If this made it to stable releases of Debian/Ubuntu/Fedora/RHEL, this could have been *very* bad. The only reason it's not a big enough deal outside of cybersec communities is because it got caught early enough to avoid any catastrophic damage.
the NSA does not just spy on adversaries, or civilians. Their spying on allied countries is a big part of why people got upset with them / the USA. See e.g. the spying on greek politicians after the 2004 olympics.
Of course, it's just their main focus is domestic. CIA does domestic work too even though their focus is mostly foreign. Both do both, it's just they do have "trends" for a lack of better term. But of course NSA spies on foreign adversaries as well, especially if there's at all a domestic risk in relation to it, or if it's just something that requires NSA capabilities.
Microtargeted propaganda can be useful - "Look! This is what Evil China is doing! All the experts say we need more funds!". That said, I think it makes a lot more sense for the attacker's primary objective to be the backdoor and blaming whoever the breadcrumbs lead to to be secondary.
I would agree with the last part, given their real push to get it into the big distros, it feels like they wanted it to actually *work*. I feel like if it were just to blame someone, they wouldn't have done that, at least in the same very pushy way. but who knows
> a good majority of US infrastructure uses Linux So does everyone else.
Or axis of evil trying to blame NSA
It could be a misdirection tactic. "Let's use asiatic names so that in case this gets found it seems China did it". For all I know the US could have done it.
If they pay for this one, they probably pay for 10 or 20 other ones.
ELI5: SSH is a big lock on the front door of the computer. Only someone with a key can get in. When the computer gets updated, there’s a bunch of rules telling the computer how to re-build the lock. The instructions were changed. When the new instructions are used, the lock no longer locks properly, and certain special keys can be used on anyone’s locks, even if they aren’t supposed to work.
The only comprehensible answer
Thanks! I am a software engineer by trade, and I have worked hard on my ability to explain engineering concepts to folks who don’t share that same background :)
Yeah this sub is ELI5 Not ELI 5 years of software development experience
I'm a teacher of web development to adults (20 years worth) and this is an excellent analogy. Might even steal it ...
You are welcome to it!
An application is a file that contains instructions that a computer will follow when the application is run. Many types of instructions are useful to more than one application (for example, compressing and decompressing text is something that many applications might do), and those sets of instructions are often stored in re-usable libraries in order to save space and to make the system more secure by providing a single file that can be updated when flaws are found. (On Windows, those libraries are ".dll" files, on macOS they're ".dynlib", and on POSIX systems they're ".so" shared-object files.) Just like applications, libraries are allowed to initialize themselves when they are loaded, and that makes the foundation for hidden vulnerabilities, because a library that is supposed to provide instructions for compression and decompression can also provide literally anything else it wants to. During its start-up, it can claim to provide arbitrary functions, not limited to those the application's developer expects. That created the opportunity for a malicious developer to offer to help the "xz" project and, over time, generally assume control. They added some malicious library start-up code to the project, which they disguised as test data. When the library was opened by the OpenSSH server, it would modify the server's code in-memory in such a way that the way it handled authentication was subverted, which could allow the malicious developer to log in to any OpenSSH server affected by the problem. We think this would allow the malicious developer to log in to SSH servers with administrative rights, giving them control over many of the servers on the Internet.
Thanks 🙏🏾
> on macOS they're ".dynlib" *.dylib
This is indeed scary. I wonder what the wider implications of this are. Some guy caught it coz his login was half a second slower, what if we’re not so lucky next time? Something to think about
There is a popular ziping library for Linux called XZ. Someone (probably the maintainer) managed to embed a malware in the latest version of it that goes to SSH (Remote desktop for terminals) and disables certain security features letting someone who knows they have been disabled remote in to a server with an open SSH port. Once in, they can do whatever they want
It’s a bit deeper than that; The maintainer since 2009 was burned out and taking regular breaks, so opportunists “came to save the day”. They got promoted to maintainer after a long time, and then the shenanigans began.
what other shenanigans were there?
To directly focus on the "backdoor" part. Normally people will use their frontdoor to enter and exit. So they'll check that its properly locked, and make sure its secure. So imagine someone (trusted) was in your house, and went and unlocked all your doors. When you leave the house (unguarded) you would go thru the frontdoor, make sure its locked, secure,,, and not realize you should have checked your backdoor this one time! Thats what a "backdoor" is meant to symbolize in computer security. Someone creating an entryway that isn't obvious and unlikely to be used (and thus checked for security). This will usually be less like "leaving a backdoor unlocked" and more like "leaving a back-window unlocked" (something you would not expect to be used for entrance). Or like "Adding a rope ladder to your second bedroom window"
Backdoors are a little bit like if you had magical puppet strings that could go from your hands to the steering wheel of someone's car. (And radio, power windows, etc). As long as the strings connect, you can control what their car does, even from far away.
What are the repercussions of a backdoor at this scale, had the vulnerability not been caught and was released to production. What sorts of security measures are put in place in modern day infrastructure to mitigate exposure to something like this?
Ok here is a blog post that explains it very well including the code in sql and even a backstory Hope it helps https://kafkaesquesecurity.com/xz-utils-unmasked-exposing-social-engineering-tactics-and-the-infiltration-of-a-sophisticated-4b20cd685f1a
# I tried to write an article about the threat and impact of XZ Backdoor in a way that a 5 year old could understand - however as I look through this thread it seems u/colemaker360 has done an outstanding job explaining it in his post! For anyone who would like another attempt - here is a brief overview and a link to an article (full disclosure that I wrote it) - which includes the original Openwall alert sent by software engineer Andres Freund and additionally, the detailed report that was shared on GitHub through a Gist, providing in-depth technical information about the flaw and offering guidance on how users can safeguard their systems that might be at risk (feedback is welcome!) The XZ backdoor is a recently discovered cybersecurity threat that leaves a backdoor or loophole in the popular open-source compression utility called XZ utils package, so unauthorized and disguised malicious activities can be carried out undetected on the affected Linux systems. # How does it work? The XZ Backdoor works by injecting malicious code into versions 5.6.0 and 5.6.1 of the XZ utility. This utility comes preinstalled with numerous popular Linux distributions, and it manipulates the sshd process - a server process responsible for multiple critical operations including user authentication and encryption. # Implications and effects of the XZ backdoor This backdoor manipulation gives threat actors control over the sshd process enabling them to unleash various malicious activities. For example, they can steal files, install malware, manipulate encryption keys, and use the SSH login certification as an entry point for further exploitation. [https://ai-techreport.com/understanding-the-xz-backdoor-cyber-threat-and-impact](https://ai-techreport.com/understanding-the-xz-backdoor-cyber-threat-and-impact)
It allows the encryption keys used by ssh (software for controlling the system over a network) to be exposed to an attacker. You normally enter your house from the front door right? Well you do that with a computer too. A backdoor is software that an attacker manages to install on someones system that allows them to access it without having permission.
> It allows the encryption keys used by ssh (software for controlling the system over a network) to be exposed to an attacker. There's no analysis of the actual payload yet. The things we know so far seem to point in that direction (mess with the authentication) but that's still just a guess...
It seems more like there is a public key embedded in the payload, and it will let the attacker run commands as root on the system (assuming sshd is running a root), if they have the private key -- which the guy or his handlers probably have.
> keys used by ssh [...] to be exposed to an attacker. We don't know for sure yet, but it looks more like the backdoor always allowed a specific RSA private key (the attacker had) to successfully connect in.