| View previous topic :: View next topic |
| Author |
Message |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3347
Location: Rasi, Finland
|
 Posted: Sun Apr 07, 2024 12:31 pm Post subject: |
 |
|
| logrusx wrote: | what you've linked looks like a pointless amateur job.
Best Regards,
Georgi |
That test shows seemingly random environment variable causing sshd to start much faster. To me that's worth investigating and certainly no pointless.
| jpsollie wrote: | | would putting this in /etc/profile be enough to make sure the kill switch is always trigged (and thus the malicious code never executed)? |
I wouldn't trust it. We only know (based on the source you gave) it makes sshd start faster.
To me it's strange that the attacker would leave an "exploit" to close the backdoor.
To his/hers own systems? There are easier ways to do that.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3140
|
| Posted: Sun Apr 07, 2024 1:24 pm Post subject: |
|
|
Yeah, why bother with setting variables, when you can just use an unaffected version?
Anyway, at least it seems that no real damage has been done, even though it could have been a total disaster.
Debian and redhat + ssh sounds like servers, and those those typically use stable releases. Bugged xz hasn't been pushed to stable yet.
If it was spotted thanks to random performance tests on a bleeding-edge installation which was exposed to the botnets bruteforcing passwords, that's a very lucky event. At this point I wonder if restricting that backdoor to debs and rpms wasn't done deliberately to hide it from testers on rolling releases.
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20067
|
| Posted: Sun Apr 07, 2024 2:28 pm Post subject: |
|
|
| jpsollie wrote: | Hello everyone,
My laptop/desktop/NAS have been using the xz-utils 5.6.0 package right from the start.
As such, they were potentially affected.
When this CVE was published, I immediately downgraded XZ, but:
1. Does it affect openssh binaries compiled with XZ as well? or only at runtime?
the idea is: let's assume I compiled a gentoo package for amd devices when xz 5.6.0 was being used,
will installing that precompiled binary automatically install the backdoor as well?
2. How can I activate the kill switch?
I read here: https://piaille.fr/@zeno/112185928685603910 that the following kill switch exists:
| Code: |
yolAbejyiejuvnup=Evjtgvsh5okmkAvj
|
would putting this in /etc/profile be enough to make sure the kill switch is always trigged (and thus the malicious code never executed)? |
One of the links i posted in your other thread answered question 1...
https://glsa.gentoo.org/glsa/202403-04
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
jpsollie
Apprentice
Joined: 17 Aug 2013
Posts: 291
|
| Posted: Sun Apr 07, 2024 2:35 pm Post subject: |
|
|
| pjp wrote: | | jpsollie wrote: | Hello everyone,
My laptop/desktop/NAS have been using the xz-utils 5.6.0 package right from the start.
As such, they were potentially affected.
When this CVE was published, I immediately downgraded XZ, but:
1. Does it affect openssh binaries compiled with XZ as well? or only at runtime?
the idea is: let's assume I compiled a gentoo package for amd devices when xz 5.6.0 was being used,
will installing that precompiled binary automatically install the backdoor as well?
2. How can I activate the kill switch?
I read here: https://piaille.fr/@zeno/112185928685603910 that the following kill switch exists:
| Code: |
yolAbejyiejuvnup=Evjtgvsh5okmkAvj
|
would putting this in /etc/profile be enough to make sure the kill switch is always trigged (and thus the malicious code never executed)? |
One of the links i posted in your other thread answered question 1...
https://glsa.gentoo.org/glsa/202403-04 |
your URL mentions a "as far as we currently understand, gentoo isn't affected".
Let's assume there's a part beyond our understanding.
For people knowing how this backdoor works, would the backdoor be needed in the runtime lzma library to be vulnerable? or will it already infect the code when we are unpacking the .tar.xz source file to compile it?
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img] |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21650
|
| Posted: Sun Apr 07, 2024 3:08 pm Post subject: |
|
|
| szatox wrote: | | Yeah, why bother with setting variables, when you can just use an unaffected version? |
Per company/state security policy, the attacker is required to run the latest available upstream release of a well-supported mainstream distribution, and not to substitute in custom binaries, so it would be a policy violation to run an unaffected version. Due to a policy oversight, using an unpublished optional setting to disable the catastrophic security weakness is not against policy, so the attacker included this as a way to protect his own systems.  | szatox wrote: | | At this point I wonder if restricting that backdoor to debs and rpms wasn't done deliberately to hide it from testers on rolling releases. |
The backdoor was somewhat fragile even on the intended target systems, so I could see that, or the related motivation of minimizing the matrix of supported systems. If the backdoor was sufficiently unstable that it started provoking widespread crashes when installed on a rolling distribution, that would bring a great deal of unwanted attention very quickly. Given some of the things that Gentoo users can do with exotic CFLAGS, I could believe that if the backdoor were allowed to run on Gentoo, then someone on Gentoo might accidentally create a configuration that provokes a crash bug in the backdoor. | jpsollie wrote: | your URL mentions a "as far as we currently understand, gentoo isn't affected".
Let's assume there's a part beyond our understanding. |
If we assume that there are parts beyond our understanding, can we not then assume one of those parts is an AI that knows how to infect every program on your computer, inject itself into the firmware for good measure, and then start a chain letter campaign to propagate itself? Worrying about what the attacker might have hidden in parts we have not yet understood allows an almost unlimited number of scenarios. Personally, I think the attacker would have prioritized first spending more effort making this reliable (no Valgrind warnings, no crashes even on unsupported configurations), then focused on making it work on more systems (e.g. arm64), and only then maybe considered giving it substantial new capabilities. | jpsollie wrote: | | For people knowing how this backdoor works, would the backdoor be needed in the runtime lzma library to be vulnerable? or will it already infect the code when we are unpacking the .tar.xz source file to compile it? |
The best reporting I have read asserts that the backdoor consists of a non-obvious modification to the xz-utils build system, which modifies the source of the xz-utils package being built to add the logic that triggers when sshd loads it in a supported target configuration. I have seen no reports of the malicious payload being able to patch any build system other than its own. I have seen no reports of the malicious payload being able to take action outside of responding to a well-formed exploit from someone who knows information that the attacker has so far managed to keep secret. |
|
| Back to top |
|
|
sam_
Developer
Joined: 14 Aug 2020
Posts: 1678
|
| Posted: Sun Apr 07, 2024 4:54 pm Post subject: |
|
|
| I don't recommend setting arbitrary strings from a not-fully-reversed payload. |
|
| Back to top |
|
|
pa4wdh
l33t
Joined: 16 Dec 2005
Posts: 812
|
| Posted: Sat Apr 13, 2024 8:53 am Post subject: |
|
|
If i look at my distfiles gentoo seems to use xz quite a lot, does this incident (and maybe this info) change that?
This topic seems to suggest something is changing, but it's not stated explicitly.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 324
Location: Naarm/Melbourne, Australia
|
| Posted: Sat Apr 13, 2024 9:26 am Post subject: |
|
|
| pa4wdh wrote: | | If i look at my distfiles gentoo seems to use xz quite a lot, does this incident (and maybe this info) change that? |
i'm not a Gentoo dev. But upthread, i wrote:
| flexibeast wrote: | Although i'm very sympathetic to the critique of the design and implementation of xz, those things aren't the fundamental cause of the current situation: fundamentally, this is a social engineering attack.
Say lzip had been used in place of xz; great. It becomes widely-used, which means there's increasing pressure on Diaz to deal with bugs and feature requests. Then Diaz starts to develop health issues which make it even more difficult to deal with those bugs and feature requests. There will be an incentive for Diaz to take on extra maintainers, and a susceptibility to being brigaded into doing so, especially when a proposed maintainer seems to have been doing good work for a couple of years.
These sort of issues will continue to arise until we work out an effective way of dealing with "the xkcd 2347 problem" ("Dependency"). |
and in a later comment:
| flexibeast wrote: | Both crowds - and in fact many FOSS users more generally - are willing to harass, insult, and encourage brigading of devs and maintainers until they get their way. Yet it's clearly not possible to always please both crowds simultaneously, or all users in general. And when devs and maintainers don't, further harassment, insults, and brigading are not uncommon.
In this overall context, if Lasse experienced brigading pressuring him to take on a new maintainer for xz-utils, it doesn't necessarily come across as "unusual behaviour, perhaps even the work of an APT", thus raising red flags; instead, it doesn't seem like particularly unusual behaviour at all. EDITED TO ADD: Indeed, there were two accounts, apparently sockpuppets, pressuring Lasse, in ways that i encounter regularly, across various FOSS projects.
Consider how that impacts volunteers' time, resources, and mental health, and how that might impact volunteers' ability to provide and maintain software and the QA (including security stuff) related to it. |
(The "EDITED TO ADD" part includes a link to Russ Cox's timeline of how this situation came about.) |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54270
Location: 56N 3W
|
|
| Back to top |
|
|
pa4wdh
l33t
Joined: 16 Dec 2005
Posts: 812
|
| Posted: Sat Apr 13, 2024 2:24 pm Post subject: |
|
|
Thanks for your answers.
I didn't intent to say that the "design flaws" are somehow related to the backdoor incident, but together they make an inferior product with a bad reputation which might be a good reason to reconsider. In the end, it's impossible to have a system without xz as long as a majority of the distfiles are distributed in xz format.
| Quote: | This info is another case of Betamax vs VHS, or possibly 8-Track vs Cassette, if your memory goes back that far  |
Yes, my memory goes back that far (i'm from the 70's)
I've always been told that a certain specific genre of movies was the main reason VHS won the race.
The link i gave had -as far as i can judge- pretty solid arguments to say xz suffers from some bad design decisions.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54270
Location: 56N 3W
|
| Posted: Sat Apr 13, 2024 3:47 pm Post subject: |
|
|
pa4wdh,
I agree with the technical points that the link makes. I've seen it before.
The point I was making with Betamax vs VHS is that technical merit is not a good discriminant.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
pa4wdh
l33t
Joined: 16 Dec 2005
Posts: 812
|
| Posted: Sat Apr 13, 2024 5:05 pm Post subject: |
|
|
I think software is different from a video system.
We could switch from xz to gzip or bzip2 tomorrow without any costs (in contrast to buying a new device that supports a different standard). Even if we only start compressing new distfiles (so we doni't waste a lot to resources re-compressing everything), the need for xz will be gone in a short time. If there's no downside to switching and better alternatives are available, why wouldn't we? I guess a similar reasoning could have been used to switch xz in the first place
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3140
|
| Posted: Sat Apr 13, 2024 5:54 pm Post subject: |
|
|
xz tends to achieve higher compression ratio than gzip or bzip, in some cases by miles. That's a pretty damn good reason to switch.
Say, I think moving from gzip to xz halved the size of log bundles on web servers I had in my care.
How many implementations with varying feature sets of xz actually exist?
Yes, I agree that allowing tones of customization to a compressor is bad for interoperability and ultimately doesn't make any sense. Is anyone actually pumping out those incompatible implementations though, or can I expect future versions to still read the old archives?
_________________
Make Computing Fun Again |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54270
Location: 56N 3W
|
| Posted: Sat Apr 13, 2024 5:58 pm Post subject: |
|
|
pa4wdh,
| Quote: | | ... tomorrow without any costs ... |
There is a great deal of inertia here and its not zero cost.
ebuild manifests need to be updated. Mirrors need to be updated. Upstream needs te be convinced so Gentoo does not do all the work.
It was brought up in https://marc.info/?l=gentoo-dev&m=171176789702553&w=2. which has a very low signal to noise ratio, so I'll refer your there for the pros and cons.
In that email thread, the driver was the backdoor, not technical merit but the costs for the change are the same.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1550
|
| Posted: Sat Apr 13, 2024 5:58 pm Post subject: |
|
|
There is no reasonable need to remove xz, to re-compress or, to switch compression of distfiles.
Having an xz archive does not mean anything. Having xz does not mean anything either. The archive itself does not contain the backdoor, the xz sources neither. And I guess the use of those controversial functions is already removed, so xz cannot be used to replace code anymore.
Best Regards,
Georgi |
|
| Back to top |
|
|
Ralphred
Guru
Joined: 31 Dec 2013
Posts: 501
|
| Posted: Sat Apr 13, 2024 10:25 pm Post subject: |
|
|
| I've not taken any notice of this since "It's not a RAT, it's RCE" was opined by people smarter than me: Has anything noteworthy been made known since this? TIA. |
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1550
|
| Posted: Sun Apr 14, 2024 6:10 am Post subject: |
|
|
| NeddySeagoon wrote: | pa4wdh,
| Quote: | | ... tomorrow without any costs ... |
There is a great deal of inertia here and its not zero cost.
ebuild manifests need to be updated. Mirrors need to be updated. Upstream needs te be convinced so Gentoo does not do all the work.
It was brought up in https://marc.info/?l=gentoo-dev&m=171176789702553&w=2. which has a very low signal to noise ratio, so I'll refer your there for the pros and cons.
In that email thread, the driver was the backdoor, not technical merit but the costs for the change are the same. |
| Eddie Chapman wrote: | No one could have been expected to foresee what's happened with xz-utils,
but now that it's here, perhaps Gentoo (and other projects that do) should
consider not relying on a single decompression algorithm for source
archives, even just as an insurance against some other yet unknown
disaster with one algorithm or another in future? |
First this is an overkill and second, who's to say the next big thing will come through a compression library? This to me is all nonsense, this is paranoia and fear talking.
I sincerely recommend people to stop giving in to paranoia and fear and start thinking rationally.
Best Regards,
Georgi |
|
| Back to top |
|
|
Ralphred
Guru
Joined: 31 Dec 2013
Posts: 501
|
| Posted: Sun Apr 14, 2024 6:34 am Post subject: |
|
|
| logrusx wrote: | | start thinking rationally |
I was quite pleased to notice that whilst the plurality of the archives in my /var/cache/distfiles were *.xz, the majority were not, but YMMV obviously.
It's also worth noting that this exploit COULD NOT AFFECT GENTOO USERS as currently described; don't mistake this statement for complacency, just try and have some realistic perspective. |
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1550
|
| Posted: Sun Apr 14, 2024 8:35 am Post subject: |
|
|
| Ralphred wrote: | | I've not taken any notice of this since "It's not a RAT, it's RCE" was opined by people smarter than me: Has anything noteworthy been made known since this? TIA. |
Actually it is both RAT and RCE at the same time, being able to give access to somebody with a particular private key and in the same time giving the possibility to replace running code and even update itself. However it isn't anymore. And it's highly unlikely it was ever used. Especially on Gentoo, unless somebody went out of their way to accommodate it. If fact if someone finds a Gentoo install that's been modified in such a way, there would be reasonable suspicion it was a malicious act.
Now we can't be sure but it looks like it was targeted at Redhat and Debian themselves. To me it appears what the attackers wanted was to be able to access their servers and any other victims were just a bonus, if at all object of interest.
Also after this xz-utils will most likely be the most scrutinized opensource package ever. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3347
Location: Rasi, Finland
|
| Posted: Sun Apr 14, 2024 2:10 pm Post subject: |
|
|
| logrusx wrote: | | and second, who's to say the next big thing will come through a compression library? |
Exactly.
We (open source community) just need to be awake... just like we did in this case. That backdoor was found pretty quickly and it only managed to spread into most bleeding edge distros and only if certain conditions were met.
I find it amusing that people are now actively looking for alternative compressors. I mean, the affected versions are now removed or blocked on every distro. It is very unlikely anyone would hack xz again.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 324
Location: Naarm/Melbourne, Australia
|
| Posted: Mon Apr 15, 2024 12:45 am Post subject: |
|
|
| Zucca wrote: | | I find it amusing that people are now actively looking for alternative compressors. I mean, the affected versions are now removed or blocked on every distro. It is very unlikely anyone would hack xz again. |
Indeed. So many people seem to be assuming this backdoor was based on a flaw in the design and code implementation of xz, or want to believe that in order to push their lzip barrow. But the backdoor wasn't based on that: it was based on two years of social engineering, which over time facilitated manipulation of the build system (gory details).
As i've said upthread, i'm sympathetic to the the arguments against the design and implementation of xz, and for the use of lzip instead, but when lzip-barrow-pushers misunderstand or misrepresent what's happened with xz, and don't seem able to explain how moving to lzip will magically avoid social engineering and build system attacks .... i feel that doesn't help the lzip cause. |
|
| Back to top |
|
|
Conditional_Zenith
Apprentice
Joined: 22 Sep 2004
Posts: 157
Location: Australia
|
| Posted: Mon Apr 15, 2024 4:27 pm Post subject: |
|
|
I know this is jumping back a bit in this thread, but wouldn't stripping binary test blobs from the build directory before build have made the attacker's job a lot harder?
Binary test blobs are sometimes necessary if they are for a tool that deals with binary files. Ideally they should be generated using a documented and repeatable process, but from what I understand we are a long way from that in upstream projects. But what would be the harm of a distributor deleting binary test blobs (at the beginning of say the configure step) before running anything? I guess the distributor may want to run the tests after but they could run a fresh unpack after building but before running the tests. I get lots of other things went wrong in this case, but getting rid of binary test blobs seems like a low risk step to make an attack like this harder next time. |
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1550
|
| Posted: Mon Apr 15, 2024 4:43 pm Post subject: |
|
|
| Conditional_Zenith wrote: | I know this is jumping back a bit in this thread, but wouldn't stripping binary test blobs from the build directory before build have made the attacker's job a lot harder?
Binary test blobs are sometimes necessary if they are for a tool that deals with binary files. Ideally they should be generated using a documented and repeatable process, but from what I understand we are a long way from that in upstream projects. But what would be the harm of a distributor deleting binary test blobs (at the beginning of say the configure step) before running anything? I guess the distributor may want to run the tests after but they could run a fresh unpack after building but before running the tests. I get lots of other things went wrong in this case, but getting rid of binary test blobs seems like a low risk step to make an attack like this harder next time. |
Trying to fix the issue by addressing a particular case is never going to work. That's all I have to say for all the suggestions that this backdoor gave rise to.
And there's no technical issue really and therefor it can't be addressed with technical measures. It's said time and time again that was a social engineering.
All attempts to fix it with technical measures are doomed to fail.
In a way this backdoor is a blessing to all of us as it teaches us lessons. Unfortunately too many people refuse to learn them and cling to the old ways from before this type of attacks existed.
Best Regards,
Georgi |
|
| Back to top |
|
|
Conditional_Zenith
Apprentice
Joined: 22 Sep 2004
Posts: 157
Location: Australia
|
| Posted: Mon Apr 15, 2024 5:21 pm Post subject: |
|
|
| logrusx wrote: | | Conditional_Zenith wrote: | I know this is jumping back a bit in this thread, but wouldn't stripping binary test blobs from the build directory before build have made the attacker's job a lot harder?
Binary test blobs are sometimes necessary if they are for a tool that deals with binary files. Ideally they should be generated using a documented and repeatable process, but from what I understand we are a long way from that in upstream projects. But what would be the harm of a distributor deleting binary test blobs (at the beginning of say the configure step) before running anything? I guess the distributor may want to run the tests after but they could run a fresh unpack after building but before running the tests. I get lots of other things went wrong in this case, but getting rid of binary test blobs seems like a low risk step to make an attack like this harder next time. |
Trying to fix the issue by addressing a particular case is never going to work. That's all I have to say for all the suggestions that this backdoor gave rise to.
And there's no technical issue really and therefor it can't be addressed with technical measures. It's said time and time again that was a social engineering.
All attempts to fix it with technical measures are doomed to fail.
In a way this backdoor is a blessing to all of us as it teaches us lessons. Unfortunately too many people refuse to learn them and cling to the old ways from before this type of attacks existed.
Best Regards,
Georgi |
"Never going to work". "Doomed to fail". Security is a continuum, not a binary. The obvious response is to ask what your definition of "work" and "success" is, as no security measure is 100% effective. Not even supporting maintainers and getting rid of toxic "users" can 100% prevent this next time. Maintainers still move on or retire. This backdoor doesn't change the fact that defense in depth is still the most effective way to address issues.
Yes social engineering is one of the things that went wrong, arguably even the biggest. That doesn't mean we should give them an easy way to deliver obfuscated binary blobs into build processes if preventing them from doing so is low cost. |
|
| Back to top |
|
|
logrusx
Veteran
Joined: 22 Feb 2018
Posts: 1550
|
| Posted: Mon Apr 15, 2024 5:25 pm Post subject: |
|
|
| Conditional_Zenith wrote: | | That doesn't mean we should give them an easy way to deliver obfuscated binary blobs into build processes if preventing them from doing so is low cost. |
I'm sorry, what's the easy way again? How many packages have you have identified using it?
Best Regards,
Georgi |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
