Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The xz package has been backdoored
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2575
Location: Here and Away Again

PostPosted: Tue Apr 02, 2024 11:39 am    Post subject: Reply with quote

colo-des wrote:
The firsts tools and listings of the objects (liblzma_la-crc64-fast.o) are appearing:
https://www.openwall.com/lists/oss-security/2024/03/31/5
https://github.com/karcherm/xz-malware

For PAM's defenders, there are some gifts.
Code:

948: 'mm_answer_pam_start\x00'
40:  'mm_do_pam_account\x00'
70:  'mm_sshpam_free_ctx\x00'
58:  'mm_sshpam_init_ctx\x00'
60:  'mm_sshpam_query\x00'
68:  'mm_sshpam_respond\x00'
c58: 'parse PAM\x00'

Keep expanding the attack surface and see how many more of these backdoors appear.
Greetings

All of those strings can be found in the 'sshd' executable (probably needs to be built with PAM support), so it might just mean they used parts of OpenSSH in there. I believe I saw someone somewhere also mention that.

See also:

- https://github.com/openssh/openssh-portable/blob/master/monitor_wrap.h#L74

Not that I'm defending or attacking the use of PAM (I don't use it). Just saying that some string being found there does not necessarily mean it's something they use as additional surface (but of course it also could).
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
modnaruved
Apprentice
Apprentice


Joined: 21 Mar 2011
Posts: 160

PostPosted: Wed Apr 03, 2024 12:56 am    Post subject: Reply with quote

Some important points that I would like to draw your attention to. I am sorry for my English.

The backdoor was identified during the detection of performance drawdown, although and insignificant.
Then, as a result of the review, a suspicious code. Moreover, the fact itself and the detection technique, the circumstances, also represent a big interest.
It probably makes sense to pay attention to the frequency of updates and generally an anomalies.
Obviously even now, it would be useful to continue conduct a comprehensive analysis and draw conclusions along the way.

Ideally there should be: a test cycle (in an isolated environment, but simulating typical environments) and "updates buffering"
Nice to have: CI, performance monitoring and analysis, identification anomalies. Something else.

Of course, it would be useful to analyze changes on a regular basis. Particular attention to analysis archives/tarballs, in particular generated ones.

Wherever there are junctions in the chain of changes during code delivery, they should especially check.
It is possible that tarballs/archives should be checked in the most thorough manner, schemes for forming tarballs on places, as well as their subsequent verification.

At least this should apply to system-wide packages and popular packages, packages on which a large number of packages depend and the necessary functionality, especially such as xz-utils.

Trust networks need some sort of diversification (in a good way)

The following are possible attack vectors (some): blobs, including tarballs and archives in general.
Back to top
View user's profile Send private message
Elleni
Veteran
Veteran


Joined: 23 May 2006
Posts: 1270

PostPosted: Wed Apr 03, 2024 4:28 pm    Post subject: Reply with quote

We have a manjaro VM that had no exposure of its ssh server to the internet, that had the affected version of 5.6.1 installed. Now when connecting to this vm through ssh (from within a secured corporate network, not from internet) - for example for administering it by an ansible server, is there any chance that the backdoor is spread to other machines, as long as they have not installed an affected ssh version anyway? Or is it enough to patch this client to a non-affected version of xz-utils?
Back to top
View user's profile Send private message
figueroa
Advocate
Advocate


Joined: 14 Aug 2005
Posts: 2963
Location: Edge of marsh USA

PostPosted: Wed Apr 03, 2024 4:39 pm    Post subject: Reply with quote

Courtesy of today's TLDR newletter, a timeline of the XZ attack:
https://research.swtch.com/xz-timeline?utm_source=tldrnewsletter
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3348
Location: Rasi, Finland

PostPosted: Wed Apr 03, 2024 10:07 pm    Post subject: Reply with quote

colo-des wrote:
I understand your point of view and respect your position, but I personally have to choose between ease of use and safety, I prefer to choose security.
Quote:
Rogues are very keen in their profession, and know already much more than we can teach them.
Your method is a bit like "Security trough Obfuscation", but I admit that you also reduce the attack vectors by not having tools found from common systems.
Minimal systems are good in that regard for example.

I mean you could have a some really obscure shell interpreter which only understands assembly commands. Then no malicious (normal) script could wreak havoc in your system, thus closing many attach vectors. But you'd sure give up the ease of use and compatibility for many other software.

I too don't use many of the softwares you listed, but I decided to keep pam just because it eases some things in my system.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
CaptainBlood
Advocate
Advocate


Joined: 24 Jan 2010
Posts: 3628

PostPosted: Wed Apr 03, 2024 11:09 pm    Post subject: Reply with quote

figueroa wrote:
Courtesy of today's TLDR newletter, a timeline of the XZ attack:
https://research.swtch.com/xz-timeline?utm_source=tldrnewsletter

+1
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here.
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. "
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1554

PostPosted: Thu Apr 04, 2024 9:15 am    Post subject: Reply with quote

Elleni wrote:
is there any chance that the backdoor is spread to other machines, as long as they have not installed an affected ssh version anyway?


No. It's a backdoor, not a virus.

Best Regards,
Georgi
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3348
Location: Rasi, Finland

PostPosted: Thu Apr 04, 2024 1:33 pm    Post subject: Reply with quote

logrusx wrote:
No. It's a backdoor, not a virus.
At this moment.

There could be more. Unlikely, but possible.
In my opinion unlikely, because xz is already very widely used software, then there's no need for a "feature" of infecting other hosts.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
Elleni
Veteran
Veteran


Joined: 23 May 2006
Posts: 1270

PostPosted: Thu Apr 04, 2024 5:20 pm    Post subject: Reply with quote

understood. Thanks for your replies, guys!
Back to top
View user's profile Send private message
modnaruved
Apprentice
Apprentice


Joined: 21 Mar 2011
Posts: 160

PostPosted: Thu Apr 04, 2024 6:42 pm    Post subject: Reply with quote

Thanks figueroa for providing interesting information.

During the reading, I thought: maybe this (backdooring) is done to distract attention? And/or is this a trial ball/balloon, one of?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21653

PostPosted: Thu Apr 04, 2024 7:35 pm    Post subject: Reply with quote

Considering the likely impact an attacker could have if this backdoor were successfully used, I think this was the intended payload, not a distraction. A distraction would need to be noticed to be distracting, and the author seems to have taken significant effort to go unnoticed.
Back to top
View user's profile Send private message
salam
Apprentice
Apprentice


Joined: 29 Sep 2005
Posts: 226

PostPosted: Fri Apr 05, 2024 8:51 am    Post subject: Reply with quote

Is it a common thing to mix source code and some binary mess? For me, this looks to be the biggest problem of the whole chain. The project cannot be considered open source any more. If foreign binary blob is allowed to be included in the compilation process, we can expect many more black boxes in linux hosts, doing random things.
Back to top
View user's profile Send private message
axl
Veteran
Veteran


Joined: 11 Oct 2002
Posts: 1144
Location: Romania

PostPosted: Fri Apr 05, 2024 9:23 am    Post subject: Reply with quote

There are all sorts of binary data in source trees these days. This is a compressing lib; it's not unnatural to think it comes with an archive or two to test its own functionality. I imagine libjpeg comes with a jpeg file too. libpng with a png file. And so on.

Frankly, some things about this backdoor are sophisticated and advanced enough to be scary. The patience. The social engineering. The way it hides itself when you try to debug. But on the other hand, there are aspects of this that do not sit right with me at all. Why, if you invested so much time and energy, would you limit yourself to just x86_64? And as sophisticated as it is in hiding itself, still, must be a major oversight to be able to find it just by doing time sshd --help.

sshd --help shouldn't take 800ms. People are going to notice that. I mean no insult to the person that found it, but, its not something that can pass easily without being found.

Also, this is a passive backdoor. It doesn't call home, it waits for home to call to it. So, you would think, if someone had the technological expertise to pull this off, and the time, and the resources, they would find something better than just affecting JUST ssh, on JUST x86_64? I mean, you wait for 2 years to infect the world, and then, you do it just 1/2 of what you could have done. Seems a bit weird. Many of us do not expose ssh to the world to begin with. And might add that some of the world has moved on arm, so seems weird to choose by yourself if you gonna do something bad, you only gonna do it to part of the world.

And third thing I want to mention is that I am not sure how much we will be able to learn ourselves from the backdoor and this whole incident. But, on the other hand, we are a very wealthy mine of information for the people that did this and are looking at us to see how we will react. I think, there are multiple entities reading everything that was analysed and said about the whole incident and are making notes on how to make it better next time.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3141

PostPosted: Fri Apr 05, 2024 10:54 am    Post subject: Reply with quote

Dunno, kinda smells like a targeted attack. Passively waiting for instructions suggests it's not about size, but how you use it. Calling home is much more visible than just sitting tight waiting for instructions.
Could also be done as the ground-work by one of the 3-letter agencies, something to be used later when a target presents itself, but probably _not_ a botnet.

Quote:
sshd --help shouldn't take 800ms. People are going to notice that. I mean no insult to the person that found it, but, its not something that can pass easily without being found.
Are they? I haven't touched sshd for years. I never knew it even recognized --help, because it's started automatically by system scripts.
I wonder what really led to the discovery that login takes too long. Was that one admin running ansible without pipelining for that modified login to become a performance problem?

Quote:
they would find something better than just affecting JUST ssh, on JUST x86_64? I mean, you wait for 2 years to infect the world, and then, you do it just 1/2 of what you could have done
Baby steps. Changing a bunch of tests at once in an already completed project is weird. A small change is easier to swallow. Creating a trail of small changes which haven't been questioned over time makes future changes easier to shrug of, because people get used to things being done certain way.
XZ has been complete for a very long time (seriously, what new feature would anyone want to add?), there is little to gain from developing it, and losing compatibility would be disastrous.
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
axl
Veteran
Veteran


Joined: 11 Oct 2002
Posts: 1144
Location: Romania

PostPosted: Fri Apr 05, 2024 11:45 am    Post subject: Reply with quote

szatox wrote:
XZ has been complete for a very long time (seriously, what new feature would anyone want to add?), there is little to gain from developing it, and losing compatibility would be disastrous.


I hear multi-core decompression was planned. since you asked. they have multi-core compression, but not decompression.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1554

PostPosted: Fri Apr 05, 2024 12:33 pm    Post subject: Reply with quote

Don't turn this into chat please.

Best regards,
Georgi
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21653

PostPosted: Fri Apr 05, 2024 12:42 pm    Post subject: Reply with quote

szatox wrote:
I wonder what really led to the discovery that login takes too long. Was that one admin running ansible without pipelining for that modified login to become a performance problem?
As I understood the original reporter's explanation, he was trying to get the system to be as quiet as possible for the benefit of a micro-benchmark he wanted to run. Random bots trying to break into sshd kept triggering the backdoor-infected sshd to run the comparatively expensive check to see if this knock was the attacker wanting the backdoor to activate. It wasn't, but validating that was slow enough that sshd was using "too much" CPU time, which piqued the reporter's interest.
axl wrote:
they would find something better than just affecting JUST ssh, on JUST x86_64?
Even on x86_64, there were systems where the backdoor did not work as the attacker likely wanted. We have reports of it triggering Valgrind errors, and outright crashing in some configurations. If the attacker cannot even spare the time and resources to make it fully reliable on one platform, porting it to other platforms is just asking for unwanted attention.
Back to top
View user's profile Send private message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3348
Location: Rasi, Finland

PostPosted: Fri Apr 05, 2024 1:30 pm    Post subject: Reply with quote

salam wrote:
Is it a common thing to mix source code and some binary mess? For me, this looks to be the biggest problem of the whole chain. The project cannot be considered open source any more. If foreign binary blob is allowed to be included in the compilation process, we can expect many more black boxes in linux hosts, doing random things.
To my knowledge, the binary data was there to be used as a testing material for compressing and decompressing data.

Somebody correct me if I'm wrong.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
colo-des
Tux's lil' helper
Tux's lil' helper


Joined: 20 May 2011
Posts: 97

PostPosted: Fri Apr 05, 2024 1:56 pm    Post subject: Reply with quote

Zucca wrote:

What opened my mind was the fact of creating my blfs using the crux packet manager "pkgutils + prt-utils"
Crux is very simple with its ports "Pkgfile", one see the options of ./configure or meson_options.txt and can define to depend or not.
With the passage of time and the updates of the "own" ports you will realize so that these dependencies
are and to what extent they are necessary, when comparing them with the USEs, one realizes several
important things, many ./configure options have no USEs, or are enabled/disabled by ebuild creator, removing flexibility and possibilities to users.
For the sake of ease, we go to a dictatorial regimen, what options are set/unset and what options have an USE.
At the end of the day this forces you to have a huge local tree full of modified ebuilds.
One begins to see that libraries have functions that can be used "badly", but everything cannot be eliminated...
I opt for extreme minimalism, the less...the better.
Meanwhile I continue to automate my ports for blfs, a not too distant day will come in which gentoo is just a memory in my life.

Regards.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21653

PostPosted: Fri Apr 05, 2024 2:02 pm    Post subject: Reply with quote

Yes, that was its stated purpose, and for the non-malicious binary parts, was its true purpose. However, I recall reading a comment elsewhere criticizing the practice of committing binary data like this that has no reproduction process. That person advocated instead that there should exist a documented and reproducible path to regenerate the test data, and arrive at the same blobs that were actually committed. As an example of what that commenter could have meant, if the test is that the decompressor correctly rejects a corrupted file, then the reproduction path might be:
  • Use seq 1 10000 to generate a file for compression.
  • Compress it.
  • Use dd bs=1 seek=N ...other-params to corrupt specific bytes in the compressed file from the previous step.
  • Run the decompressor on the resulting damaged file and verify it fails as intended.
While the person committing these blobs may have followed those steps, the steps were not documented and committed, so no one could rerun them to verify that the blob really was just some corruption of an innocuous file. Once there was a pattern of committing blobs that downstream could not reproduce at will, the attacker could readily commit a non-reproducible malicious blob and expect that no one would notice it.
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2575
Location: Here and Away Again

PostPosted: Sat Apr 06, 2024 1:23 am    Post subject: Reply with quote

For the architecture, indeed, why make things more complicated if you don't need more than a specific one?

All in all, at least at this point, it seems they were not after "infecting the whole world", but something more specific... but I doubt we'll ever know for sure.

Regarding test data, Lasse Collin (Larhzu at #tukaani / Libera) mentioned he created the test files of old mostly using a hexeditor (as mentioned in 'tests/files/README' [1]), but the newer ones were created using a generator(?) by Jia.

I also think it seems normal to have them, but the method they were being created there is not the way to do it (and Lasse was not happy about it either I believe).

1. https://git.tukaani.org/?p=xz.git;a=blob;f=tests/files/README;h=e987a5193194907a75fe38bb36106dfd3e0f115f
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
salam
Apprentice
Apprentice


Joined: 29 Sep 2005
Posts: 226

PostPosted: Sat Apr 06, 2024 9:52 am    Post subject: Reply with quote

I got the point with the legit test data, what worries me is the ability to include literally anything in the final binary. If the build chain is not locked down to the sources directory, that should contain only readable files, the backdoor could be able to pull that additional data from anywhere(would the same work if the binary was summoned by a tool like wget?). Something like in a container environment, mount only the directories required for building, disable network access, build the package and merge it if the task was a success.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21653

PostPosted: Sat Apr 06, 2024 3:14 pm    Post subject: Reply with quote

Locking down the build environment is a good idea, but that is usually left to the distributor, rather than built into every package individually. This is safer, in part because if it was part of the package, a malicious developer could "accidentally" disable the lockdown, as Jia Tan apparently did with the test for Landlock support. When the distributor is responsible for the lockdown, we can assume that the malicious developer must actively work to escape it, rather than committing a deniable bug that prevents the lockdown from enforcing its rules.

To your point about wget, Portage's network-sandbox FEATURES entry would block that avenue. I don't know if the targeted distributions implement an equivalent sandbox.
Back to top
View user's profile Send private message
jpsollie
Apprentice
Apprentice


Joined: 17 Aug 2013
Posts: 291

PostPosted: Sun Apr 07, 2024 4:35 am    Post subject: Reply with quote

Hello everyone,

My laptop/desktop/NAS have been using the xz-utils 5.6.0 package right from the start.
As such, they were potentially affected.
When this CVE was published, I immediately downgraded XZ, but:

1. Does it affect openssh binaries compiled with XZ as well? or only at runtime?
the idea is: let's assume I compiled a gentoo package for amd devices when xz 5.6.0 was being used,
will installing that precompiled binary automatically install the backdoor as well?

2. How can I activate the kill switch?
I read here: https://piaille.fr/@zeno/112185928685603910 that the following kill switch exists:

Code:

yolAbejyiejuvnup=Evjtgvsh5okmkAvj


would putting this in /etc/profile be enough to make sure the kill switch is always trigged (and thus the malicious code never executed)?
_________________
The power of Gentoo optimization (not overclocked): [img]https://www.passmark.com/baselines/V10/images/503714802842.png[/img]
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1554

PostPosted: Sun Apr 07, 2024 10:54 am    Post subject: Reply with quote

When I read nonsense like this:

Quote:
Quote:


Quote:
Can anyone explain the seemingly suspicious string at L115?


There's a theory this was one way to disable the malware (source).
@stan423321 yeah it's a kill switch


I'm sometimes tempted to answer questions like that:

jpsollie wrote:

would putting this in /etc/profile be enough to make sure the kill switch is always trigged (and thus the malicious code never executed)?


with "no, you need to burn your laptop.".

You can sleep safely (an expression in my native language), nothing happened to your laptop...

Unless it was exposed to the internet and the attacker knew there was an active backdoor on it and they purposefully accessed it and did some damage. But then no killswitch can undo that damage or whatever they did.

But still, you can sleep safely because the backdoor was designed to build in conditions different than what Gentoo provides by default.

p.s. what you've linked looks like a pointless amateur job.

Best Regards,
Georgi
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3, 4, 5  Next
Page 3 of 5

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum