| View previous topic :: View next topic |
| Author |
Message |
jmglov
Retired Dev
Joined: 03 Aug 2002
Posts: 23
Location: Yokohama, Japan
|
 Posted: Fri Aug 23, 2002 12:30 am Post subject: Trojans, buffer overflows, and elitism, oh my! |
 |
|
You guys realise that OpenSSH was trojaned about two weeks ago, right? OpenBSD.org got cracked,[1] and however cracked them injected some trojaned software into the distribution servers. So, if weird stuff is happening with your sshd, do not walk, run to verify that your ebuild did not include a trojaned tarball!
This is a few weeks *after* a buffer overflow popped up in OpenSSH which was the first remote exploit in five years or so for OpenBSD.
But how the OpenBSD team *handled* the vulnerability was simply criminal^W amazing! They (read: Theo) released an terse advisory[2] on Bugtraq saying simply that there was a bug in all versions of the OpenSSH sshd prior to 3.3. Theo's suggestion:
| Theo DeRaadt wrote: | However, everyone should update to OpenSSH 3.3 immediately, and enable priv seperation in their ssh daemons, by setting this in your /etc/ssh/sshd_config file:
UsePrivilegeSeparation yes
Depending on what your system is, privsep may break some ssh functionality. However, with privsep turned on, you are immune from at least one remote hole. Understand? |
Yes, Theo, we understand that you think we are all complete morons and that you are trying to pull a fast one on us. But what is the fast one? Again, I quote:
| Theo DeRaadt wrote: | | 3.3 does not contain a fix for this upcoming bug. |
So why should we upgrade to 3.3 again? He leaves us in suspense by apparently changing the subject completely:
| Theo DeRaadt wrote: | | If priv seperation does not work on your operating system, you need to work with your vendor so that we get patches to make it work on your system. Our developers are swamped enough without trying to support the myriad of PAM and other issues which exist in various systems. You must call on your vendors to help us. |
Yes! A call to arms! We must call on our vendors! We can save the Internet! But what is "priv separation", and why do we need it?
| Theo DeRaadt wrote: | | Basically, OpenSSH sshd(8) is something like 27000 lines of code. A lot of that runs as root. But when UsePrivilegeSeparation is enabled, the daemon splits into two parts. A part containing about 2500 lines of code remains as root, and the rest of the code is shoved into a chroot-jail without any privs. This makes the daemon less vulnerable to attack. |
OK, sounds like a great idea. So what is the caveat?
| Theo DeRaadt wrote: | | We've been trying to warn vendors about 3.3 and the need for privsep, but they really have not heeded our call for assistance. They have basically ignored us. Some, like Alan Cox, even went further stating that privsep was not being worked on because "Nobody provided any info which proves the problem, and many people dont trust you theo" and suggested I "might be feeding everyone a trojan" (I think I'll publish that letter -- it is just so funny). |
Oh, apparently it is a political issue? Damn that Alan Cox! What does he know? I wish that Theo *had* published Alan Cox's email, that would have been some excellent reading. But I guess he realised that the joke would have been on him in the end!
| Theo DeRaadt wrote: | | So, if vendors would JUMP and get it working better, and send us patches IMMEDIATELY, we can perhaps make a 3.3.1p release on Friday which supports these systems better. So send patches by Thursday night please. Then on Tuesday or Wednesday the complete bug report with patches (and exploits soon after I am sure) will hit BUGTRAQ. |
Oh, I see. So we are pushing out a version of OpenSSH that does not work on a lot of systems? And we are giving the vendors until Thursday to fix things before we drop the details onto Bugtraq? Hmm, that sounds a bit like ISS X-Force's vulnerability policy...[11] [12] [13] [14] [15]
| Theo DeRaadt wrote: | | Let me repeat: even if the bug exists in a privsep'd sshd, it is not exploitable. Clearly we cannot yet publish what the bug is, or provide anyone with the real patch, but we can try to get maximum deployement of privsep, and therefore make it hurt less when the problem is published. |
Wait, you are repeating? But you have not said that until now... But, OK, so we are pimping out privsep, and it sounds like A Good Thing(tm). But wait, you said:
| Theo DeRaadt wrote: | | If priv seperation does not work on your operating system, you need to work with your vendor so that we get patches to make it work on your system. Our developers are swamped enough without trying to support the myriad of PAM and other issues which exist in various systems. |
Well, call me crazy, but why do we want code that is so untested and new that it does not even run on a lot of systems on our production servers? Our security-critical servers? Why?
| Theo DeRaadt wrote: | So please push your vendor to get us maximally working privsep patches as soon as possible!
We've given most vendors since Friday last week until Thursday to get privsep working well for you so that when the announcement comes out next week their customers are immunized. That is nearly a full week
(but they have already wasted a weekend and a Monday). Really I think this is the best we can hope to do (this thing will eventually leak,
at which point the details will be published). |
Ah, so now it becomes clear. Theo wants *our* help to bully the vendors into foisting this untested, unworking code on us.
And *this* takes the cake:
| Theo DeRaadt wrote: | | Customers can judge their vendors by how they respond to this issue. |
No, Theo, we can judge the OpenSSH team by their willingness to allow *you* to shovel FUD into the mouths of the unwashed masses to get some of your code born before it is fully grown. WTF, I ask?
And it turns out that ISS X-Force was the team that discovered the vulnerability![3] Interestingly enough, in their advisory, they provide some details:
| ISS X-Force wrote: | | A vulnerability exists within the "challenge-response" authentication mechanism in the OpenSSH daemon (sshd). |
OK, so the vuln is known. Why could the OpenSSH team patch it, then? Is there no work-around? Well, according to ISS X-Force,[3] there *is*!
| ISS X-Force wrote: | ISS X-Force recommends that system administrators disable unused OpenSSH authentication mechanisms. Administrators can remove this vulnerability by disabling the Challenge-Response authentication parameter within the OpenSSH daemon configuration file. This filename and path is typically: /etc/ssh/sshd_config. To disable this parameter, locate the
corresponding line and change it to the line below:
ChallengeResponseAuthentication no
The "sshd" process must be restarted for this change to take effect. This workaround will permanently remove the vulnerability. |
So, why did Theo not tell us this? Because that would not accomplish his agenda of pushing the privsep code out! It would have protected a lot of systems, however. Criminal.
The official OpenSSH advisory on the vuln hit later that day.[4] In this, the OpenSSH team suggests the same fix as X-Force. But how many sysadmins fell for the FUD of a few days prior blindly upgraded to 3.3? I almost did. Had Theo not taken a poke at Alan Cox in his announcement, I would have thought nothing of it and upgraded, with a sigh of relief at disaster averted.
Interestingly enough, the OpenSSH advisory underwent *three more* revisions! [5] [8] (Sorry about the missing third revision, it was not posted to Bugtraq, and is not on OpenSSH's web page.) In the second[5], they tell us that PAMAuthenticationViaKbdInt is also vulnerable to the buffer overflow. But the bit on privsep is right below it:
| OpenSSH Team wrote: | Short-Term Solution:
Disable ChallengeResponseAuthentication in sshd_config.
and
Disable PAMAuthenticationViaKbdInt in sshd_config.
Alternatively you can prevent privilege escalation
if you enable UsePrivilegeSeparation in sshd_config. |
Amazing. And in the fourth revision of the *same* advisory[8], we see more of the same, plus some major backpedaling (see 6. Release Process).
At least not all vendors were bullied, though many were. Sun's Darren Moffat shows the correct reaction,[7] from a *real* Unix vendor:
| Darren Moffat wrote: | | The patch that Sun produces to fix this issue will not contain the new OpenSSH Privsep support as it is not yet stable enough on Solaris due to interactions with PAM and BSM auditing, this may appear in a future release - Sun is working with the OpenSSH devlopers on the PAM problems and once a working OpenSSH with PAM and BSM is available we will re-evaluate our position on Privsep. |
I have also provided some more reading for the curious:
- Some excellent forensic work by Rapid7[6]
- GOBBLES takes a poke at Theo and releases a 'sploit for OpenBSD[9]
- Global InterSec provides an excellent summation of the whole sordid affair and some POC code for Linux[10]
Wow, what a rant. But I *do* feel better now! ;)
--Josh "Security through making fun of Theo *is* a valid method" Glover
[1] OpenSSH Security Advisory (adv.trojan)
http://online.securityfocus.com/archive/1/285554
[2] Upcoming OpenSSH vulnerability
http://online.securityfocus.com/archive/1/278755
[3] ISS Advisory: OpenSSH Remote Challenge Vulnerability
http://online.securityfocus.com/archive/1/278818
[4] OpenSSH Security Advisory
http://online.securityfocus.com/archive/1/279117
[5] Revised OpenSSH Security Advisory (2nd)
http://online.securityfocus.com/archive/1/279134
[6] Rapid7 - How to reproduce OpenSSH Overflow
http://online.securityfocus.com/archive/1/279430
[7] Sun statement on the OpenSSH Remote Challenge Vulnerability
http://online.securityfocus.com/archive/1/279680
[8] Revised OpenSSH Security Advisory (4th)
http://online.securityfocus.com/archive/1/280070
[9] Proof of Concept Code for OpenSSH
http://online.securityfocus.com/archive/1/280029
[10] Global InterSec 2002062801
http://online.securityfocus.com/archive/1/280511
[11] ISS X-Force: Apache HTTP Server Exploit in Circulation
http://bvlive01.iss.net/issEn/delivery/xforce/alertdetail.jsp?oid=20524
[12] Gobbles's apache-nosejob.c
http://online.securityfocus.com/archive/1/278233
[13] ISS Advisory clarification
http://online.securityfocus.com/archive/1/278189
[14] (Michael Stone) Re: ISS Advisory clarification
http://online.securityfocus.com/archive/1/278208
[15] (security curmudgeon) Re: ISS Advisory clarification
http://online.securityfocus.com/archive/1/278210
Last edited by jmglov on Mon Aug 26, 2002 9:01 pm; edited 1 time in total |
|
| Back to top |
|
 |
rac
Bodhisattva
Joined: 30 May 2002
Posts: 6553
Location: Japanifornia
|
| Posted: Mon Aug 26, 2002 9:13 pm Post subject: |
|
|
This post originally appeared as part of this thread, and has been split into its own thread here after discussion with the poster.
_________________
For every higher wall, there is a taller ladder |
|
| Back to top |
|
|
EPrime
Tux's lil' helper
Joined: 10 Aug 2002
Posts: 80
Location: Denmark
|
| Posted: Tue Aug 27, 2002 12:06 am Post subject: |
|
|
| Quote: | The question is rather, "why should we be running largely untested, unaudited code on mission-critical and/or security-sensitive servers?"
My answer is, "we should not!" |
All software, even when thoroughly tested, has bugs. Pushing a major security improvement makes sense considering the product.
Even if his not mentioning the cram-fix was a deliberate act, they may have realised that it existed only after the initial publishing of the privsep-workaround was made. They may have had to move quickly to announce a workaround, and just careless in not probing for alternatives. Maybe naive of me, but at least not painfully so 
| Quote: | | The code is so new that the OpenSSH team has not been able to port it to many different platforms. |
I installed it from source on a Mandrake box, well before a package was released, with no problems. I choose sometimes to place greater trust in new software with a major design flaw fixed. After all, it's fairly easy to verify that it works and production environments should be using process monitors anyway.
| Quote: | | I happen to think that Sun is right in the case. So why did Red Hat and Mandrake quickly release packages and Sun not? Because, in the end, Sun *has* to be accountable to its customers. |
Perhaps Sun found the code not to be working, whereas rh/mdk had no problems and thus were able to use the suggested fix? Sun also enjoys living in a world of more money - suggesting to their customers that they purchase a commercial SSH server would not make them run screaming away to another vendor.
| Quote: | | Red Hat has shown in the past that it is willing to push out beta-level code in its actual releases (and I refer to the gcc-2.96 fiasco[1]). |
Yet they remain a popular choice of distribution for businesses. Only goes to show that technicians still don't run the world 
| Quote: | | The reason that I posted my largely political attack on how the OpenSSH team handled the whole PrivSep issue is that I felt they used a technical vulnerability to slingshot a political issue past otherwise reluctant vendors. |
I'd prefer to call his agenda technical rather than political, since the latter implies some hidden, ulterior motive. Assuming you are correct, I don't think it's much of a crime. As you say, the vendors were reluctant and in this world you need to get peoples attention before they actually listen. I can see how that can be very frustrating when you're spending a lot of time on a free, useful product with a major improvement that people ignored because no bug had been found in it yet.
Anyway, I have most certainly not been following matters as closely as you have and may place too much faith in people I do not know. It is nice to keep an open mind, though  |
|
| Back to top |
|
|
jmglov
Retired Dev
Joined: 03 Aug 2002
Posts: 23
Location: Yokohama, Japan
|
| Posted: Tue Aug 27, 2002 1:47 pm Post subject: |
|
|
| EPrime wrote: | | All software, even when thoroughly tested, has bugs. Pushing a major security improvement makes sense considering the product. |
I do not disagree with your statement, but I, unlike Machiavelli, do *not* believe that the end justifies the means.
I disagree with the OpenSSH team on both technical *and* political grounds. Technical, because they pushed out PrivSep on the world before PrivSep was ready. Political, because they took advantage of a vulnerability in their code to try and pull the wool over everyone's eyes with their incomplete and misleading advisories.
The bottom line is, when the OpenSSH team releases official advisories, people should be able to trust that they are really trying to give us the information that we need to make a fix. I for one do *not* trust the OpenSSH team anymore.
| EPrime wrote: | | Even if his not mentioning the cram-fix was a deliberate act, they may have realised that it existed only after the initial publishing of the privsep-workaround was made. They may have had to move quickly to announce a workaround, and just careless in not probing for alternatives. Maybe naive of me, but at least not painfully so :wink: |
Nope, if you read the fourth revision of the advisory,[1] the OpenSSH team tries to justify the fact that they released the upcoming vulnerability notification saying "upgrade and use PrivSep" when they *knew* of the exact solution.
| EPrime wrote: | | I installed it from source on a Mandrake box, well before a package was released, with no problems. I choose sometimes to place greater trust in new software with a major design flaw fixed. After all, it's fairly easy to verify that it works and production environments should be using process monitors anyway. |
I disagree that it is easy to verify that it works. Without reading through the code and running stress and penetration tests, it ain't easy.
| EPrime wrote: | | Quote: | | I happen to think that Sun is right in the case. So why did Red Hat and Mandrake quickly release packages and Sun not? Because, in the end, Sun *has* to be accountable to its customers. |
Perhaps Sun found the code not to be working, whereas rh/mdk had no problems and thus were able to use the suggested fix? Sun also enjoys living in a world of more money - suggesting to their customers that they purchase a commercial SSH server would not make them run screaming away to another vendor. |
True, Sun's customers tradionally have deeper pockets. But how did Sun get so many large corporate customers? By taking care of them.
| EPrime wrote: | | Quote: | | Red Hat has shown in the past that it is willing to push out beta-level code in its actual releases (and I refer to the gcc-2.96 fiasco[1]). |
Yet they remain a popular choice of distribution for businesses. Only goes to show that technicians still don't run the world :lol: |
Don't get me wrong--I think Red Hat has done more for Linux and Open Source / Free Software in general than almost any other company. I have and will support them financially because of what they are doing. But I no longer feel that Red Hat is what I want to run on *my* servers and personal workstations. My users run Red Hat, largely because of all the tools that exist to make administration of them easy for me to automate.
But I do not trust new releases of Red Hat without reading the changelog carefully anymore. Which is undeniably A Good Thing(tm), so I guess I should thank them for including a compiler that, while not broken itself, broke a lot of software in a "stable" release of their distribution. ;)
| EPrime wrote: | | I'd prefer to call his agenda technical rather than political, since the latter implies some hidden, ulterior motive. |
I think that you are correct in saying that the latter implies some hidden, ulterior motive. I believe that Theo *has* such a motive, and that is to stroke his famously oversized ego. He wants to be universally recognised as the best. That alone is not a bad thing, but he seems to want to prove it constantly, often at the cost of quality in his software.
| EPrime wrote: | | Assuming you are correct, I don't think it's much of a crime. As you say, the vendors were reluctant and in this world you need to get peoples attention before they actually listen. I can see how that can be very frustrating when you're spending a lot of time on a free, useful product with a major improvement that people ignored because no bug had been found in it yet. |
I maintain that spreading FUD is no way to bring the reluctant vendors in line. Ever.
--Josh
[1] Revised OpenSSH Security Advisory (4th)
http://online.securityfocus.com/archive/1/280070
_________________
Josh Glover <jmglov@gentoo.org>
Gentoo Developer (http://dev.gentoo.org/~jmglov/) |
|
| Back to top |
|
|
EPrime
Tux's lil' helper
Joined: 10 Aug 2002
Posts: 80
Location: Denmark
|
| Posted: Thu Aug 29, 2002 8:09 pm Post subject: |
|
|
| I can see that you're a reasonable person with good arguments, and am inclined to agree with most you say. In any case, I don't have any additional aguments to offer and so it's best for me to withdraw at this point. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
