Gentoo 1.4 Webserver got rooted tonight.

Message

Vancouverite · Post by **Vancouverite** » Fri May 23, 2003 10:42 am

Judging by the damage done I think it's fair to say this wasn't a very sophisticated hacker. Probably a scripted exploit or a bad cgi script allowing arbitrary command execution and they just ran an xterm pointed at their IP. Without knowing the configuration of everything the how is anyones guess. Better luck next time.

[edit] You might want to take a look at User-mode Linux
[another edit] After you have re-installed everything run a good nessus scan and deal with any issues it reports.

xedx · Post by **xedx** » Fri May 23, 2003 10:46 am

best bet with be a local user compromise or a weak password

paul138 · Post by **paul138** » Fri May 23, 2003 1:38 pm

I think though that part of the problem lies in how the system is customized eg. USE flags.

The USE flags play alot with how things are compiled and configured. Say you had ldap, mysql and pam in your USE; your Pure-FTP daemon would be built with extra functionality. When I go to the Pure FTP website and look for this info, the first thing I see in the docs is:

Code: Select all

If you never heard about LDAP before, *DON'T* enable LDAP support in
Pure-FTPd. LDAP is useless if you don't have to manage many shared accounts.

It says the same thing as well for MySQL support. OK, so they dont say anything about it being insecure, but how much has this functionality been tested as opposed to the daemon as a whole which has probobly been tested much more.

Now you have this added functionality which has not been thuroughly tested in some packages and it's live to the Internet.

Of course, I'm not saying the Pure FTP guys dont test their code, nor am I picking on the program. It's only an example (NO FLAMES). I'm also not picking on portage. What it all boils down to is being careful with your flags.

Maybe it's possilbe to run a Gentoo system like your average run-of-the-mill Linux distro where you download packages and install them (eg. RPM)? Certainly it is (in theory), you keep a local build of your remote system(s) and build the packages (static if needed) and install them with the -k option. While I have not tested this 100% I am willing to try it out and report my findings.

There are currently 20 Gentoo systems (servers) in this office and 4 in production (last I counted). So I have plenty to play with.

Maybe a little more involvment in the Gentoo-Hardened project would be an asset (I dont see a lot of activity).

In the meantime, be carefull with those USE flags, they may be adding very experimental code to production systems!

-P (A Gentoo BOFH)

Dalrain · Post by **Dalrain** » Fri May 23, 2003 4:34 pm

Hmm....so I was thinking on this, and I was wondering what kernel version you were running? emerge -u world will download and unpack the source, but of course not do any special installation in that regard. If you were one of the people using gentoo-sources that couldn't upgrade due to the IDE code (and still cannot, such as myself...) then perhaps you still had the local exploit open from a bit back?

Also, I'm not sure just how important this box is, but you could always use a machine with networked logging, or go with the old trick that kills trees....set up an old, old, printer and have it print logs. If it's just a little user box, then perhaps that's a little extreme...but hey, it's a little hard to delete paper unless they're there.

I also agree on the above comment about USE flags. grsecurity is a savior IMHO as well. ACLs = teh win

paul138 · Post by **paul138** » Fri May 23, 2003 4:49 pm

Dalrain wrote:Hmm....so I was thinking on this, and I was wondering what kernel version you were running?

The exploit was local (you needed to log in first) so, maybe they cracked into a user account and went from there. Another reason to take gcc off of the machine.

Dalrain wrote:Also, I'm not sure just how important this box is, but you could always use a machine with networked logging, or go with the old trick

You could also use networked logging ( a little tricky, but easy once learned) with syslog-ng. You could also impliment snort on another machine with a one-way sniffing cable (do a Google).

gigel · Post by **gigel** » Fri May 23, 2003 5:22 pm

i do not think gcc is a issue here...
when someone managed to break into your server,it'just not imprtant if u have or not gcc installed..tha basic ideea is that the cracker got acces to your box,and therefor it is compromised..i think it is senseless trying to encrease secutiry from this point(once the cracker is loged into the box)...
so i prefer to let gcc on my box instead of heaving outdated software...

the ideea is to prevent the remote attackers!!
local users can be easily managed
just chmod 700 /bin /sbin /usr/bin /usr/sbin u.s.w...

main problem in security is remote attacks,not the local ones...

another security increasing option is to modify the source code of the servers...what do i mean by that...well
lets take apache for instance..supose we have installed apache 1.3.27..but we want the cracker to think we're using apache 1.3.13 or even IIS ...if we manage to do that we put the cracker on a bad path..looking for exploits for another version...this may sound stupid but it's not...imho

i bet this is a verry well used tehnique...just take a look at netcraft and make some searches...who the heck is using apache 1.3.1x nowdays??

//edit:if u look at the screenshot in my signature u get what i mean..that was so simple to do..now i'm on researching how to modify apache

paul138 · Post by **paul138** » Fri May 23, 2003 8:57 pm

mortix wrote:i do not think gcc is a issue here...
when someone managed to break into your server,it'just not imprtant if u have or not gcc installed..tha basic ideea is that the cracker got acces to your box,and therefor it is compromised..i think it is senseless trying to encrease secutiry from this point(once the cracker is loged into the box)...
so i prefer to let gcc on my box instead of heaving outdated software...

So, in short, you would not mind that they could emerge packages, build kernel modules or simply eat up 100% of your processor?

gigel · Post by **gigel** » Fri May 23, 2003 9:25 pm

paul138 wrote:
mortix wrote:i do not think gcc is a issue here...
when someone managed to break into your server,it'just not imprtant if u have or not gcc installed..tha basic ideea is that the cracker got acces to your box,and therefor it is compromised..i think it is senseless trying to encrease secutiry from this point(once the cracker is loged into the box)...
so i prefer to let gcc on my box instead of heaving outdated software...
So, in short, you would not mind that they could emerge packages, build kernel modules or simply eat up 100% of your processor?

yes,you're right

if they could manage to get to this level(just login into my box)
it means that my effort of securing my server was useless
once you're cracked(of course u should realize it) there is no alternative but to reinstall .........
if the crackers manage to break into the system they could use corrupt binaries instead of compiling them...
they could download an exploited version of (lets say) ls ...so every time you (as root) type ls u open a random port ..guess you'll get the picture...

the main ideea is:
if the cracker managed to login to your box(_this_ is the hard part)with or without gcc you are kaputt...
as a sysadmin you must prevent this from happening!!
and this has nothing to do with gcc.....

regards!

Vancouverite · Post by **Vancouverite** » Fri May 23, 2003 10:34 pm

mortix wrote:the main ideea is:
if the cracker managed to login to your box(_this_ is the hard part)with or without gcc you are kaputt...
as a sysadmin you must prevent this from happening!!
and this has nothing to do with gcc.....

Precisely. If someone gets a shell you're shit out of luck. Most hacks are due to poorly configured servers, bad CGI scripts...etc. Remember the apache.org defacement when it was rooted (by grey hats as a proof of concept) because of bad configuration settings and nothing else. Security by obscurity is a joke IMO and a false confidence booster for the lazy. Real security is about doing a lot of little things right, monitoring logs and keeping current with patches.

christsong84 · Post by **christsong84** » Sat May 24, 2003 12:55 am

Vancouverite wrote:
mortix wrote:the main ideea is:
if the cracker managed to login to your box(_this_ is the hard part)with or without gcc you are kaputt...
as a sysadmin you must prevent this from happening!!
and this has nothing to do with gcc.....
Precisely. If someone gets a shell you're shit out of luck. Most hacks are due to poorly configured servers, bad CGI scripts...etc. Remember the apache.org defacement when it was rooted (by grey hats as a proof of concept) because of bad configuration settings and nothing else. Security by obscurity is a joke IMO and a false confidence booster for the lazy. Real security is about doing a lot of little things right, monitoring logs and keeping current with patches.

don't forget backups. (of everything...data AND logs)

ebrostig · Post by **ebrostig** » Sat May 24, 2003 3:09 am

I'm sorry to hear that you were rooted.

I would recommend using Tripwire or similar system to monitor the system.

Also, remember the following:

- Any system facing the Internet has to be considered compromised.

Based on this, store data that the users can access through encrypted connections only from the outside (where the webserver is located) to your database (on the inside). Also make use of DMZ's and 2 NIC's. Don't run any service or open any port against the Net except for the prots that are neeed (80 on a webserver, mayeb 443 for SSL). Don't accept normal login on any NIC that faces the Net.

And as other people have mentioned, scrap the disks and reformat everything, re-install from trusted sources. Your box is compromised and has to be considered until recreated.

Good Luck!

Erik

chmod · Post by **chmod** » Sat May 24, 2003 9:22 am

Below is the entire .bash_history from the box's root account. I was able to recover this file using debugfs on my / parition. Domain names were changed to protect my box.

Code: Select all

w
id
locate httpd.conf
pwd
find / -name httpd.conf
cat /usr/local/apache/conf/httpd.conf | grep ServerName
find / -name httpd.conf
cat /usr/local/apache/conf/httpd.conf | grep ServerName
ps ax
kill -9 17371
kill -9 17208
kill -9 cd /etc
cd /etc
w
ls
cat hosts
cd /web
ls
cd domain.com
ls
cat index.htm
mv index.html index.bak.html
mv index.htm index.bak.htm
echo Perfect.BR > index.htm
ls
cat index.
cat index.htm
cd html/
ls
cd ..
cd ..
ls
w
cd domain2.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd ..
cd domain3.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd ..
cd domain4.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd ..
ls
cd domain5.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd ..
cd domain6.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd html/
ls
cd /web
ls
cd domain7.com
ls
mv index.php index.bak.php
echo Perfect.BR > index.php
cd /root
rm .bash_history
ls -al
ls -la
rm /var/log
rm -rf /var/log
exit

Even though this evidence leads me to believe it was nothing more than a kiddie defacing some sites, I will still rebuild the box just to be sure. This time following the steps of the gentoo security doc. I have recently installed snort on the other boxes on the network, and have switched all my SSHD servers to use RSA keys instead of passwords, and only use version 2 of ssh. Thanks for the tips and links everyone. If I can post more stuff i will.

beowulf · Post by **beowulf** » Sat May 24, 2003 10:12 am

The kid took the time to back up the files...

Did a search on Google...
http://www.google.ca/search?q=perfect.b ... arch&meta=

It appears that it was done by a group of kids that deface web sites regularly. Their irc channel is #perfect_br but i didn't notice a server listed. The email is perfectbr@mail.com, which could be a fake, but appears to be the groups signature on all sites.

Apparently the group is from Brazil, or Brazillian in nationality. Based on one of his defacements in which he writes "Perfect.BR again - No war for oil!! brazil rlz!! - perfectbr@mail.com"

Also appears that they post their members nickname on a few pages... RE: http://safemode.org/mirror/2002/01/03/w ... bjects.ch/

Another note, it does not appear to be a gentoo specific crack. Since the user tried to find httpd.conf, but gentoo has been using apache.conf since 1.1 ( or earlier i believe). So for that reason, i believe it was a standard attack... not targetted specifically at gentoo.

In any case, sorry to hear about the hack... hope you can get everything back up quickly...

Slynix · Post by **Slynix** » Sat May 24, 2003 11:57 am

http://www.dominasecurity.com/hackerz/perfectbr.htm

puddpunk · Post by **puddpunk** » Sat May 24, 2003 8:16 pm

Basically, when your running a server, the best thing to do is find versions that you need and stick to them. If you need the extra functionality that an updated version gives, or there is a security patch, then by all means, upgrade, but blindly upgrading things, especially on a closed environment like a server could spell trouble.

xedx · Post by **xedx** » Mon May 26, 2003 3:33 am

Have any idea which kind of exploit they used to root the box?

gilesc · Post by **gilesc** » Sat May 31, 2003 10:48 am

emerge -u world every few days will not update your system.

This however, will:

Code: Select all

#!/bin/bash
emerge sync
emerge -u world

Are you sure you were updating your system?

Do ensure you do a full re-build, the attacker will have installed a backdoor onto your system, and although you killed the PID any command such as 'ls' could be trojaned to restart it.

paul138 · Post by **paul138** » Mon Jun 02, 2003 11:54 am

Slynix wrote:http://www.dominasecurity.com/hackerz/perfectbr.htm

Those d00ds are lam3. lol, looks like beginners.

PS: chmod Good work recovering the bash history. Too bad logs under /var could not be recovered. It's funny (sorry) to see that he kept typing w to see who was logged on.

uzik · Post by **uzik** » Mon Jun 02, 2003 4:23 pm

chmod wrote:I am running a Gentoo 1.4 box on a dual Xeon 2.4Ghz Intel board. I installed 1.4RC2 on it and have kept it updated by emerging "-u world" every few days.

Don't do that! It puts a lot of needless load on the servers.
Trading new bugs you haven't identified for bugs you already know
seems rather counter productive to me.

chmod wrote: Tonight I went to a site of mine, to find the index page had been moved with a new one in it's place. Also the intruder kindly removed the /var/log directory, and my last year or so of logs.

Where do I start even looking to find out who did this? Is there a likely hole in gentoo because of this? I'm no security buff, but I have kept openssh updated, and have no other users of this machine.

Can someone help me start hunting for what caused this? I'm not as much interested in catching the cracker as I am in preventing this from happening again.

A. Look for time and date stamps that are inappropriate. This
is probably impossible for you since you update everything every
week anyway.

B. Reformat and restore from the last good backup

C. Remove **everything** that isn't absolutely necessary.

D. Review your firewall script.

E. Mail the logs to yourself on another box nightly.
If they don't show up you'll know it fairly quickly.

F. The Gentoo servers are vulnerable to hackers too.
Emerge with care, you don't know what you're downloading.

G. Consider putting vulnerable services (such as dns bind)
on a separate server with write protected media. They can
hack it but if they can't write to the disk it won't matter.

paul138 · Post by **paul138** » Mon Jun 02, 2003 4:29 pm

I'm pretty sure we covered all of this already

idl · Post by **idl** » Mon Jun 02, 2003 5:02 pm

Koon wrote:As a conclusion, we sure hope there is no unknown vuln in the wild in a common package, but we cannot help you without an intimate knowledge of everything that was done and installed on your machine, the physical context around it, etc...

There are actualy quite a lot of them... crackers don't disclose their exploits, many don't even tell their friends.

As for your logs, well thats a nobrainer with any type of illegal activity, cover your tracks.

FTP is a protocol... don't call it vulnerable, that depends on the ftpd. There are plenty of secure deamons out there.

The sad truth is - If a cracker wasn't root on your box then they will get it. Sure you can make it damn hard for them.. but no box is impregnable.

My adivce to you is do your homework, there is a lot you can do to prevent attacks. You may also want to think about setting up a log server.

gigel · Post by **gigel** » Mon Jun 02, 2003 5:20 pm

paul138 wrote:I'm pretty sure we covered all of this already

yep,you're right..
though i must say this also
netstat -ltun is your friend....
also nmap and nessus

chrisis · Post by **chrisis** » Thu Sep 25, 2003 9:40 pm

paul138 wrote: Do not install X windows on a server. This has proved fatal many times (especially where web servers are concerned, it is possible to start an xterm on a remote system with the right peice of buggy code and netcat).

How do you do this? I recently installed a box in what I planned to be a server-config, included USE="-X" in my make.conf, but emerge system still installed X.

What have I done wrong?

If possible, remove gcc alltogether. Keep a mirror of the system at your office and build the updates as packages at the office then scp them (or take them on a CDR) to the remote machine. Use emerge -k [package] to install it, no gcc needed.

Is there a way to do this but still be able to install new packages on the server? For me this is my big dilemma with gentoo. Installing requires a compiler, but a compiler on a server is a security risk! Any suggestions for overcoming this paradox?

devon · Post by **devon** » Thu Sep 25, 2003 11:11 pm

chrisis wrote:
paul138 wrote:Do not install X windows on a server. This has proved fatal many times (especially where web servers are concerned, it is possible to start an xterm on a remote system with the right peice of buggy code and netcat).
How do you do this? I recently installed a box in what I planned to be a server-config, included USE="-X" in my make.conf, but emerge system still installed X.

When installing programs, use the -pv options for pretend/verbose output. Remove USE flags to get rid of X dependecies as needed.

Another option is to make USE="-*" in /etc/make.conf and use the -pv options to add USE flags as needed.

Blahbbs · Post by **Blahbbs** » Wed Oct 01, 2003 9:19 pm

I know this has probably been beaten to death, but while 'emerge -u world' might update your packages, don't you have to restart most services for the changes to take effect?

Say I'm running Apache 1.3.22 (maybe this is a bad example), and I keep running 'emerge -u world' everyday. Now portage says I've got 1.3.28 on my machine. But... isn't the running version of Apache still 1.3.22 until I shut it down and restart it?

Gentoo 1.4 Webserver got rooted tonight.

local

Re: Cracked

Re: Cracked

How

were you really updating?

Re: Gentoo 1.4 Webserver got rooted tonight.