hanzo4 process, have I been owned?

[Letharion]

I was gonna restart apache.
Apache goes down as intended, but won't start because it can't claim port 443.

ps fax shows that I have _loads_ of instances of app /tmp/hanzo4 running.
I killall immediately, and see 3 files in /tmp/
hanzo3, hanzo4 and ss. All have a+x rights, and are owned by the only other user that has an account on my server. (He does not have root access, nor is he a wheels member)
He's a friend, so I can't imagine he'd to anything deliberately.

After the kill, apache starts up correctly.

chkrootkit doesn't find anything suspicious.
No gaping security holes that I know of (like old kernels or such)
Running hardend-sources, hardened profile.

What should I do?

[Hu]

The described behavior is either malicious, or indicative of a very naive user. Is your friend experienced with Linux? Were any of the hanzo processes listening on port 443? Low ports are restricted to administrative users, so if it was listening on 443, it had elevated privileges at some point. What types of things should your friend have been doing on the server? Is it possible he installed a vulnerable application, which was then exploited?

It helps that you were running a hardened profile, though it is not a complete guarantee of immunity. You should also mount /tmp as noexec, since it is so popular for exploits to write code there.

[Letharion]

He is definitely not experienced with linux. I think he doesn't even know he can get a shell, much less how to use it for anything. All he does is use winscp to send work to my box that I should take a look at. I never did check if the process were listening to 443, I kinda panicked and killed it. But it certainly seems like that, since apache couldn't claim it.

Damn. I suppose I'd better do a reinstall.

What should really be on /tmp? How large should I make it?

[Anarcho]

At first you should backup the suspicious files in /tmp for a further analyses. Maybe even post it somewhere like rapidshare etc that someone here can take a look.

Your best settings for /tmp depends on your usage. Some programms like to put lots of temp data in there and e.g. PHP stores session files in there. But I think in most cases a 1 - 5 GB partition will be sufficient. If you have lot's of RAM you could even use a tmpfs mount.
_________________
...it's only Rock'n'Roll, but I like it!
HOWTO:WLAN mit OpenVPN absichern | TOOL:useedit - USE-flag editor/changer

[Letharion]

Rapidshare link http://rapidshare.com/files/286405081/hanzo.tbz2.html

Hmm, now I also have a socket, called conftest24184, created by root.
That has got to suck. My first idea was to just shut the server down, but then I thought maybe there some use in leaving it up and running, to see what happens?

[Anarcho]

After investigating arround in the files, it seems that these are parts of scripts kiddie attack files.

1. ss seems to be a port scanner. Most possibly scanning for SSH servers, as

2. hanzo3 and hanzo4 are SSH bruteforce attack programms


Most probably the "andreas" account has been hacked because of a week password. After that the bruteforce scanner installed itself on your system. It seems to be very unlikely that it claimed port 443 as this even couldn't be done while apache was running. I think the port 443 problem was because of a not finished apache process (I've seen this a lot of times before).

First thing todo is change the password of andreas. But what you already should have done before would be to disable shell access if he only uses scp. Install and configure scponly the next time!

The open question is if the attackers got root access. I would say no, but there are doubts. So, best is to reinstall the machine.

The contest socket: Could it be that you are emerging something at the moment?
_________________
...it's only Rock'n'Roll, but I like it!
HOWTO:WLAN mit OpenVPN absichern | TOOL:useedit - USE-flag editor/changer

[Letharion]

[Anarcho]

[Letharion]

Oh, nice. One learns every day
Is there any reasonable way of figuring out how where the attack came from?

[Anarcho]

You can see when the user has logged in by:

[Letharion]

Yep. Unless he recently moved to Ukraine. Which I doubt.
http://whois.domaintools.com/195.69.220.2
Is there a point to send an abuse mail somewhere?

[Anarcho]

[Letharion]

Thanks.

Gonna have to read up on enforcing better password security.

[Anarcho]

And for this kind of things:

1. Use scponly where applicable

2. Disable password auth and only use public/private key based authentication in SSH
_________________
...it's only Rock'n'Roll, but I like it!
HOWTO:WLAN mit OpenVPN absichern | TOOL:useedit - USE-flag editor/changer

[Sadako]

You've said you're using hardened-sources and a hardened profile, perhaps you should look into enabling grsec's RBAC system.
It'll require some work to get up and running as you need, but if you're really security concise it can help lock down your server a lot.

Also, Hu already recommended mounting /tmp noexec, strongly recommended together with /home, if possible, but hardened-sources has a "Trusted Path Execution" feature, which can deny the execution of files in world-writable directories or directories not owned by root.
It's not foolproof, but it's easy to implement, and should be at least as effective as noexec mounts if they aren't an option.
_________________
"You have to invite me in"

[Hu]

Hopeless, thanks for the reminder about TPE. I should have brought it up, but I became focused on features present in vanilla systems.

To extend on my earlier comment about noexec, there are several noX options that can be useful to limit misuse of user-writable areas.

Use nodev to disallow opening any devices in that filesystem. This can be safely set on all real filesystems, assuming you keep /dev in a udev-managed tmpfs. Setting nodev on /dev would be very bad. This protects you from attackers who create block devices to gain direct access to the underlying filesystem. Its value is mitigated by the requirement that an attacker have privilege in order to create the device, in which case he could just attack the real /dev. Despite this, it is free protection, so you may as well enable it where possible.

Use noexec to disallow execution of programs on the mounted volume. Use it for /tmp and /dev/shm. Use it for user home directories if there is no legitimate development activity on the server. Use it for /var if you have moved $PORTAGE_TMPDIR elsewhere, or if you are not emerging anything at the time. Most packages will need /var/tmp to allow exec in order for emerge to work properly. Do not use it for / or /usr.

Use nosuid to disallow use of setuid programs on the mounted volume. This is less strict than noexec, but is more strict than not specifying either. Prefer noexec where you can. If the system has legitimate development activity on it, you may not be able to mount /home as noexec, but you should still be able to use nosuid on /home. Again, do not use it for / or /usr.

Use ro to disallow writes to the filesystem. This may be useful to deter some simple minded attacks, but an adversary with privilege can remount read-write. Use it for /usr and possibly /, if you do not mind /root being read only.

As Hopeless said, look into the RBAC system. The options I describe are easy to activate and are better than doing nothing, but they are generally coarse, and RBAC can give you much more detailed control, if you have the time to get the policy correct.

[Letharion]

Wow, lots of good advice

I appreciate it

Will try to get that re-install under way today (it's a VM, so I need to get down to the hosters location) and then I will look into options for hardening even further.

[sera]