Dissecting rogue processes - Notes

[msalerno]

I have read many posts on these forums where people find unknown/hacked processes wasting valuable resources. I have also seen many replies to these posts on how to get more details about what the process is actually doing. I think it would be beneficial if we were able to create a sticky topic that includes different methods of figuring out what the unknown process is doing. Here is my attempt.

Taking into consideration the following post, I would have recommended the following steps:

1. Capture the output of 'lsof -n -P -p <pid of process> > ~/proflsof.log'
2. Record as much data as possible about the proc, your going to need a few screens
3. strace -p <pid of process>
- probably a good idea to add '| tee ~/procstrace.log' to the strace
- If possible, collect the strace data for a few minutes.

strace -p <pid of process>
- probably a good idea to add '| tee ~/procstrace.log' to the strace

echo CWD `readlink /proc/<pid of process>/cwd` > ~/procinfo.log
stat -L /proc/<pid of process>/exe >> ~/procinfo.log
cat /proc/<pid of process>/cmdline
cat /proc/<pid of process>/environ
cat /proc/<pid of process>/
tar -cvf badproc.tar /proc/<pid of process>/
tar --dereference -cvf ~/badexec.tar `readlink /proc/18729/exe`

4. Kill the process and move the files off of the box.

Files and dirs to evaluate in the badproc.tar:

proc/<pid of process>/cmdline
proc/<pid of process>/environ
proc/<pid of process>/comm
proc/18729/fd/

I know that this is far from complete, but I'm hoping that others will contribute.

[BradN]

If the rogue process isn't smart enough to detect it, run it under strace and you'll have a very good idea what it's doing on your system and where it's connecting out to on the internet.

[msalerno]

I would think it would be best to do that in an isolated environment.

[Hu]

[msalerno]

Agreed, I updated the initial post with your recommendations.

Thanks

[phajdan.jr]

[BradN]

[msalerno]

True, but with the data collected from /proc, you could probably launch the app using the same env and args within an isolated environment. To capture the strace from the time of execution. Of course dependencies would be a pain unless you backed up all of the pids in /proc, even then it wouldn't get everything, but you could get an idea of what it was trying to do.

[Hu]

I do not think that anyone has suggested that rerunning it in an isolated environment is a bad idea. However, there seems to be some confusion and/or disagreement about the merits of attaching strace to the already running exploit in the original environment upon first discovering that the system is compromised. In my opinion, doing so has no downside and some potential upside. There is no downside since the machine is already running hostile code, so there is no way to know in advance that the exploit has not already done all the damage it was meant to do. Attaching strace to it might provoke it to become more destructive, but that would be a weird thing for an attacker to do. There is some upside since you might learn something useful about the process from the strace. You can always SIGKILL it when you are done tracing.

[msalerno]

I completely agree, launching the program with strace and attaching to an existing pd both can provide valuable information. There is no debate there.

[Hu]

For the invocation of lsof, could you edit the initial post to suggest that it be run with -n -P? This will suppress symbolic lookups, which both speeds up its execution and makes the output more deterministic. When reverse resolution is done, connections to remote hosts may be shown with the host name, which can be ambiguous if the name resolves to more than one address. Users can always run the reverse lookups later if they want to see the symbolic names.

[msalerno]

Good point, I updated the post.