What/Who's screwing with my Apache server?

[WarMachine]

Well, I ran into an odd problem. Here's what began:

I could access the website when connected to the LAN that the server is on. Outside the network, I kept getting the 'connection refused' error. When outside the network, pulling up the page by the server's IP brought it up. Did a quick port scan, 80 was open as usual. DNS resolving correctly. No other trouble SSH'ing into the server. No recent changes to ANY package since the last time the site was working properly.

Rebooted the server, and I was still having the same issue (hell it works for windows, might as well give it a try. At this point I'm pretty stumped)

Ran through some log files and config files to make sure nothing had changed. Nothing odd in /tmp, nothing odd on netstat -a or ps -ax, nothing I didn't remember in root or the (only) user's .bash_history, either.

I did run into this when checking apache's error_log:

[Sat Aug 20 08:37:14 2005] [error] [client 67.8.208.152] File does not exist: /www/favicon.ico
[Sat Aug 20 09:46:40 2005] [error] [client 70.171.47.211] File does not exist: /www/favicon.ico
[Sat Aug 20 11:16:46 2005] [error] [client 67.8.208.152] File does not exist: /www/favicon.ico
[Sat Aug 20 12:50:35 2005] [error] [client 210.243.12.2] script not found or unable to stat: /var/www/localhost/cgi-bin/awstats
[Sat Aug 20 12:50:35 2005] [error] [client 210.243.12.2] script not found or unable to stat: /var/www/localhost/cgi-bin/awstats.pl
[Sat Aug 20 12:50:36 2005] [error] [client 210.243.12.2] File does not exist: /www/cgi
[Sat Aug 20 12:50:36 2005] [error] [client 210.243.12.2] File does not exist: /www/awstats
[Sat Aug 20 12:50:37 2005] [error] [client 210.243.12.2] script not found or unable to stat: /var/www/localhost/cgi-bin/stats
[Sat Aug 20 12:50:38 2005] [error] [client 210.243.12.2] File does not exist: /www/stats
[Sat Aug 20 12:50:38 2005] [error] [client 210.243.12.2] File does not exist: /www/awstats.pl
[Sat Aug 20 12:50:39 2005] [error] [client 210.243.12.2] File does not exist: /www/cgi
[Sat Aug 20 21:48:03 2005] [error] [client 70.116.155.88] request failed: URI too long (longer than 8190)
[Sat Aug 20 22:02:27 2005] [error] [client 70.171.47.211] File does not exist: /www/favicon.ico
[Sun Aug 21 06:07:30 2005] [error] [client 67.8.208.152] File does not exist: /www/favicon.ico

[Tue Aug 23 06:38:16 2005] [error] [client 218.159.143.53] request failed: error reading the headers
[Tue Aug 23 21:16:43 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico
[Tue Aug 23 23:50:03 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico
[Thu Aug 25 07:39:29 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/MSADC
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/c
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/d
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/_vti_bin
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/_mem_bin
[Thu Aug 25 07:39:30 2005] [error] [client 68.37.85.64] File does not exist: /www/msadc
[Thu Aug 25 07:39:31 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:31 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:31 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:31 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 07:39:32 2005] [error] [client 68.37.85.64] File does not exist: /www/scripts
[Thu Aug 25 09:43:15 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:15 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:15 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:16 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:16 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:16 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:17 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:17 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Thu Aug 25 09:43:20 2005] [error] [client 200.168.190.201] script not found or unable to stat: /var/www/localhost/cgi-bin/openwebmail
[Fri Aug 26 05:42:46 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico
[Fri Aug 26 05:53:08 2005] [error] [client 68.220.135.46] File does not exist: /www/favicon.ico
[Fri Aug 26 05:53:13 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico
[Fri Aug 26 05:53:13 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico
[Fri Aug 26 05:57:05 2005] [notice] caught SIGTERM, shutting down
[Fri Aug 26 05:58:54 2005] [notice] Digest: generating secret for digest authentication ...
[Fri Aug 26 05:58:54 2005] [notice] Digest: done
[Fri Aug 26 05:58:57 2005] [notice] Apache/2.0.54 (Gentoo/Linux) mod_ssl/2.0.54 OpenSSL/0.9.7e PHP/4.4.0 configured -- resuming normal operations
[Fri Aug 26 06:03:51 2005] [error] [client 10.0.0.99] File does not exist: /www/favicon.ico

This server was functioning as a private message board running phpBB for a private group of personal friends. Without knowing too much about the exploits for phpBB, I'm pretty sure I can chalk this one up to it (and well, me for running it).

How can I get the webserver securely up and running again?

[jsfan]

I actually don't see the point with your log files. What I can see is that no error was reported after you
had restarted Apache except the missing favicon.ico. But what was written to the access_log? Do the
requested pages appear as served there?

Try ssh'ing to the server in 2 xterms and perform a browser request while

Code: Select all

tail -f /var/log/apache/error_log
tail -f /var/log/apache/access_log

are running. Tell us if they did change at all or if they stay the same when you get the RST packet.

You could also stop apache and try running it in foreground without using the init script. That way
you'll see errors logged to stderr.

[WarMachine]

What I was thinking is that someone is trying to run exploits on the webserver. I noticed the attempts on scripts/ and an attempt on AWstats and webmail utilities I don't run. Is there anything I should be looking for to see if something was exploited?

BTW rebuilding Apache (I was already using the most up to date version in portage) did work for making the webserver useable.

[jsfan]

I'd not say that I know a lot about forensics. You could check what is in the class app-forensics in the portage tree.

However, I'd not worry too much about being hacked. Scan your machine from outside and check if there are any open ports that should be closed.

Of course, there's still a chance that the machine was compromised even if you don't find anything there.
However, I've had problems with apache segfaulting, too, and could only resolve them by doing a re-emerge
with deep dependencies. Although I don't know what exactly the problem was, I'm sure that the machine has
not been hacked.

If you use phpBB you should check the advisories regularly at least. You can use glsa-check for that. For an
admin it's even better to read what is on Bugtraq and Full-Disclosure, however.

[DaveArb]

I don't see what you're seeing, in terms of potential vulnerabilities. I see typical probing from dingalings attached to the Internet. Someone at a school in Taiwan looking for some packages that aren't installed, a Comcast USA user hoping to find a Microsoft web server, a Brazilian user confused about where to find an openwebmail site.

Welcome to the Internet, it's a jungle out there. As js_fan indicated, what your webserver actually responded to successfully is of interest, it refusing to serve to people looking for exploits, is desired behavior.

Dave