apache 2 appears to be dying at random!

[chashab]

I've been running apache-2.0.54-r8 for a few weeks now. twice in the last week i have found it not running.

question 1: if apache dies, will this show up in the logs at all? if so, where?

question 2: anyone experienced this at all?

i was just going to downgrade, but i get errors on start, and am too tired to sort this out tonight....

[hanj]

I would look for any segfaults in /var/log/apache/error_log. Do you happen to use LDAP... and/or save PHP sessions to place other than /tmp? I've run into a few problems with either one of those two. If you can recreate the failure, you could try running strace on apache.. to see exactly where/why it died. Also, are you sure it's not related to logrotate? Are you certain that apache dies at random times?

HTH
hanji

[chashab]

thanks for the tips. i don't use LDAP and PHP sessions are at default (/tmp).

strace... i might have to try that.

i do use log rotate... do you mind explaining how this might be causing this? a quick google search didn't reveal much.

basically i start apache using: /etc/init.d/apache2 start and a couple days later apache is not running (no process running at all). so i start it again.

thanks!

[hanj]

Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.

Have you grep'd through your error_logs and looked for segfault?

Code: Select all

cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM'

I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.

strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.

hanji

[chashab]

thanks, searching for SIGTERM did the trick. I was wondering what to search for. by the way, i was wrong. logrotate isn't installed. i thought i had installed it, but had not.

unfortunetly, the only intresting thing in the logs is as follows, previous to that there are only the usual ".ico not found" type messages. mind applying your wealth of experience once again?

Code: Select all

[Wed Aug 03 21:16:59 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:05 2005] [error] child process 11125 still did not exit, sending a SIGKILL
[Wed Aug 03 21:17:06 2005] [notice] caught SIGTERM, shutting down

i also combed through the other apache logs (access_log, etc.) around that time and nothing of interest turned up

[chashab]

thanks for your help with these logs. i discovered a developer was restarting apache from webmin. i had a custom command that just ran:

/etc/init.d/apache2 restart

and as you said, apache was started before the child processes had died.

obviously the init script must be changed to wait for child processes to die before starting up again. how did you implement your fix?

[hanj]

The first problem I had was with the auth_digest_module. It seemed that would hang things up on restart (logrotate related), so I quit using that module:

/etc/apache2/conf/apache2.conf

Code: Select all

#LoadModule auth_digest_module            modules/mod_auth_digest.so

I would look into how webmin handles the restart as well. Based off of the doc...

http://httpd.apache.org/docs/2.0/stopping.html

You may want to change your /etc/conf.d/apache2:
change

Code: Select all

RESTARTSTYLE="graceful"
to 
RESTARTSTYLE="restart"

HTH
hanji

[chashab]

for webmin, i had just made a custom command that ran /etc/init.d/apache2 restart

your right, changing the restart style to restart would probably solve the problem. still gracefull should still work, should it not? so why doesn't it? i don't see any problems with the way the init.d script does it.

this only happens when requests are being served - iow, when you want gracefull to work.

i'll repost about this, maybe get some attention from some other apache users.

edit: my post on apache gracefull restart failing

[deoren]

Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.

Have you grep'd through your error_logs and looked for segfault?

Code: Select all

cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM'

I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.

strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.

hanji

That was the issue I was having.

After changing

Code: Select all

            /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true

to

Code: Select all

            /etc/init.d/apache2 restart 1> /dev/null || true

there were no more segfaults.

[kommissar]

After changing

Code: Select all

            /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true

to

Code: Select all

            /etc/init.d/apache2 restart 1> /dev/null || true

there were no more segfaults.

Me too... I don't understand why this works can you explain what this is doing?

[deoren]

I believe that in my case it is like hanj said:

Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died.

Here are the relevant sections of /etc/init.d/apache2:

Code: Select all

start() {
        checkconfig || return 1
        ebegin "Starting apache2"
        [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
        [ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars
        env -i PATH=$PATH /sbin/start-stop-daemon --quiet \
                --start --startas /usr/sbin/apache2 \
                --pidfile ${PIDFILE} -- -k start ${APACHE2_OPTS}
        eend $?
}

stop() {
        ebegin "Stopping apache2"
        /usr/sbin/apache2ctl stop >/dev/null
        start-stop-daemon -o --quiet --stop --pidfile ${PIDFILE}
        eend $?
}

As you'll notice the control script /usr/sbin/apache2ctl is used to stop Apache2 and Gentoo's /sbin/start-stop-daemon cleans up the pid file. When starting Apache2 back /sbin/start-stop-daemon handles that. I'm sure there is more to it than that, but that is what I understand of the process.

The important part to me is that Gentoo's script for handling Apache2 is reliable. I was worried after doing research that one of the modules I was relying on was responsible for the segfaults. I'm SO glad that isn't so.