| View previous topic :: View next topic |
| Author |
Message |
chashab
n00b
Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia
|
 Posted: Thu Aug 04, 2005 5:05 am Post subject: apache 2 appears to be dying at random! |
 |
|
I've been running apache-2.0.54-r8 for a few weeks now. twice in the last week i have found it not running.
question 1: if apache dies, will this show up in the logs at all? if so, where?
question 2: anyone experienced this at all?
i was just going to downgrade, but i get errors on start, and am too tired to sort this out tonight.... |
|
| Back to top |
|
 |
hanj
Veteran
Joined: 19 Aug 2003
Posts: 1490
|
| Posted: Thu Aug 04, 2005 7:54 am Post subject: |
|
|
I would look for any segfaults in /var/log/apache/error_log. Do you happen to use LDAP... and/or save PHP sessions to place other than /tmp? I've run into a few problems with either one of those two. If you can recreate the failure, you could try running strace on apache.. to see exactly where/why it died. Also, are you sure it's not related to logrotate? Are you certain that apache dies at random times?
HTH
hanji |
|
| Back to top |
|
|
chashab
n00b
Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia
|
| Posted: Thu Aug 04, 2005 4:33 pm Post subject: |
|
|
thanks for the tips. i don't use LDAP and PHP sessions are at default (/tmp).
strace... i might have to try that.
i do use log rotate... do you mind explaining how this might be causing this? a quick google search didn't reveal much.
basically i start apache using: /etc/init.d/apache2 start and a couple days later apache is not running (no process running at all). so i start it again.
thanks! |
|
| Back to top |
|
|
hanj
Veteran
Joined: 19 Aug 2003
Posts: 1490
|
| Posted: Thu Aug 04, 2005 4:38 pm Post subject: |
|
|
Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.
Have you grep'd through your error_logs and looked for segfault?
| Code: | | cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM' |
I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.
strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.
hanji |
|
| Back to top |
|
|
chashab
n00b
Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia
|
| Posted: Thu Aug 04, 2005 5:18 pm Post subject: |
|
|
thanks, searching for SIGTERM did the trick. I was wondering what to search for. by the way, i was wrong. logrotate isn't installed. i thought i had installed it, but had not.
unfortunetly, the only intresting thing in the logs is as follows, previous to that there are only the usual ".ico not found" type messages. mind applying your wealth of experience once again?
| Code: | [Wed Aug 03 21:16:59 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:05 2005] [error] child process 11125 still did not exit, sending a SIGKILL
[Wed Aug 03 21:17:06 2005] [notice] caught SIGTERM, shutting down |
i also combed through the other apache logs (access_log, etc.) around that time and nothing of interest turned up |
|
| Back to top |
|
|
chashab
n00b
Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia
|
| Posted: Thu Aug 04, 2005 6:04 pm Post subject: |
|
|
thanks for your help with these logs. i discovered a developer was restarting apache from webmin. i had a custom command that just ran:
/etc/init.d/apache2 restart
and as you said, apache was started before the child processes had died.
obviously the init script must be changed to wait for child processes to die before starting up again. how did you implement your fix? |
|
| Back to top |
|
|
hanj
Veteran
Joined: 19 Aug 2003
Posts: 1490
|
| Posted: Thu Aug 04, 2005 6:45 pm Post subject: |
|
|
The first problem I had was with the auth_digest_module. It seemed that would hang things up on restart (logrotate related), so I quit using that module:
/etc/apache2/conf/apache2.conf
| Code: | | #LoadModule auth_digest_module modules/mod_auth_digest.so |
I would look into how webmin handles the restart as well. Based off of the doc...
http://httpd.apache.org/docs/2.0/stopping.html
You may want to change your /etc/conf.d/apache2:
change
| Code: | RESTARTSTYLE="graceful"
to
RESTARTSTYLE="restart" |
HTH
hanji |
|
| Back to top |
|
|
chashab
n00b
Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia
|
| Posted: Thu Aug 04, 2005 8:04 pm Post subject: |
|
|
for webmin, i had just made a custom command that ran /etc/init.d/apache2 restart
your right, changing the restart style to restart would probably solve the problem. still gracefull should still work, should it not? so why doesn't it? i don't see any problems with the way the init.d script does it.
this only happens when requests are being served - iow, when you want gracefull to work.
i'll repost about this, maybe get some attention from some other apache users.
edit: my post on apache gracefull restart failing |
|
| Back to top |
|
|
deoren
Apprentice
Joined: 23 Dec 2004
Posts: 227
Location: USA
|
| Posted: Mon Sep 19, 2005 1:56 pm Post subject: |
|
|
| hanj wrote: | Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.
Have you grep'd through your error_logs and looked for segfault?
| Code: | | cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM' |
I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.
strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.
hanji |
That was the issue I was having.
After changing
| Code: | | /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true |
to
| Code: | | /etc/init.d/apache2 restart 1> /dev/null || true |
there were no more segfaults. |
|
| Back to top |
|
|
kommissar
Tux's lil' helper
Joined: 19 May 2005
Posts: 78
|
| Posted: Thu Sep 22, 2005 4:21 am Post subject: |
|
|
| deoren wrote: |
After changing
| Code: | | /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true |
to
| Code: | | /etc/init.d/apache2 restart 1> /dev/null || true |
there were no more segfaults. |
Me too... I don't understand why this works can you explain what this is doing? |
|
| Back to top |
|
|
deoren
Apprentice
Joined: 23 Dec 2004
Posts: 227
Location: USA
|
| Posted: Thu Sep 22, 2005 10:49 am Post subject: |
|
|
I believe that in my case it is like hanj said:
| Quote: | | Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. |
Here are the relevant sections of /etc/init.d/apache2:
| Code: | start() {
checkconfig || return 1
ebegin "Starting apache2"
[ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
[ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars
env -i PATH=$PATH /sbin/start-stop-daemon --quiet \
--start --startas /usr/sbin/apache2 \
--pidfile ${PIDFILE} -- -k start ${APACHE2_OPTS}
eend $?
}
stop() {
ebegin "Stopping apache2"
/usr/sbin/apache2ctl stop >/dev/null
start-stop-daemon -o --quiet --stop --pidfile ${PIDFILE}
eend $?
} |
As you'll notice the control script /usr/sbin/apache2ctl is used to stop Apache2 and Gentoo's /sbin/start-stop-daemon cleans up the pid file. When starting Apache2 back /sbin/start-stop-daemon handles that. I'm sure there is more to it than that, but that is what I understand of the process.
The important part to me is that Gentoo's script for handling Apache2 is reliable. I was worried after doing research that one of the modules I was relying on was responsible for the segfaults. I'm SO glad that isn't so.  |
|
| Back to top |
|
|
|
Networking & Security
