Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
apache 2 appears to be dying at random!
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
chashab
n00b
n00b


Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia

PostPosted: Thu Aug 04, 2005 5:05 am    Post subject: apache 2 appears to be dying at random! Reply with quote

I've been running apache-2.0.54-r8 for a few weeks now. twice in the last week i have found it not running.

question 1: if apache dies, will this show up in the logs at all? if so, where?

question 2: anyone experienced this at all?

i was just going to downgrade, but i get errors on start, and am too tired to sort this out tonight....
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1266

PostPosted: Thu Aug 04, 2005 7:54 am    Post subject: Reply with quote

I would look for any segfaults in /var/log/apache/error_log. Do you happen to use LDAP... and/or save PHP sessions to place other than /tmp? I've run into a few problems with either one of those two. If you can recreate the failure, you could try running strace on apache.. to see exactly where/why it died. Also, are you sure it's not related to logrotate? Are you certain that apache dies at random times?

HTH
hanji
Back to top
View user's profile Send private message
chashab
n00b
n00b


Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia

PostPosted: Thu Aug 04, 2005 4:33 pm    Post subject: Reply with quote

thanks for the tips. i don't use LDAP and PHP sessions are at default (/tmp).

strace... i might have to try that.

i do use log rotate... do you mind explaining how this might be causing this? a quick google search didn't reveal much.

basically i start apache using: /etc/init.d/apache2 start and a couple days later apache is not running (no process running at all). so i start it again.

thanks!
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1266

PostPosted: Thu Aug 04, 2005 4:38 pm    Post subject: Reply with quote

Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.

Have you grep'd through your error_logs and looked for segfault?

Code:
cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM'


I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.

strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.

hanji
Back to top
View user's profile Send private message
chashab
n00b
n00b


Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia

PostPosted: Thu Aug 04, 2005 5:18 pm    Post subject: Reply with quote

thanks, searching for SIGTERM did the trick. I was wondering what to search for. by the way, i was wrong. logrotate isn't installed. i thought i had installed it, but had not.

unfortunetly, the only intresting thing in the logs is as follows, previous to that there are only the usual ".ico not found" type messages. mind applying your wealth of experience once again?

Code:
[Wed Aug 03 21:16:59 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:16:59 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16947 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:01 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 11125 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:03 2005] [warn] child process 16917 still did not exit, sending a SIGTERM
[Wed Aug 03 21:17:05 2005] [error] child process 11125 still did not exit, sending a SIGKILL
[Wed Aug 03 21:17:06 2005] [notice] caught SIGTERM, shutting down

i also combed through the other apache logs (access_log, etc.) around that time and nothing of interest turned up
Back to top
View user's profile Send private message
chashab
n00b
n00b


Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia

PostPosted: Thu Aug 04, 2005 6:04 pm    Post subject: Reply with quote

thanks for your help with these logs. i discovered a developer was restarting apache from webmin. i had a custom command that just ran:

/etc/init.d/apache2 restart

and as you said, apache was started before the child processes had died.

obviously the init script must be changed to wait for child processes to die before starting up again. how did you implement your fix?
Back to top
View user's profile Send private message
hanj
Veteran
Veteran


Joined: 19 Aug 2003
Posts: 1266

PostPosted: Thu Aug 04, 2005 6:45 pm    Post subject: Reply with quote

The first problem I had was with the auth_digest_module. It seemed that would hang things up on restart (logrotate related), so I quit using that module:

/etc/apache2/conf/apache2.conf

Code:
#LoadModule auth_digest_module            modules/mod_auth_digest.so


I would look into how webmin handles the restart as well. Based off of the doc...

http://httpd.apache.org/docs/2.0/stopping.html

You may want to change your /etc/conf.d/apache2:
change
Code:
RESTARTSTYLE="graceful"
to
RESTARTSTYLE="restart"


HTH
hanji
Back to top
View user's profile Send private message
chashab
n00b
n00b


Joined: 16 Jun 2004
Posts: 71
Location: Republic of Alumbia

PostPosted: Thu Aug 04, 2005 8:04 pm    Post subject: Reply with quote

for webmin, i had just made a custom command that ran /etc/init.d/apache2 restart

your right, changing the restart style to restart would probably solve the problem. still gracefull should still work, should it not? so why doesn't it? i don't see any problems with the way the init.d script does it.

this only happens when requests are being served - iow, when you want gracefull to work.

i'll repost about this, maybe get some attention from some other apache users.

edit: my post on apache gracefull restart failing
Back to top
View user's profile Send private message
deoren
Apprentice
Apprentice


Joined: 23 Dec 2004
Posts: 227
Location: USA

PostPosted: Mon Sep 19, 2005 1:56 pm    Post subject: Reply with quote

hanj wrote:
Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died. I've run into this problem on one server.. where I had to make it sleep for 10 seconds before the 'start' call.

Have you grep'd through your error_logs and looked for segfault?

Code:
cat /var/log/apache2/error_log | grep -E 'segfault|SIGTERM'


I think the answer will be around that point. Once you find a time of the fault.. you can less the file and see what happened before the the fault.

strace won't do you much.. unless you can replicate the fault. Meaning.. if you know a certain PHP script crashes apache.. you can run strace and see where apache died.

hanji


That was the issue I was having.

After changing
Code:
            /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true


to
Code:
            /etc/init.d/apache2 restart 1> /dev/null || true


there were no more segfaults.
Back to top
View user's profile Send private message
kommissar
Tux's lil' helper
Tux's lil' helper


Joined: 19 May 2005
Posts: 78

PostPosted: Thu Sep 22, 2005 4:21 am    Post subject: Reply with quote

deoren wrote:

After changing
Code:
            /bin/kill -HUP `cat /var/run/apache2.pid` 1> /dev/null || true


to
Code:
            /etc/init.d/apache2 restart 1> /dev/null || true


there were no more segfaults.


Me too... I don't understand why this works can you explain what this is doing?
Back to top
View user's profile Send private message
deoren
Apprentice
Apprentice


Joined: 23 Dec 2004
Posts: 227
Location: USA

PostPosted: Thu Sep 22, 2005 10:49 am    Post subject: Reply with quote

I believe that in my case it is like hanj said:

Quote:
Sometimes when logrotate is rotating the scripts.. it tries to restart apache before all the child process have died.


Here are the relevant sections of /etc/init.d/apache2:

Code:
start() {
        checkconfig || return 1
        ebegin "Starting apache2"
        [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
        [ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars
        env -i PATH=$PATH /sbin/start-stop-daemon --quiet \
                --start --startas /usr/sbin/apache2 \
                --pidfile ${PIDFILE} -- -k start ${APACHE2_OPTS}
        eend $?
}

stop() {
        ebegin "Stopping apache2"
        /usr/sbin/apache2ctl stop >/dev/null
        start-stop-daemon -o --quiet --stop --pidfile ${PIDFILE}
        eend $?
}


As you'll notice the control script /usr/sbin/apache2ctl is used to stop Apache2 and Gentoo's /sbin/start-stop-daemon cleans up the pid file. When starting Apache2 back /sbin/start-stop-daemon handles that. I'm sure there is more to it than that, but that is what I understand of the process.

The important part to me is that Gentoo's script for handling Apache2 is reliable. I was worried after doing research that one of the modules I was relying on was responsible for the segfaults. I'm SO glad that isn't so. :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum