Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Apache crashes every now and then
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
baak6
Tux's lil' helper
Tux's lil' helper


Joined: 30 Aug 2003
Posts: 141

PostPosted: Mon Feb 21, 2005 6:00 pm    Post subject: Apache crashes every now and then Reply with quote

Hello friends.

I'm running apache 2.0.52-r2 with SSL and PHP 4.3.10. A few days ago it started crashing for some reason, this is what the logs say:
Code:

[...]
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] Child 11310 returned a Fatal error...\nApache is exiting!
[Mon Feb 21 17:09:21 2005] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Mon Feb 21 17:09:21 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:34 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:11:03 2005] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Mon Feb 21 17:55:18 2005] [notice] Digest: generating secret for digest authentication ...
[Mon Feb 21 17:55:18 2005] [notice] Digest: done
[Mon Feb 21 17:55:19 2005] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon Feb 21 17:55:19 2005] [notice] Apache/2.0.52 (Gentoo/Linux) mod_ssl/2.0.52 OpenSSL/0.9.7e PHP/4.3.10 configured -- resuming normal operations

After re-emerging apache once it complained about not being able to use some file in /var/cache/apache2/, I checked and the apache2 dir didn't exist, so I made it and it stopped complaining and started, that's possibly related, it was something regarding SSL and that mutex thing...

Does anyone know what's going on and how to fix this? I'm going crazy over complaints about the server being down. :-(

Thanks in advance.
Back to top
View user's profile Send private message
j-m
Retired Dev
Retired Dev


Joined: 31 Oct 2004
Posts: 975

PostPosted: Mon Feb 21, 2005 6:08 pm    Post subject: Reply with quote

You need /var/cache/apache2/ for mod_ssl. There should be no problems once this missing directory exists. Check the owner and rights.

Code:

drwxr-xr-x   2 apache apache      48 May 27  2004 apache2
Back to top
View user's profile Send private message
baak6
Tux's lil' helper
Tux's lil' helper


Joined: 30 Aug 2003
Posts: 141

PostPosted: Mon Feb 21, 2005 6:10 pm    Post subject: Reply with quote

It was previously
Code:

drwxr-xr-x   2 apache root     72 feb 21 17:55 apache2

I changed it to apache:apache...I hope it works, thanks! :-)
Back to top
View user's profile Send private message
midknight_gentoo
n00b
n00b


Joined: 13 Jul 2003
Posts: 22

PostPosted: Tue Mar 01, 2005 12:58 am    Post subject: Reply with quote

my apache is doing the same thing now...
did this resolve the issue???
Back to top
View user's profile Send private message
baak6
Tux's lil' helper
Tux's lil' helper


Joined: 30 Aug 2003
Posts: 141

PostPosted: Thu Mar 03, 2005 4:25 am    Post subject: Reply with quote

It stopped crashing but now it started again...this i s driving me nuts.
Back to top
View user's profile Send private message
tlarco
n00b
n00b


Joined: 26 Feb 2004
Posts: 7

PostPosted: Mon Mar 07, 2005 4:24 pm    Post subject: Reply with quote

I also have the same thing happening. The error sounds like a limit on the number of logins or procs a UID is allowed to run and the resource becomes unavailable. I had some values defined in

/etc/security/limits.conf

that were suggested in the Gentoo Linux Security Guide (Code Listing 6.1). I am not sure if Apache uses these vaules, but the threshold being crossed could be set somewhere in apache itself and not in Linux.

Did you also apply hardening steps from the Gentoo Security Guide? This is the first time I have had this problem with the 10 Gentoo servers I use every day. The server in question is the only one with an entry in /etc/security/limits.conf. I removed the entries and will wait to see if it happens again. Hope this helps!

Tony
Back to top
View user's profile Send private message
Herring42
Guru
Guru


Joined: 10 Mar 2004
Posts: 373
Location: Buckinghamshire

PostPosted: Fri Mar 11, 2005 9:44 pm    Post subject: Reply with quote

I've been having these problems as well.

It became most noticable after I installed egroupware, especially when I connected kontact's contacts to it. It would crash every five minutes or so. Disconnecting kontact has improved things, but it still dies every so often.
_________________
"The problem with quotes on the internet is that it is difficult
to determine whether or not they are genuine." -- Abraham Lincoln
Back to top
View user's profile Send private message
midknight_gentoo
n00b
n00b


Joined: 13 Jul 2003
Posts: 22

PostPosted: Sat Mar 19, 2005 9:39 pm    Post subject: Reply with quote

i believe now i can trace mine to after i edited the limits.conf file for teh security howto... and it must be starving apache.. so it crashes.
is there a way in that file to allow apache more processes and not the whole system.
Back to top
View user's profile Send private message
midknight_gentoo
n00b
n00b


Joined: 13 Jul 2003
Posts: 22

PostPosted: Sun Mar 20, 2005 1:36 am    Post subject: Reply with quote

mine is now crashing all the damn time...
i had to disable ssl cause as soon as i go to horde it crashes right away

i removed all the stuff in the limits.conf as well....
this bothers me
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Mon Mar 21, 2005 10:24 pm    Post subject: Reply with quote

I am replying because I am in search of the same answer.. my apache is hard limited to 15 processes and nothing I do with any of the limits stuff seems to change that.
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Tue Mar 22, 2005 1:46 am    Post subject: Reply with quote

https://bugs.gentoo.org/show_bug.cgi?id=64700
Back to top
View user's profile Send private message
Ubnormal
n00b
n00b


Joined: 20 Aug 2004
Posts: 17

PostPosted: Thu May 05, 2005 10:07 pm    Post subject: Resource temporarily unavailable: setuid: unable to change t Reply with quote

Hi all!

I have the same problem, my Apache 2.0.54 craches a few moments after startup with the following lines in the error_log:

Code:

[Fri May 06 00:56:46 2005] [notice] Apache configured -- resuming normal operations
[Fri May 06 00:56:46 2005] [info] Server built: May  5 2005 23:51:35
[Fri May 06 00:56:46 2005] [debug] worker.c(1632): AcceptMutex: sysvsem (default: sysvsem)
[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri May 06 00:56:47 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Fri May 06 00:56:47 2005] [alert] Child 3542 returned a Fatal error...\nApache is exiting!
[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:48 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:48 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:53 2005] [error] child process 3521 still did not exit, sending a SIGKILL
[Fri May 06 00:56:53 2005] [error] child process 3523 still did not exit, sending a SIGKILL


When i compile apache with mpm-prefork i have error_log with multiple "(22)Invalid argument: couldn't grab the accept mutex" lines.

I have 2Gb ram, 99% idle system, clear /etc/security/limits.conf, ulimit shows "unlimited". So "Resource temporarily unavailable" message in error_log looks very strange...

It seems /etc/init.d/apache doesn't use start-stop-daemon as described in the upper post but the problem very similar. Is there any step by step instruction to solve the problem?
Back to top
View user's profile Send private message
krimhum
n00b
n00b


Joined: 18 Nov 2005
Posts: 1

PostPosted: Fri Nov 18, 2005 7:05 pm    Post subject: Reply with quote

I'm having the same problems and am seeking a solution.

I read on one of the apache sites that changing your AcceptMutex value will fix the issue.

I changed mine to "AcceptMutex fcntl" and am going to wait a week and see if it still crashes.

Regards,
Back to top
View user's profile Send private message
jensb
n00b
n00b


Joined: 20 Dec 2005
Posts: 2

PostPosted: Tue Dec 20, 2005 12:53 am    Post subject: apache not using pam to fork Reply with quote

Hi,

I had the same problem with mysql. It also used to many processes. Even though I changed the values for

Code:
mysql     hard      nproc      XX


in /etc/security/limits.conf didn't result in a change of behaviour, which I could verify in the logfiles. I use hardened sources with logging of resource violations enabled. The log still showed the same limit of YY processes that was violated To narrow down the problem I changed the wildcard entry for all users

Code:
*       hard        nproc       YY


to a different value (ZZ) than before. Now after crashing mysql again the log showed ZZ processes as violated limit. That means that this file was the reason for the behaviour. Still I didn't understand why the system didn't see XX as limit since mysql ran as user mysql. The reason is - as mentioned earlier -

https://bugs.gentoo.org/show_bug.cgi?id=64700

When mysql is started the user is switched from root to mysql. This switch does not query the pam. I downloaded the diffs provided by the devolopers that took care of this matter and recompiled the start-stop-daemom with those changes. The start-stop-daemon starts mysql and with the new version querys the pam for the userchange. That way all the limits are set correctly.

A similar problem applies to apache, I think. The root process spawns child processes that run under the uid of the configured user (i.e. apache) but does not query the pam while doing that and therefore no limits can be set. Maybe that is also here the reason for this error.

Jens
Back to top
View user's profile Send private message
jensb
n00b
n00b


Joined: 20 Dec 2005
Posts: 2

PostPosted: Tue Dec 20, 2005 9:55 pm    Post subject: solution Reply with quote

For me the a little change in /etc/init.d/apache2 did the trick:

Code:

start() {
   checkconfig || return 1
   ebegin "Starting apache2"
   [ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
   [ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars
   ulimit -u XX
   ${APACHE2} ${APACHE2_OPTS} -k start
   eend $?
}


I added the line: ulimit -u XX which limits the number of processes per user.

Limits don't apply for root; they are shown though when using

Code:
ulimit -a


but they have no effect. I guess these limits are now set for the session, so that all the child processes even though they're not root inherit these settings. Using this method work so far for me and now its possible to apply all the other limits like cpu-time, file-size etc. too.

Jens
Back to top
View user's profile Send private message
Rainbow goblin
Tux's lil' helper
Tux's lil' helper


Joined: 29 Feb 2004
Posts: 132

PostPosted: Tue Apr 25, 2006 8:09 am    Post subject: Reply with quote

You me have rescueed, thanks! :D
Back to top
View user's profile Send private message
padde
n00b
n00b


Joined: 19 Jan 2005
Posts: 52
Location: Germany

PostPosted: Wed Dec 20, 2006 9:22 am    Post subject: Reply with quote

Then - what's the solution? Will there be an update? I'm experiencing this error since today (running subversion on my apache since yesterday, so I think this has something to do with it).

http://rafb.net/paste/results/N94yxi51.html

I'm feeling _quite_ uncomfortable with a crashing webserver...
Back to top
View user's profile Send private message
padde
n00b
n00b


Joined: 19 Jan 2005
Posts: 52
Location: Germany

PostPosted: Wed Jan 30, 2008 1:40 pm    Post subject: again! Reply with quote

It happened again today. After a subversion update (Tue Jan 29 02:20:13 2008 >>> dev-util/subversion-1.4.6). I didn't restart apache after the update (which is probably the cause), and a few hours later during a (harmless) attack consisting of around 10-20 accesses/second to some non-existing php scripts the server went down with the following log entries:

Code:

...
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:27 2008] [alert] Child 4646 returned a Fatal error... Apache is exiting!
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex
...


Funny thing was that I immediately found this thread (and my own >1 year old post) ;)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum