[SOLVED] Memory leak in Apache-2.2.8 + PHP-5.2.6_rc1-r1

[astor84]

I have two http servers that I recently migrated our company's website and web services to. Both run apache-2.2.8 and php-5.2.6_rc1-r1 (although I was running 5.2.5 until last week when the new version was stabilized on x86). Before migrating our service to these new boxes, we ran apache-1.3.34 and php-5.2.5 on a single web server.

Last week I noticed that I was completely unable to log into one of the boxes via ssh, despite the fact that it responded to ping. With some investigation (and a friendly guy at the datacenter) I was able to get back into the box. Once ssh'd in, I took a look at top and noticed that all 2GB of my physical memory and all 2GB of my swap were used and that apache was the one using all of the memory. I restarted apache, the memory usage went down, and I figured perhaps it was a one-off issue.

The next day I looked again and both boxes were nearly out of memory. Again, I restarted apache on each box and the memory usage went down drastically (from around 3.8GB including physical and swap to about 180MB). Since then, I have cranked down MaxRequestPerChild (from 10000 to 1500) and played with a number of other settings, but have been quite unable to keep Apache from eating up all the RAM on my system. It seems like about every 24 hours or so, I'm out of RAM again and swapping like it's going out of style.

I have set MaxRequestsPerChild down to 500 now, in hopes that this will keep me from having to manually restart apache before it runs out of RAM, but of course, I need about another 24 hours to test it.

On my old server (with apache-1.3.34 + php-5.2.5) I never had this problem, so it seems logical that it's somehow related to apache2. Most of my content is dynamic, so I'm guessing it's an apache2 + mod_php issue.

Has anyone else had a similar problem? Is there anything obvious that I should be looking for?

Here's my emerge --info, in hopes that it will help:

Code: Select all

c01-http02 modules.d # emerge --info
Portage 2.1.4.4 (default-linux/x86/2007.0/server, gcc-4.1.2, glibc-2.6.1-r0, 2.6.23-gentoo-r3 i686)
=================================================================
System uname: 2.6.23-gentoo-r3 i686 Intel(R) Xeon(TM) CPU 3.06GHz
Timestamp of tree: Wed, 12 Mar 2008 12:00:01 +0000
app-shells/bash:     3.2_p17-r1
dev-lang/python:     2.4.4-r9
dev-python/pycrypto: 2.0.1-r6
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.61-r1
sys-devel/automake:  1.10
sys-devel/binutils:  2.18-r1
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.26
virtual/os-headers:  2.6.23-r3
ACCEPT_KEYWORDS="x86"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=pentium4 -O2 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/udev/rules.d"
CXXFLAGS="-march=pentium4 -O2 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FEATURES="ccache distlocks metadata-transfer parallel-fetch sandbox sfperms strict unmerge-orphans userfetch"
GENTOO_MIRRORS="http://distfiles.gentoo.org http://gentoo.ccccom.com"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
SYNC="rsync://216.109.213.221/gentoo-portage"
USE="apache2 bash-completion berkdb bzip2 calendar cgi cli cracklib crypt dri fastcgi fortran gd gpm iconv ipv6 isdnlog mailwrapper memcache mhash midi mudflap mysql ncurses nls nptl nptlonly openmp pam pcre perl php php5 postgres pppd python readline reflection session simplexml snmp spl sse sse2 sse3 ssl tcpd truetype unicode x86 xml xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" APACHE2_MPMS="prefork" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" USERLAND="GNU" VIDEO_CARDS="apm ark chips cirrus cyrix dummy fbdev glint i128 i740 i810 imstt mach64 mga neomagic nsc nv r128 radeon rendition s3 s3virge savage siliconmotion sis sisusb tdfx tga trident tseng v4l vesa vga via vmware voodoo"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, LINGUAS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY

Any takers?

EDIT: I almost forgot: I am using eAccelerator as well (which I used on my old server without issue). I am open to the possibility that this could be a problem and will be doing more testing later this week, especially if anyone thinks this is the problem.

[Daivil]

Hi,

I'm getting exactly the same problem (same apache version btw).

I tried almost every possible mpm configurations without any convenient result. I get a lot of traffic on that box and, right now, the only solution for me was to reload apache every 15 minutes via crontab (very bad I know...).

Any help would be appreciated...

Gaëtan

[piejagr]

I'm having the same problem.

Now I'm trying to downgrade php. If that won't help then I'll try downgrading apache.

[Daivil]

Let us know if that did the trick...

[piejagr]

I have downgraded both packages and still I'm having memory leaks.

I tried to recompile the both packages without use flag "threads" and instead enabling "sharedmem" use flag.
Currently I'm not experiencing any memory leaks but I must test that a longer period.

Maybe try it with your box'es and give info if it helped.

[Daivil]

Actually, I can't find any "sharedmem" option on Apache ebuild.

This said, neither PHP nor Apache have "threads" flag enabled by default.

[ingenious]

I've got exactly the same problem on a dedicated server running debian... But in my case it happened over night - it worked fine for over two months, and then suddenly (no updates or something) my memory is simply not enough...

[piejagr]

Apache doesn't have the sharemem flag only php. The leak in my case has only appeared when accessing .php files

Code: Select all

piejagr ~ # emerge -pvt php

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild   R   ] dev-lang/php-5.2.6_rc1-r1  USE="apache2 berkdb bzip2 cgi cli crypt ctype curl curlwrappers exif fastbuild force-cgi-redirect ftp gd hash iconv imap ldap mysql mysqli ncurses nls pcre pdo posix readline reflection session sharedext sharedmem simplexml sockets spell spl ssl tokenizer truetype unicode xml xsl zip zlib -adabas -bcmath -birdstep -calendar -cdb -cjk -concurrentmodphp -db2 -dbase -dbmaker -debug -discard-path -doc -empress -empress-bcs -esoob -fdftk -filter -firebird -flatfile -frontbase -gd-external -gdbm -gmp -inifile -interbase -iodbc -ipv6 (-java-external) -json -kerberos -ldap-sasl -libedit -mcve -mhash -msql -mssql -oci8 -oci8-instant-client -odbc -pcntl -pic -postgres -qdbm -recode -sapdb -snmp -soap -solid -sqlite -suhosin -sybase -sybase-ct -sysvipc -threads -tidy -wddx -xmlreader -xmlrpc -xmlwriter -xpm -yaz -zip-external" 

I have a vps serwer with guaranteed memory of 128 MB but max mem. which I can have is 512 MB.
The solution which I have posted worked for me, but I don't know if it will work on yours boxes.

[roymaster]

Do you all use eAccelerator?

Or simply the same php version?

I am using apache 2.2.6 and php 5.2.6RC1blahblah without such problem. Serving about 50 requests per second.

R.

[spottraining]

are your problems solved?

[wschalk]

Hi,

I am having the same problem and it really really sucks. Could somebody please file a bug? I cannot do it as I have to go to work now.

Thanks!

Bye,
Werner

[spottraining]

First we need to know - is it apache, PHP or eAccelerator problem.

I am waiting right now with update.

[astor84]

Bumping thread because I believe this problem may be fixed by the recent update of apache to 2.2.8-r3.

According to the changelog, the package bump is due to a security vunerability related to a memory leak in mod_ssl. I installed apache-2.2.8-r3 today and so far the memory usage on my HTTP servers is staying very reasonable, despite heavy day-time traffic (1-2 requests per second).

This seems to make sense, because about 95% of the traffic through our webservers goes through mod_ssl before being sent to the client.

I recommend upgrading apache to 2.2.8-r3 for anyone else who was having this problem. However, I am going to reserve judgement until I see what the memory usage looks like on Tuesday (as before it took a couple of days to truly get out of hand and start swap thrashing).

[egandt]

I just tried with:
www-servers/apache-2.2.8-r3 and dev-lang/php-5.2.6_rc4 and it still fails since I'm using sharedmem and threads:


*
* USE flag 'sharedmem' conflicts with these USE flag(s):
* threads
*
* You must disable these conflicting flags before you can emerge this package.
* You can do this by disabling these flags in /etc/portage/package.use:
* =dev-lang/php-5.2.6_rc4 -threads
*
*
* ERROR: dev-lang/php-5.2.6_rc4 failed.
* Call stack:
* ebuild.sh, line 49: Called pkg_setup
* php-5.2.6_rc4.ebuild, line 94: Called php5_2-sapi_pkg_setup
* php5_2-sapi.eclass, line 319: Called php5_2-sapi_check_use_flags
* php5_2-sapi.eclass, line 211: Called phpconfutils_use_conflict 'pkg_setup' 'pkg_setup'
* phpconfutils.eclass, line 167: Called die
* The specific snippet of code:
* die "Conflicting USE flags found"
* The die message:
* Conflicting USE flags found
*
* If you need support, post the topmost build error, and the call stack if relevant.
* A complete build log is located at '/var/tmp/portage/dev-lang/php-5.2.6_rc4/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/dev-lang/php-5.2.6_rc4/temp/die.env'.
*


adding "=dev-lang/php-5.2.6_rc4 -threads" does not help as apache that returns:

* You need to enable USE flag 'threads' to build a thread-safe version
* of dev-lang/php for use with www-servers/apache


Note that "=dev-lang/php-5.2.6_rc4 -sharedmem" will allow it to compile, but does it break something else is the question?

ERIC

[astor84]

It's been 6 days now since I upgraded and it seems like the memory leak it gone. I still have 500M of memory free and am hardly using any swap (whereas before I would be out of memory, using a ton of swap, and constantly having apache processes die when they can't malloc).

Solved!