Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[RESOLVED] New Versions of APC Hanging Apache
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Karma T. Foxx
n00b
n00b


Joined: 04 Apr 2008
Posts: 23
Location: Toronto

PostPosted: Wed Apr 18, 2012 2:50 pm    Post subject: [RESOLVED] New Versions of APC Hanging Apache Reply with quote

Hello,

After a recent update to a high volume web server (~300 cache hits per second) 32-bit VM pecl-apc-3.1.9-r2 and pecl-apc-3.1.7 with PHP 5.3.2 consistently hang apache-2.2.22. I had to find this through lengthy process of elimination because search as I might nothing relevant shows up in syslog's or apache's logs. After a few minutes, apache will just sit there, virtually idle and dishing out connection reset errors to clients. A cold start brings it back around but only momentarily. Everything runs fine (if not slowly) long-term when APC is disabled.

I noticed that there are new locking mechanisms configurable through the USE flags on 3.1.9 and locking problems sounds exactly like what I'm experiencing so I tried changing from POSIX threads mutual exclusion to semaphore with much the same results. I then downgraded APC to the lowest available stable version, which does not have USE flags for locking mechanisms (3.1.7) and I'm still having trouble.

APC is only being used as an opcode cache on this system; I'm not aware of any software taking advantage of its shared memory capabilities.

Here is my config file:
Code:
extension=apc.so
apc.enabled="1"
apc.shm_segments="8"
apc.shm_size="32M"
apc.num_files_hint="1512"
apc.ttl="1200"
apc.user_ttl="1200"
apc.gc_ttl="1200"
apc.cache_by_default="1"
apc.filters="(.*cache.*)|(.*tmp.*)"
;apc.mmap_file_mask="/tmp/apcphp5.XXXXXX"
apc.slam_defense="0"
apc.file_update_protection="10"
apc.enable_cli="0"
apc.max_file_size="2M"
apc.stat="1"
apc.write_lock="1"
apc.report_autofilter="0"
apc.include_once_override="0"
apc.rfc1867="1"
apc.rfc1867_prefix="upload_"
apc.rfc1867_name="APC_UPLOAD_PROGRESS"
apc.rfc1867_freq="40k"
apc.localcache="0"
apc.localcache.size="512"
apc.coredump_unmap="0"


emerge --info
Code:

# emerge --info
Portage 2.1.10.49 (hardened/linux/x86, gcc-4.5.3, glibc-2.13-r4, 3.2.12-gentoo i686)
=================================================================
System uname: Linux-3.2.12-gentoo-i686-Intel-R-_Xeon-R-_CPU_E5506_@_2.13GHz-with-gentoo-2.0.3
Timestamp of tree: Tue, 17 Apr 2012 18:00:01 +0000
app-shells/bash:          4.2_p20
dev-java/java-config:     2.1.11
dev-lang/python:          2.6.5-r2, 2.7.2-r3, 3.1.2-r3, 3.2.2
dev-util/cmake:           2.8.1-r2
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13::<unknown repository>, 2.68
sys-devel/automake:       1.4_p6-r1, 1.5-r1, 1.6.3-r1, 1.7.9-r2, 1.8.5-r4, 1.9.6-r3, 1.10.3, 1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.4.6-r1, 4.5.3-r2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.13-r4
Repositories: gentoo
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA dlj-1.1"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=i686 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs -fstack-protector-all"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=i686 -pipe -fomit-frame-pointer -mno-tls-direct-seg-refs -fstack-protector-all"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://gentoo.mirrors.tds.net/gentoo ftp://gentoo.cites.uiuc.edu/pub/gentoo/ http://gentoo.cites.uiuc.edu/pub/gentoo/"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en_CA en"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="acc acl agent apache2 bcmath berkdb bzip2 cgi cli courier cracklib crypt ctype curl curlwrappers cxx dri encode exif fastcgi ffmpeg filter flac frontend ftp gd gdbm geoip gpm hardened hash iconv imap ithreads jpeg json mmx modules mudflap mysql mysqli ncurses nls nptl nptlonly offensive ogg openmp pam pax_kernel pcre pdo php pic png postfix pppd readline rrdtool server session simplexml snmp sockets socks5 speex sse sse2 ssl svg sysfs tcpd theora threads tiff truetype unicode urandom usb vda vhosts vorbis win32codecs wmf x86 xml xorg xsl xvid zip zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="access auth auth_dbm auth_anon auth_digest alias file-cache echo charset-lite cache disk-cache mem-cache ext-filter case_filter case-filter-in deflate mime-magic cern-meta expires headers usertrack unique-id proxy proxy-connect proxy-ftp proxy-http info include cgi cgid dav dav-fs vhost-alias speling rewrite log_config logio env setenvif mime status autoindex asis negotiation dir imap actions userdir so filter unique_id authz_host exif truetype mod_security mod_evasion mod_bandwidth mod_throttle mod_cgi mod_rewrite authn_file auth_basic auth_digest authz_user" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en_CA en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON


Any insight would be greatly appreciated! Thanks.

UPDATE I can confirm that this is true with or without stack smashing protection enabled.
_________________
http://foxpa.ws - Gentoo, Xen and more!


Last edited by Karma T. Foxx on Thu Jun 28, 2012 1:22 am; edited 1 time in total
Back to top
View user's profile Send private message
Karma T. Foxx
n00b
n00b


Joined: 04 Apr 2008
Posts: 23
Location: Toronto

PostPosted: Thu Jun 28, 2012 1:22 am    Post subject: Reply with quote

Silly me, looks like the problem had to do with (unintentionally) switching to the threaded mpm_worker - or consequentially - thread safe PHP.

Back on mpm_prefork now and performance is looking better.

http://foxpa.ws/2012/06/27/mpm_worker-and-thread-safe-php-are-toxic-to-apc/
_________________
http://foxpa.ws - Gentoo, Xen and more!
Back to top
View user's profile Send private message
gr0x0rd
Guru
Guru


Joined: 12 Aug 2006
Posts: 311

PostPosted: Mon Jul 16, 2012 1:54 am    Post subject: Reply with quote

KTF, would you mind posting your ebuild versions and USE flags? I haven't had APC working since PHP5.2, but if you've gotten it working with 3, it would be great to know [detailed] how.

Cheers!
_________________
~g
emerge -FML...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum