apache broken after upgrade

[redwood]

I have an i686 asterisk/freepbx gentoo server running apache2.
After an 'emerge -uvDN world' yesterday, I followed the guidelines for upgrading to apache-2.2.6-r6 :
http://www.gentoo.org/doc/en/apache-upgrading.xml#use-2.2.6-r4

Apache starts, but I can no longer connect. apachetop shows nothing happening.
I've tried downgrading to earlier versions of apache and using the old apache2-builtin-mods
instead of APACHE2_MODULES and APACHE2_MPMS. But nothing has worked. Of course the problem
may have nothing to do with the apache changes, it could be some other package upgrade which broke apache.

#eix www-servers/apache
Installed versions: 2.2.6-r5(2)(23:02:18 12/18/07)(debug mpm-worker ssl threads -doc -ldap -mpm-event -mpm-itk -mpm-peruser -mpm-prefork -no-suexec -selinux -static-modules)

# lsof | grep http
apache2 29773 root 3u IPv6 1585739 TCP *:https (LISTEN)
apache2 29773 root 5u IPv6 1585744 TCP *:http (LISTEN)

# equery depends www-servers/apache
[ Searching for packages depending on www-servers/apache... ]
dev-lang/php-5.2.5-r1 (apache2? =www-servers/apache-2*)
dev-perl/Apache-Test-1.29 (www-servers/apache)
dev-php5/eaccelerator-0.9.5.1 (apache2? =www-servers/apache-2*)
dev-util/subversion-1.4.5 (apache2? =www-servers/apache-2*)
sci-misc/boinc-5.8.15 (server? www-servers/apache)
virtual/httpd-cgi-0 (www-servers/apache)
www-apache/mod_chroot-0.5 (=www-servers/apache-2*)
www-apache/mod_ftpd-0.14-r1 (=www-servers/apache-2.2*)
www-apache/mod_perl-2.0.3-r1 (=www-servers/apache-2*)
www-apache/mod_suphp-0.6.2-r3 (=www-servers/apache-2*)
www-apache/mod_umask-0.1.0 (=www-servers/apache-2*)


# eix dev-libs/apr
[I] dev-libs/apr
Available versions:
(0) 0.9.12 [M](~)0.9.13
(1) 1.2.11 [m](~)1.2.12
{debug doc ipv6 urandom}
Installed versions: 0.9.12(20:54:02 12/18/07)(ipv6 -urandom)
1.2.11(1)(16:19:43 12/18/07)(debug ipv6 -doc -urandom)
Homepage: http://apr.apache.org/
Description: Apache Portable Runtime Library

[I] dev-libs/apr-util
Available versions:
(0) 0.9.12 0.9.12-r1 [M](~)0.9.13
(1) 1.2.10 [m](~)1.2.12
{berkdb doc gdbm ldap mysql postgres sqlite sqlite3}
Installed versions: 0.9.12-r1(20:54:48 12/18/07)(berkdb gdbm -ldap)
1.2.10(1)(16:21:56 12/18/07)(berkdb gdbm -doc -ldap -mysql -postgres -sqlite -sqlite3)
Homepage: http://apr.apache.org/
Description: Apache Portable Runtime Utility Library



I've also rebuilt the apache tool stack following the guide
http://www.gentoo.org/doc/en/apache-troubleshooting.xml
but still no luck. When I try to connect with firefox, I simply get the status bar message
"Waiting to connect to http://mysite ..."
But /var/log/apache/access_log shows nothing happening.



When I startup apache I get these strange debug statements which I don't get on my four other apache servers:
# head /var/log/apache2/startuperror.log
POOL DEBUG: [29771/3083323600] PALLOC ( 1201/ 15271/ 15739) 0x080A0510 "process" <strings/apr_strings.c:78> (24/24/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 13383/ 13867/ 15851) 0x080A07E0 "pconf" <config.c:1954> (627/627/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 13455/ 13939/ 15923) 0x080A07E0 "pconf" <file_io/unix/open.c:239> (628/628/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 13471/ 13955/ 15939) 0x080A07E0 "pconf" <config.c:1972> (629/629/0)
POOL DEBUG: [29771/3083323600] PALLOC ( 13485/ 13969/ 15953) 0x080A07E0 "pconf" <strings/apr_strings.c:362> (630/630/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 13653/ 14137/ 16121) 0x080A07E0 "pconf" <network_io/unix/sockaddr.c:378> (631/631/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 14181/ 14665/ 16649) 0x080A07E0 "pconf" <config.c:261> (632/632/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 14185/ 14669/ 16653) 0x080A07E0 "pconf" <mod_so.c:116> (633/633/0)
POOL DEBUG: [29771/3083323600] PALLOC ( 14205/ 14689/ 16673) 0x080A07E0 "pconf" <tables/apr_tables.c:88> (634/634/0)
POOL DEBUG: [29771/3083323600] PCALLOC ( 15229/ 15713/ 17697) 0x080A07E0 "pconf" <tables/apr_tables.c:66> (635/635/0)



Does anybody know what these POOL DEBUG statements mean?

[redwood]

I found this reference in the gentoo forums about apr verbose debugging:
https://forums.gentoo.org/viewtopic-t-583880-highlight-pool+debug.html

So I decided to disable the verbose debugging:
#USE="-debug" emerge apr

Then I tested apache:

# apache2 --help
apache2: Syntax error on line 93 of /etc/apache2/httpd.conf: Cannot load /usr/lib/apache2/modules/mod_dir.so into server: /usr/lib/apache2/modules/mod_dir.so: undefined symbol: apr_pool_join

OK, so I then re-emerged apache
#emerge apache

Now apache works!!!

Previously, I had emerged apr, apr-util, and apache umpteen times but apache would never work.
So I conclude that there is a bug in apr's debug handling.