Apache: No listening sockets available, shutting down

[Cygon]

Hi!

My Apache is refusing to start with the error message in this thread's subject line:

Code: Select all

~ # apache2
no listening sockets available, shutting down
Unable to open logs

But I can't figure out what is wrong. There is no other process blocking port 80:

Code: Select all

~ # netstat -anA inet

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:64738           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8005          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:10022           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8009            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:783           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:52885         127.0.0.1:783           TIME_WAIT
tcp        0      0 127.0.0.1:3306          127.0.0.1:58043         ESTABLISHED
tcp        0      0 127.0.0.1:3306          127.0.0.1:58050         ESTABLISHED
tcp        0      0 127.0.0.1:3306          127.0.0.1:58051         ESTABLISHED
tcp        0      0 127.0.0.1:58043         127.0.0.1:3306          ESTABLISHED
tcp        0      0 10.8.0.1:8080           10.8.0.6:35415          FIN_WAIT2
tcp        0    232 10.8.0.1:10022          10.8.0.6:52870          ESTABLISHED
tcp        0      0 127.0.0.1:58050         127.0.0.1:3306          ESTABLISHED
tcp        0      0 127.0.0.1:3306          127.0.0.1:58045         ESTABLISHED
tcp        0      0 127.0.0.1:58049         127.0.0.1:3306          ESTABLISHED
tcp        0      0 127.0.0.1:3306          127.0.0.1:58049         ESTABLISHED
tcp        0      0 10.8.0.1:56114          10.8.0.6:19090          ESTABLISHED
tcp        0      0 127.0.0.1:58045         127.0.0.1:3306          ESTABLISHED
tcp        0      0 127.0.0.1:58051         127.0.0.1:3306          ESTABLISHED
tcp        0      0 10.8.0.1:8080           10.8.0.6:35414          TIME_WAIT

I verified via nmap from an outside server, port 80 is indeed closed.

I then moved all files in /etc/apache2/vhosts.d to another directory (so there are no "Listen" directives at all for Apache). This seems to have no effect on the problem

Finally, I re-emerged (--oneshot) Apache, no luck either.

Can anyone imagine what might be going wrong? I'd also be grateful for any debugging tips you could give me ("apache --help" just outputs the error message, is there anything like "apache --debug" or "apache --verbose" ?)

[Manko10]

Make sure that Apache is started as root and you have exactly one

Code: Select all

Listen 80

directive in your configs.

[Cygon]

Make sure that no other service is occupying port 80, Apache is started as root and you have exactly one

Code: Select all

Listen 80

directive in your configs.

You didn't read my post at all, did you?

Because I was expecting this, I took all effort to show that port 80 is not blocked. And to remove any and all "Listen" directives (in fact, all VirtualHost configs) from my Apache.

[Manko10]

I just read the part with the blocking service not carefully enough, but I've already edited that out, you were only to fast.
But you should still check your Listen directives. You need exactly one, not two and not zero. Perhaps you had two before and now you have none.

[Cygon]

Maybe I should add that my Apache was working fine for about a year now and that I didn't change its configuration recently. The error just popped up after I rebooted my webserver.

There's nothing written to /var/logs/apache2/* either, which is why I resorted to running apache directly from the shell. Here's what apache2 -X -e Debug procures:

Code: Select all

[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module alias_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module auth_basic_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authn_anon_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authn_file_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authz_default_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authz_groupfile_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authz_host_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module authz_user_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module autoindex_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module deflate_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module dir_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module env_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module filter_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module headers_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module log_config_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module mime_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module mime_magic_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module negotiation_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module rewrite_module
[Sat Dec 04 18:37:39 2010] [debug] mod_so.c(246): loaded module speling_module
no listening sockets available, shutting down
Unable to open logs

[Cygon]

I just read the part with the blocking service not carefully enough, but I've already edited that out, you were only to fast.
But you should still check your Listen directives. You need exactly one, not two and not zero. Perhaps you had two before and now you have none.

Alright alright

I only have the default vhost configurations left in my /etc/apache2/vhosts.d:

• 00_default_ssl_vhost.conf (not enabled)
• 00_default_vhost.conf

There's only one Listen 80 directive in there.

Just for fun, I tried adding a second one, but this didn't change the output. Hrm...

[Manko10]

What about your /etc/apache2/httpd.conf? Is there any Listen directive? Is this directive outside of Vhost config tags?

[Hu]

Since you can run Apache from a shell, try running it under strace to see exactly what errors it encounters before it stops.

[Cygon]

What about your /etc/apache2/httpd.conf? Is there any Listen directive? Is this directive outside of Vhost config tags?

I haven't touched my httpd.conf, so it should still be identical to the one in the Apache ebuild. Just did a quick grep, no "Listen" in there.
The only Listen directive I have (in 00_default_vhost.conf) is outside of the vhost tags.

In my shell tests, I did forget to specify the defined (-D DEFAULT_VHOST -D PHP5 etc.), so I wasn't testing with the same environment as the init script. I'll have to pick all the defines from my /etc/conf.d/apache2 and retry the shell test with those, will report back

[Cygon]

That didn't give me the error message I was hoping for:

Code: Select all

apache2 -D DEFAULT_VHOST -D INFO -D SSL -D LANGUAGE -D SUEXEC -D PHP5 -D PYTHON -D DAV -D DAV_FS -D AUTH_DIGEST -D SVN -D PROXY

(no output at all, but Apache terminates)

In /var/log/apache2/error_log:

Code: Select all

[Sat Dec 04 19:14:53 2010] [info] Init: Initialized OpenSSL library

I'll try if I can get more info with strace as Hu recommended.

[Cygon]

Just to be thorough, I investigated my SSL certificates (as Apache terminated right after initializing OpenSSL according to error_log).

It looks like my server "lost" the partition the SSL certificates where stored on, thus Apache couldn't read them. I would have hoped for a clearer error message (like "Couldn't read SSL certificate file from /restricted/directory/domain.com.crt") but, oh well, at least I found the problem now.

The certificates were on a softraid-1 partition no less. This is going to be a very interesting evening - but that's another issue.

Thanks for the help so far! It's great to have so many friendly people on this forum

[Hu]

I would have hoped for a clearer error message (like "Couldn't read SSL certificate file from /restricted/directory/domain.com.crt") but, oh well, at least I found the problem now.

Once you have the immediate problem fixed, please consider filing an enhancement request with the Apache project to improve the error message.

[madchaz]

Looks like everyone missed the last line of error.

Code: Select all

Unable to open logs

you have read/write permisions with the user apache runs on to those files? Is the FS full by any chance?

[Cygon]

No problem there

One might guess that it's because my log files were also on the disappeared RAID partition, but that's not the case either. It's just a strange a follow-up error, the log files are writable and there's space for at least another 100 years of uptime. After I restored my certificates to a new location and updated the links, Apache came up fine and logging is working as it should.