Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Apache eats space between restarts
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Tue Nov 20, 2012 3:38 pm    Post subject: Apache eats space between restarts Reply with quote

Restarting apache brings my df usage down by 10 percent or more, even though /var/log/apache2 has nothing over 600k in it due to my syslog-ng/logrotate config. Why is apache eating up space and what files/dirs are taking up that kind of space between apache restarts? When I restart, I have not deleted anything from /var/log/apache2, so it has to be elsewhere. /tmp? /var/tmp?
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
Carnildo
Guru
Guru


Joined: 17 Jun 2004
Posts: 585

PostPosted: Wed Nov 21, 2012 11:14 am    Post subject: Reply with quote

When you rotate your log files, do you tell Apache to close the old one and start writing to the new one?
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Wed Nov 21, 2012 1:45 pm    Post subject: Reply with quote

As far as I can tell. I have this in /etc/logrotate.d/syslog-ng:

Code:

/var/log/apache2/* {
    compress
    size 1M
    rotate 5
    missingok
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
    endscript
}


Although, come to think of it I should fork that off to a separate logrotate.d file for apache so I can manage individual web sites.
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Thu Nov 29, 2012 6:02 pm    Post subject: Reply with quote

I just restarted apache - and did absolutely nothing else - and that freed up about 2 GB of space.

Why?
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
faemin
n00b
n00b


Joined: 16 Oct 2012
Posts: 22

PostPosted: Sat Dec 01, 2012 7:12 am    Post subject: Reply with quote

...

Last edited by faemin on Sun Dec 02, 2012 9:38 pm; edited 1 time in total
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Sat Dec 01, 2012 6:49 pm    Post subject: Reply with quote

I found out about htcacheclean and now have this running:

Code:

htcacheclean -d 1440 -p /var/cache/apache2/ -l 5M


But I still see df usage creeping up, and it's due to Apache somehow.

I didn't have lsof, but I emerged it. I see some output, but am not sure what to do with it.

/etc/apache2/httpd.conf:

Code:



ServerRoot "/usr/lib64/apache2"

LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfDefine CACHE>
LoadModule cache_module modules/mod_cache.so
</IfDefine>
LoadModule cgi_module modules/mod_cgi.so
LoadModule cgid_module modules/mod_cgid.so
<IfDefine DAV>
LoadModule dav_module modules/mod_dav.so
</IfDefine>
<IfDefine DAV>
LoadModule dav_fs_module modules/mod_dav_fs.so
</IfDefine>
<IfDefine DAV>
LoadModule dav_lock_module modules/mod_dav_lock.so
</IfDefine>
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
<IfDefine CACHE>
LoadModule disk_cache_module modules/mod_disk_cache.so
</IfDefine>
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule ext_filter_module modules/mod_ext_filter.so
<IfDefine CACHE>
LoadModule file_cache_module modules/mod_file_cache.so
</IfDefine>
LoadModule filter_module modules/mod_filter.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
<IfDefine INFO>
LoadModule info_module modules/mod_info.so
</IfDefine>
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
<IfDefine CACHE>
LoadModule mem_cache_module modules/mod_mem_cache.so
</IfDefine>
LoadModule mime_module modules/mod_mime.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule speling_module modules/mod_speling.so
<IfDefine SSL>
LoadModule ssl_module modules/mod_ssl.so
</IfDefine>
<IfDefine STATUS>
LoadModule status_module modules/mod_status.so
</IfDefine>
LoadModule unique_id_module modules/mod_unique_id.so
<IfDefine USERDIR>
LoadModule userdir_module modules/mod_userdir.so
</IfDefine>
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so

User apache
Group apache
Include /etc/apache2/modules.d/*.conf
Include /etc/apache2/vhosts.d/*.conf

CustomLog /var/log/apache2/access_log combined

# vim: ts=4 filetype=apache

ServerName (ip_address)


/etc/conf.d/apache2:

Code:


# /etc/conf.d/apache2: config file for /etc/init.d/apache2

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D AUTH_IMAP -D PERL"


(I deleted all the commented out lines that are there when installed.)

I have several vhosts, and they all more or less look like:

Code:

<VirtualHost *:80>
ServerAdmin (email_address)
DocumentRoot "/path/to/location"
ServerName (domain.com)
ServerAlias (www.domain.com)
<Directory "/path/to/location">
    Options Indexes FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>
</VirtualHost>
CustomLog /var/log/apache2/access_log_domain combined


Does anything look like it would cause apache to fill up disk space somewhere?
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
faemin
n00b
n00b


Joined: 16 Oct 2012
Posts: 22

PostPosted: Sun Dec 02, 2012 1:40 am    Post subject: Reply with quote

...

Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2454
Location: Silver City, NM

PostPosted: Sun Dec 02, 2012 2:01 am    Post subject: Reply with quote

As Carnildo mentioned, as long as a file is open in a program it is not actually deleted even though you've removed all directories entries for that file with the rm command.

The simplest thing is to simply restart Apache after rotating the log files. This is generally done by sending it a HUP signal. See here:
https://httpd.apache.org/docs/2.2/stopping.html
Quote:
Signal: HUP
apachectl -k restart

Sending the HUP or restart signal to the parent causes it to kill off its children like in TERM, but the parent doesn't exit. It re-reads its configuration files, and re-opens any log files. Then it spawns a new set of children and continues serving hits.
Back to top
View user's profile Send private message
faemin
n00b
n00b


Joined: 16 Oct 2012
Posts: 22

PostPosted: Sun Dec 02, 2012 4:57 am    Post subject: Reply with quote

...

Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2454
Location: Silver City, NM

PostPosted: Sun Dec 02, 2012 5:17 am    Post subject: Reply with quote

faemin wrote:
He posted that he has already restarted apache several times....
The OP actually said:
Quote:
Restarting apache brings my df usage down by 10 percent or more, even though /var/log/apache2 has nothing over 600k in it due to my syslog-ng/logrotate config.

This is exactly what I was talking about. There was nothing really wrong, The OP just didn't understand what was happening. Once you realize inodes for open files don't magically disappear when the files are deleted then it all makes sense. It's like hardlinks. You need to remove every hardlink to a file before the actual file gets deleted and frees up its disk space. I believe there is an accompanying problem of losing log information between the time the logs are rotated and when Apache is restarted.

You just need to restart Apache after you rotate the logs. This is pretty standard stuff.
Back to top
View user's profile Send private message
faemin
n00b
n00b


Joined: 16 Oct 2012
Posts: 22

PostPosted: Sun Dec 02, 2012 6:26 am    Post subject: Reply with quote

...

Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2454
Location: Silver City, NM

PostPosted: Sun Dec 02, 2012 7:28 am    Post subject: Reply with quote

I didn't say it was a hard link. I said it was like a hard link. I was trying to explain how it works. If you have N hard links to one file then all N need to be removed before you can free up space from the file (barring truncation). Likewise when N programs have a particular file open then all N programs need to close the file before you can free up its disk space (again, barring truncation).

It is true that copy-and-truncate is an alternative approach to log rotation but is is usually only used when the program that controls the logs cannot be told to re-open its log files. It is less safe and less efficient than the standard approach. As the logrotate man page explains:
Code:
copytruncate
    Truncate the original log file in place after creating a copy,
    instead  of moving  the old log file and optionally creating a new
    one.  It can be used when some program cannot be told to close its
    logfile and thus  might  continue  writing  (appending)  to  the
    previous log file forever.   Note that there is a very small time
    slice between copying the  file  and  truncating it, so some
    logging data might be lost.  When this option is used, the cre-
    ate option will have no effect, as the old log file stays in
    place.
Apache is not broken in this way. It was designed to respond to the HUP signal by re-reading its config files and re-opening its log files which was explained on the Apache documentation page I linked to. Furthermore, I didn't say restarting Apache was the only way to deal with the problem, I said it was the simplest.

I didn't post here to get in a series of fights with you. I simply wanted to help audiodef understand and fix his problem.
Back to top
View user's profile Send private message
faemin
n00b
n00b


Joined: 16 Oct 2012
Posts: 22

PostPosted: Sun Dec 02, 2012 8:38 am    Post subject: Reply with quote

...

Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
cach0rr0
Moderator
Moderator


Joined: 13 Nov 2008
Posts: 4121
Location: Houston, Republic of Texas

PostPosted: Sun Dec 02, 2012 10:56 am    Post subject: Reply with quote

This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying.

Address the original poster's questions or shut up.

audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Sun Dec 02, 2012 3:38 pm    Post subject: Reply with quote

cach0rr0 wrote:
This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying.

Address the original poster's questions or shut up.

audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity.


No worries. I appreciate that people were trying to help. :)

I don't mind having Apache restart after log rotation. Could I simply put apache2ctl restart in the postrotate section of /etc/logrotate.d/apache2?

Two things:

1. I noticed that -k is not a valid option for apache2ctl.

2. Whether I use apache2ctl or /etc/init.d/apache2 restart, I have to do it twice because the first time, I get "start-stop-daemon: /usr/sbin/apache2 is already running" and the second time I get "apache2 not running (no pid file)" but it starts up and all is well again. If I restart immediately after restarting, there is no such problem - this only happens after apache has already been running for a while.
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2454
Location: Silver City, NM

PostPosted: Sun Dec 02, 2012 7:49 pm    Post subject: Reply with quote

I don't have Apache installed here but you just need to get the pid of the parent httpd process. The Apache page I linked to suggests using `cat /usr/local/apache2/logs/httpd.pid` to get the pid in which case the command would be:
Code:
kill -HUP `cat /usr/local/apache2/logs/httpd.pid`

I'd bet the pid file on your system lives elsewhere.
Back to top
View user's profile Send private message
cach0rr0
Moderator
Moderator


Joined: 13 Nov 2008
Posts: 4121
Location: Houston, Republic of Texas

PostPosted: Mon Dec 03, 2012 4:21 am    Post subject: Reply with quote

BitJam wrote:

I'd bet the pid file on your system lives elsewhere.


easy enough to figure out

Code:

pidof apache2


There should already be a

Code:

  /etc/init.d/apache2 reload > /dev/null 2>&1 || true


in /etc/logrotate.d/apache2

the reload() function does an apachectl -k restart already

IOW, this *should* already be functional/working. Why it isn't, I'm not sure.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Mon Dec 03, 2012 2:23 pm    Post subject: Reply with quote

cach0rr0 wrote:

There should already be a

Code:

  /etc/init.d/apache2 reload > /dev/null 2>&1 || true


in /etc/logrotate.d/apache2

the reload() function does an apachectl -k restart already

IOW, this *should* already be functional/working. Why it isn't, I'm not sure.


There wasn't, and I had to create /etc/logrotate.d/apache2 manually.

That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?

While I'm at it, I'm wondering if I can make my logrotate config for apache more efficient. It goes on like this:

Code:

/var/log/apache2/access_log {
    compress
    size 1M
    rotate 2
    missingok
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
        /etc/init.d/apache2 reload > /dev/null 2>&1 || true
    endscript
}

/var/log/apache2/access_log_abusedmen {
    compress
    size 2M
    rotate 5
    missingok
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
        /etc/init.d/apache2 reload > /dev/null 2>&1 || true
    endscript
}

/var/log/apache2/access_log_audiodef {
    compress
    size 1M
    rotate 2
    missingok
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
        /etc/init.d/apache2 reload > /dev/null 2>&1 || true
    endscript
}


There are several more similar entries. Basically, the only thing I want done differently for each vhost is the number of rotates.
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2454
Location: Silver City, NM

PostPosted: Mon Dec 03, 2012 5:53 pm    Post subject: Reply with quote

audiodef wrote:
That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?
The "reload" is a very good way to do it. It does a graceful restart which is explained here:
https://httpd.apache.org/docs/2.2/stopping.html#graceful
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 5256

PostPosted: Tue Dec 04, 2012 3:36 pm    Post subject: Reply with quote

BitJam wrote:
audiodef wrote:
That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?
The "reload" is a very good way to do it. It does a graceful restart which is explained here:
https://httpd.apache.org/docs/2.2/stopping.html#graceful


Thanks. :) That bypasses whatever is causing apache to stutter when I try a hard restart after it's been running for a while.
_________________
Gentoo Studio: http://gentoostudio.org
Facebook: http://www.facebook.com/gentoostudio
G+: https://plus.google.com/113947758237122861689/posts
Pappy's Kernel Seeds: http://kernel-seeds.org
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum