View previous topic :: View next topic |
Author |
Message |
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
|
Back to top |
|
|
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Wed Nov 21, 2012 11:14 am Post subject: |
|
|
When you rotate your log files, do you tell Apache to close the old one and start writing to the new one? |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
|
Back to top |
|
|
faemin n00b
Joined: 16 Oct 2012 Posts: 22
|
Posted: Sat Dec 01, 2012 7:12 am Post subject: |
|
|
...
Last edited by faemin on Sun Dec 02, 2012 9:38 pm; edited 1 time in total |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
Posted: Sat Dec 01, 2012 6:49 pm Post subject: |
|
|
I found out about htcacheclean and now have this running:
Code: |
htcacheclean -d 1440 -p /var/cache/apache2/ -l 5M
|
But I still see df usage creeping up, and it's due to Apache somehow.
I didn't have lsof, but I emerged it. I see some output, but am not sure what to do with it.
/etc/apache2/httpd.conf:
Code: |
ServerRoot "/usr/lib64/apache2"
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfDefine CACHE>
LoadModule cache_module modules/mod_cache.so
</IfDefine>
LoadModule cgi_module modules/mod_cgi.so
LoadModule cgid_module modules/mod_cgid.so
<IfDefine DAV>
LoadModule dav_module modules/mod_dav.so
</IfDefine>
<IfDefine DAV>
LoadModule dav_fs_module modules/mod_dav_fs.so
</IfDefine>
<IfDefine DAV>
LoadModule dav_lock_module modules/mod_dav_lock.so
</IfDefine>
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
<IfDefine CACHE>
LoadModule disk_cache_module modules/mod_disk_cache.so
</IfDefine>
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule ext_filter_module modules/mod_ext_filter.so
<IfDefine CACHE>
LoadModule file_cache_module modules/mod_file_cache.so
</IfDefine>
LoadModule filter_module modules/mod_filter.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
<IfDefine INFO>
LoadModule info_module modules/mod_info.so
</IfDefine>
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
<IfDefine CACHE>
LoadModule mem_cache_module modules/mod_mem_cache.so
</IfDefine>
LoadModule mime_module modules/mod_mime.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule speling_module modules/mod_speling.so
<IfDefine SSL>
LoadModule ssl_module modules/mod_ssl.so
</IfDefine>
<IfDefine STATUS>
LoadModule status_module modules/mod_status.so
</IfDefine>
LoadModule unique_id_module modules/mod_unique_id.so
<IfDefine USERDIR>
LoadModule userdir_module modules/mod_userdir.so
</IfDefine>
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
User apache
Group apache
Include /etc/apache2/modules.d/*.conf
Include /etc/apache2/vhosts.d/*.conf
CustomLog /var/log/apache2/access_log combined
# vim: ts=4 filetype=apache
ServerName (ip_address)
|
/etc/conf.d/apache2:
Code: |
# /etc/conf.d/apache2: config file for /etc/init.d/apache2
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -D AUTH_IMAP -D PERL"
|
(I deleted all the commented out lines that are there when installed.)
I have several vhosts, and they all more or less look like:
Code: |
<VirtualHost *:80>
ServerAdmin (email_address)
DocumentRoot "/path/to/location"
ServerName (domain.com)
ServerAlias (www.domain.com)
<Directory "/path/to/location">
Options Indexes FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
CustomLog /var/log/apache2/access_log_domain combined
|
Does anything look like it would cause apache to fill up disk space somewhere? _________________ decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN |
|
Back to top |
|
|
faemin n00b
Joined: 16 Oct 2012 Posts: 22
|
Posted: Sun Dec 02, 2012 1:40 am Post subject: |
|
|
...
Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Sun Dec 02, 2012 2:01 am Post subject: |
|
|
As Carnildo mentioned, as long as a file is open in a program it is not actually deleted even though you've removed all directories entries for that file with the rm command.
The simplest thing is to simply restart Apache after rotating the log files. This is generally done by sending it a HUP signal. See here:
https://httpd.apache.org/docs/2.2/stopping.html Quote: | Signal: HUP
apachectl -k restart
Sending the HUP or restart signal to the parent causes it to kill off its children like in TERM, but the parent doesn't exit. It re-reads its configuration files, and re-opens any log files. Then it spawns a new set of children and continues serving hits. |
|
|
Back to top |
|
|
faemin n00b
Joined: 16 Oct 2012 Posts: 22
|
Posted: Sun Dec 02, 2012 4:57 am Post subject: |
|
|
...
Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Sun Dec 02, 2012 5:17 am Post subject: |
|
|
faemin wrote: | He posted that he has already restarted apache several times.... | The OP actually said: Quote: | Restarting apache brings my df usage down by 10 percent or more, even though /var/log/apache2 has nothing over 600k in it due to my syslog-ng/logrotate config. |
This is exactly what I was talking about. There was nothing really wrong, The OP just didn't understand what was happening. Once you realize inodes for open files don't magically disappear when the files are deleted then it all makes sense. It's like hardlinks. You need to remove every hardlink to a file before the actual file gets deleted and frees up its disk space. I believe there is an accompanying problem of losing log information between the time the logs are rotated and when Apache is restarted.
You just need to restart Apache after you rotate the logs. This is pretty standard stuff. |
|
Back to top |
|
|
faemin n00b
Joined: 16 Oct 2012 Posts: 22
|
Posted: Sun Dec 02, 2012 6:26 am Post subject: |
|
|
...
Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Sun Dec 02, 2012 7:28 am Post subject: |
|
|
I didn't say it was a hard link. I said it was like a hard link. I was trying to explain how it works. If you have N hard links to one file then all N need to be removed before you can free up space from the file (barring truncation). Likewise when N programs have a particular file open then all N programs need to close the file before you can free up its disk space (again, barring truncation).
It is true that copy-and-truncate is an alternative approach to log rotation but is is usually only used when the program that controls the logs cannot be told to re-open its log files. It is less safe and less efficient than the standard approach. As the logrotate man page explains: Code: | copytruncate
Truncate the original log file in place after creating a copy,
instead of moving the old log file and optionally creating a new
one. It can be used when some program cannot be told to close its
logfile and thus might continue writing (appending) to the
previous log file forever. Note that there is a very small time
slice between copying the file and truncating it, so some
logging data might be lost. When this option is used, the cre-
ate option will have no effect, as the old log file stays in
place. | Apache is not broken in this way. It was designed to respond to the HUP signal by re-reading its config files and re-opening its log files which was explained on the Apache documentation page I linked to. Furthermore, I didn't say restarting Apache was the only way to deal with the problem, I said it was the simplest.
I didn't post here to get in a series of fights with you. I simply wanted to help audiodef understand and fix his problem. |
|
Back to top |
|
|
faemin n00b
Joined: 16 Oct 2012 Posts: 22
|
Posted: Sun Dec 02, 2012 8:38 am Post subject: |
|
|
...
Last edited by faemin on Sun Dec 02, 2012 9:50 pm; edited 1 time in total |
|
Back to top |
|
|
cach0rr0 Bodhisattva
Joined: 13 Nov 2008 Posts: 4123 Location: Houston, Republic of Texas
|
Posted: Sun Dec 02, 2012 10:56 am Post subject: |
|
|
This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying.
Address the original poster's questions or shut up.
audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity. _________________ Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
Posted: Sun Dec 02, 2012 3:38 pm Post subject: |
|
|
cach0rr0 wrote: | This thread will get back on topic ASAP, or posts will start disappearing. The tangents are annoying. Really, really, really annoying.
Address the original poster's questions or shut up.
audiodef - sorry I have nothing useful to add, just popping in to address this current exercise in stupidity. |
No worries. I appreciate that people were trying to help.
I don't mind having Apache restart after log rotation. Could I simply put apache2ctl restart in the postrotate section of /etc/logrotate.d/apache2?
Two things:
1. I noticed that -k is not a valid option for apache2ctl.
2. Whether I use apache2ctl or /etc/init.d/apache2 restart, I have to do it twice because the first time, I get "start-stop-daemon: /usr/sbin/apache2 is already running" and the second time I get "apache2 not running (no pid file)" but it starts up and all is well again. If I restart immediately after restarting, there is no such problem - this only happens after apache has already been running for a while. _________________ decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Sun Dec 02, 2012 7:49 pm Post subject: |
|
|
I don't have Apache installed here but you just need to get the pid of the parent httpd process. The Apache page I linked to suggests using `cat /usr/local/apache2/logs/httpd.pid` to get the pid in which case the command would be: Code: | kill -HUP `cat /usr/local/apache2/logs/httpd.pid` |
I'd bet the pid file on your system lives elsewhere. |
|
Back to top |
|
|
cach0rr0 Bodhisattva
Joined: 13 Nov 2008 Posts: 4123 Location: Houston, Republic of Texas
|
Posted: Mon Dec 03, 2012 4:21 am Post subject: |
|
|
BitJam wrote: |
I'd bet the pid file on your system lives elsewhere. |
easy enough to figure out
There should already be a
Code: |
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
|
in /etc/logrotate.d/apache2
the reload() function does an apachectl -k restart already
IOW, this *should* already be functional/working. Why it isn't, I'm not sure. _________________ Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
Posted: Mon Dec 03, 2012 2:23 pm Post subject: |
|
|
cach0rr0 wrote: |
There should already be a
Code: |
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
|
in /etc/logrotate.d/apache2
the reload() function does an apachectl -k restart already
IOW, this *should* already be functional/working. Why it isn't, I'm not sure. |
There wasn't, and I had to create /etc/logrotate.d/apache2 manually.
That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart?
While I'm at it, I'm wondering if I can make my logrotate config for apache more efficient. It goes on like this:
Code: |
/var/log/apache2/access_log {
compress
size 1M
rotate 2
missingok
sharedscripts
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
endscript
}
/var/log/apache2/access_log_abusedmen {
compress
size 2M
rotate 5
missingok
sharedscripts
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
endscript
}
/var/log/apache2/access_log_audiodef {
compress
size 1M
rotate 2
missingok
sharedscripts
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
/etc/init.d/apache2 reload > /dev/null 2>&1 || true
endscript
}
|
There are several more similar entries. Basically, the only thing I want done differently for each vhost is the number of rotates. _________________ decibel Linux: https://decibellinux.org
Github: https://github.com/Gentoo-Music-and-Audio-Technology
Facebook: https://www.facebook.com/decibellinux
Discord: https://discord.gg/73XV24dNPN |
|
Back to top |
|
|
BitJam Advocate
Joined: 12 Aug 2003 Posts: 2508 Location: Silver City, NM
|
Posted: Mon Dec 03, 2012 5:53 pm Post subject: |
|
|
audiodef wrote: | That brings up another question, though: is "reload" enough for apache to release its hold on files and clear things up, versus an outright restart? | The "reload" is a very good way to do it. It does a graceful restart which is explained here:
https://httpd.apache.org/docs/2.2/stopping.html#graceful |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
|
Back to top |
|
|
|