| View previous topic :: View next topic |
| Author |
Message |
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
 Posted: Mon Feb 08, 2010 6:41 pm Post subject: Var partition getting full [CLOSED] |
 |
|
On one of my machines, my /var dir is at about 86%. Removing stuff in quarantine and tmp and some old logs. It brought it down to 86%.
I removed stuff in /var/tmp/portage as well. As well as some old modules dir for old kernels in lib/modules.
du -s * gives me
| Code: |
74780 amavis
105944 cache
65288 db
12 dcc
4 empty
34210688 lib
8 lock
12848 log
16 lost+found
0 mail
76 run
992 spool
4 state
32 tmp
4691560 www
|
Note lib has /var/lib/mysql which is a separate partition, and /var/www is a separate partition as well.
Does all of this sound about right? Logrotate keeps about 4 weeks of back logs compressed.
Last edited by gohmdoree on Sun Feb 14, 2010 6:56 am; edited 1 time in total |
|
| Back to top |
|
 |
timeBandit
Administrator
Joined: 31 Dec 2004
Posts: 2667
Location: here, there or in transit
|
| Posted: Mon Feb 08, 2010 6:53 pm Post subject: |
|
|
Difficult to say, absent any information about the /var filesystem itself (FS type, partition size, etc.).
Unmount /var/lib/mysql and /var/www, then re-run du -sh /var/*--it's possible there are files hiding under the mount points. If that is not the case, please post the output of df -h | nl and nl /etc/fstab, in addition to the output from the re-run of du.
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others. |
|
| Back to top |
|
|
MotivatedTea
Apprentice
Joined: 06 Nov 2006
Posts: 267
Location: Toronto, Canada
|
| Posted: Mon Feb 08, 2010 7:05 pm Post subject: |
|
|
The "-x" parameter to "du" will ignore subdirectories mounted on different filesystems.
Well, your biggest remaining directories are "cache", "amavis", and "db". I don't use amavis, but it seems to be some kind of anti-spam app? So I'd guess you've got databases or training data in there, or something like that, which seems reasonable. I'm not a portage developer, but it looks like /var/cache/edb has portage dependency information, and my directory is about the same size as yours. I think /var/db/pkg is where portage stores information about what you've installed, so that will grow as you install apps (and presumably shrink when you uninstall them).
So, at first glance I don't see anything unusual. You know what you're using your machine for best, though. If you think something is taking up too much space, take a look inside the directories. There are a few programs out there that will give you a graphical view of your directories. I use "filelight" (KDE-based); I believe a GNOME equivalent is "Baobab", but I've never used that. I used to like "kdirstat", but it got removed with KDE3.
By the way, a really handy script for cleaning out /var/log/portage is logclean, which you can get here. It has options to delete logs for packages you no longer have installed, and compress logs for packages older than some cutoff. |
|
| Back to top |
|
|
timeBandit
Administrator
Joined: 31 Dec 2004
Posts: 2667
Location: here, there or in transit
|
| Posted: Mon Feb 08, 2010 7:16 pm Post subject: |
|
|
| MotivatedTea wrote: | | The "-x" parameter to "du" will ignore subdirectories mounted on different filesystems. |
Thanks, I always forget about those same-filesystem-only options. 
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Mon Feb 08, 2010 9:59 pm Post subject: |
|
|
thanks for the tip about "-x"
The filesystem is ext3 and /var is about 5GB. /var/tmp is 5GB. /var/lib/mysql is about 40GB and /var/www is about 150GB I think.
I actually suspected there might have been something underlying the mounted partitions, but there was nothing there after I had unmounted.
df -h | nl's output
| Code: |
1 Filesystem Size Used Avail Use% Mounted on
2 /dev/sda3 966M 50M 867M 6% /
3 udev 10M 160K 9.9M 2% /dev
4 /dev/sda5 4.7G 138M 4.4G 4% /tmp
5 /dev/sda6 7.1G 4.7G 2.0G 71% /usr
6 /dev/sda7 2.4G 68M 2.2G 3% /opt
7 /dev/sda8 19G 383M 18G 3% /home
8 /dev/sda9 4.7G 3.9G 610M 87% /var
9 /dev/sda10 4.7G 138M 4.4G 4% /var/tmp
10 shm 998M 0 998M 0% /dev/shm
11 /dev/sda11 38G 33G 3.0G 92% /var/lib/mysql
12 /dev/sda12 147G 4.7G 134G 4% /var/www
|
nl /etc/fstab, note, removed all commented lines
| Code: |
15 /dev/sda1 /boot ext2 noauto,noatime 1 2
16 /dev/sda2 none swap sw 0 0
17 /dev/sda3 / ext3 noatime 0 1
18 /dev/sda5 /tmp ext3 nodev,nosuid,noexec 1 2
19 /dev/sda6 /usr ext3 noatime,nodev 1 2
20 /dev/sda7 /opt ext3 noatime,nodev 1 2
21 /dev/sda8 /home ext3 noatime,nodev,nosuid 1 2
22 /dev/sda9 /var ext3 noatime,nodev 1 2
23 /dev/sda10 /var/tmp ext3 nodev,nosuid 1 2
24 /dev/sda11 /var/lib/mysql ext3 noatime,nodev 1 2
25 /dev/sda12 /var/www/ ext3 noatime,nodev 1 2
29 proc /proc proc nodev,nosuid,noexec 0 0
34 shm /dev/shm tmpfs nodev,nosuid,noexec 0 0
|
Did you want to output of du -x *? It shows all individual files as well as directories. Lot of output. Looks like there is a lot in /var/lib an db/pkg. |
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Tue Feb 09, 2010 2:32 pm Post subject: |
|
|
You can locate the largest files in /var without descending into other filesystems: | Code: | | find /var -xdev -printf '%s %p\n' | sort -gr | head |
|
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Tue Feb 09, 2010 4:24 pm Post subject: |
|
|
| Code: |
find /var -xdev -printf '%s %p\n' | sort -gr | head
|
gave me
| Code: |
47079936 /var/lib/clamav/main.cld
42217472 /var/amavis/.spamassassin/bayes_toks
26838134 /var/amavis/amavis.log.gz
21253696 /var/lib/clamav/main.cvd
14575772 /var/lib/clamav/main.inc/main.ndb
9938432 /var/lib/clamav/daily.cld
8257536 /var/log/ulogd.syslogemu-20100207.gz
4960256 /var/amavis/.spamassassin/bayes_seen
4735470 /var/lib/clamav/main.inc/main.db
4335069 /var/lib/clamav/main.inc/main.mdb
|
|
|
| Back to top |
|
|
Mike Hunt
Watchman
Joined: 19 Jul 2009
Posts: 5287
|
| Posted: Tue Feb 09, 2010 4:47 pm Post subject: |
|
|
Looks like /var/lib/clamav might be a big consumer of disk space: | Code: | | find /var/lib/clamav -printf '%s %p\n' | awk '{ SUM += $1} END { print SUM/1024/1024 " MB"}' |
|
|
| Back to top |
|
|
timeBandit
Administrator
Joined: 31 Dec 2004
Posts: 2667
Location: here, there or in transit
|
| Posted: Tue Feb 09, 2010 4:58 pm Post subject: |
|
|
| gohmdoree wrote: | I actually suspected there might have been something underlying the mounted partitions, but there was nothing there after I had unmounted.
df -h | nl's output
| Code: | | 9 /dev/sda10 4.7G 138M 4.4G 4% /var/tmp |
|
Did you check under /var/tmp as well? I assume you did but ask because it wasn't mentioned earlier. This is a prime candidate for "mystery" allocation, as the wasted space would be in many small files and it's easy to overlook the mess from a failed merge. Otherwise, as noted clamav and amavis look like your big offenders.
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Tue Feb 09, 2010 5:52 pm Post subject: |
|
|
I just checked, but nothing was in the /var/tmp directory with the partition unmounted.
I tried gzipping the various clamav files, but clamav complains that the touched files are not valid files. Was looking for some documentation on this. Any pointers? |
|
| Back to top |
|
|
bmadonna1401
n00b
Joined: 31 Jul 2009
Posts: 10
Location: Washington, DC
|
| Posted: Thu Feb 11, 2010 9:46 pm Post subject: |
|
|
| My messages log file seems to be what has my var folder full by far. I'm moving it off to a safe place to archive and will create a new one. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Thu Feb 11, 2010 9:59 pm Post subject: |
|
|
| i use logrotate to rotate out and compress, keeping up to 4 weeks worth. maybe give that one a go. |
|
| Back to top |
|
|
MotivatedTea
Apprentice
Joined: 06 Nov 2006
Posts: 267
Location: Toronto, Canada
|
| Posted: Thu Feb 11, 2010 10:00 pm Post subject: |
|
|
| gohmdoree wrote: | | I tried gzipping the various clamav files, but clamav complains that the touched files are not valid files. Was looking for some documentation on this. Any pointers? |
You can't just start randomly compressing some program's data files and expect the program to be happy about it.
Your "df" output says you're using 3.9GB in your /var partition. That would be exclusive of anything you have mounted under it. My /var/lib/clamav directory is only 104MB. How big is yours? If it's 3.9GB, then you've got a problem with clamav. If it's only a couple hundred MB, then you're looking in the wrong place.
It's hard for us to tell you where your disk space has gone, since we don't have access to your machine. Have you tried a graphical program like filelight, as I suggested? It will show you visually exactly what directories are taking up most of your space. You'll want to configure it (Settings > Configure Filelight) not to cross filesystem boundaries. You're likely to find that much faster than looking at rows of "du" output at a command prompt. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Thu Feb 11, 2010 10:12 pm Post subject: |
|
|
| Its a remote system without any kind of X apps on it. I'll keep looking. Haven't had a chance to look at it much since. thanks for the suggestion MotivatedTea. |
|
| Back to top |
|
|
MotivatedTea
Apprentice
Joined: 06 Nov 2006
Posts: 267
Location: Toronto, Canada
|
| Posted: Thu Feb 11, 2010 10:39 pm Post subject: |
|
|
I found a script that does a similar thing but generates an HTML report as output, so it doesn't require X-Windows. You may want to give it a try:
http://www.studiosells.net/msells/duHTMLtree/
You pipe the output of du (and don't forget the -x parameter in your case) to a file, and then run it through the script to generate a report. The computer you generate the report on doesn't have to be the same one you run du on. I've used it before with mixed results: it worked for me when I ran it on directories with a few GB, but when I ran it on an entire 500GB partition, the output was obviously wrong... There may be a limit to the amount of data it can process. I don't know perl, so I didn't look into it. You could give it a try, though -- your partition isn't that big. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Thu Feb 11, 2010 10:44 pm Post subject: |
|
|
| Thanks MotivatedTea. That is pretty generous. I'll give it a look after work today. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 2:37 am Post subject: |
|
|
Tried the duhtmltree script. Ran it to look for files over 500M. It only showed my www which is mounted as a separate partition. I did run du with the -x option. Just in case I didn't look, I unmounted /var/www and looked and there was nothing there.
The disk now shows as 99% full.
du -xks /var/*
| Code: |
79700 /var/amavis
105964 /var/cache
65540 /var/db
12 /var/dcc
4 /var/empty
109888 /var/lib
8 /var/lock
21440 /var/log
16 /var/lost+found
0 /var/mail
80 /var/run
288 /var/spool
4 /var/state
676588 /var/tmp
4715492 /var/www
|
The numbers just don't add up. The /var is about 5GB. I'm assuming the /var/www contents are the main reason, but that doesn't make sense since it is a separate partition.
Any other thoughts? Clearly something is amiss. I didn't do something correctly or I'm overlooking something obvious. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 3:08 am Post subject: |
|
|
Looking at my initial post, du -s * included the contents of my lib/mysql and www in /var. with du -sx *, lib/mysql isn't included, but www still is.
I'm guessing this is my issue somehow, which still doesn't make sense to me. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 4:05 am Post subject: |
|
|
I unmounted /var/www, then ran the following
| Code: |
du -x * | awk '{SUM +1 $1} END {print SUM}'
|
Output was
I mounted /var/www, then reran and got
Not sure if my awk is correct, but was hoping to figure this out. [/code] |
|
| Back to top |
|
|
MotivatedTea
Apprentice
Joined: 06 Nov 2006
Posts: 267
Location: Toronto, Canada
|
| Posted: Sun Feb 14, 2010 4:20 am Post subject: |
|
|
You're not running "du" correctly. If you run "du -xks /var/*", or if you're inside /var and you run "du -x *", you will include the contents of /var/www and /var/tmp but not /var/lib/mysql, because "du -xks /var/*" expands to "du -skx /var/tmp /var/www" etc (including any other files/directories you have directly inside var). Remember, unlike DOS, bash expands wildcards before passing the arguments to the program. Any files or directories starting with a dot are going to be skipped too, because the shell won't expand those for "*". Instead, you just want to run "du -x /var". That should give you the output for /var, but exclude anything mounted under it. Pass the output of that command to duHTMLtree and see what it gives you. Also, since you don't know right now whether your space is being lost to a few huge files or a lot of tiny ones, you might not want to search just for >500MB files.
From your most recent du output, if you omit /var/tmp and /var/www, that only covers about 374MB of disk space, which is nowhere close to the 3.9GB that df says you're using. Do you have any hidden files or directories in /var? Do "ls -a /var" and see if any filename starts with a dot.
Another thing: On Linux, if you delete a file or directory that a program is using, the file will "disappear" from the directory listings, but the space won't be reclaimed until the program exits or you reboot. (Technically, you don't need to reboot, but doing so guarantees that all programs using the partition get killed and the partition gets unmounted and remounted.) If you haven't already done so, you may want to reboot. Maybe fsck /var as well, just to make sure. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 4:34 am Post subject: |
|
|
| Thanks for the thorough post MotivatedTea. I'll give those corrections a try and maybe even reboot as well. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 5:01 am Post subject: |
|
|
| duHTMLtree showed only about 370 MB. I will have to do a reboot and see if that will free up the questionable space. Rebuilding my toolchain now, so will do after and see where I get. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 6:44 am Post subject: |
|
|
| rebooted the machine and it never came up. well, its a remote dedicated machine. i'm told its sitting at the login prompt now. maybe more is going on than i can see. i'm guessing the partition is full or was unmountable. not sure. investigating now. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Sun Feb 14, 2010 6:56 am Post subject: |
|
|
The extra reboot request got it back up. /var is now at 12%. Thanks MotivatedTea for all of the advice and lessons, as well timeBandit and Mike Hunt. Learned a lot through this experience.
I guess I needed that reboot to actually "free up" space. |
|
| Back to top |
|
|
gohmdoree
Guru
Joined: 12 Oct 2004
Posts: 532
|
| Posted: Thu Apr 08, 2010 5:42 pm Post subject: |
|
|
question about this. my server has been up about 50 days or so. i noticed had some issues of some services running. i did a df, and my /var was at 100%, though running through du -s, etc didn't really show me anything outstanding.
i did a reboot as last time and now it is at 14%. is there something else to think about going on here? |
|
| Back to top |
|
|
|
Other Things Gentoo
