Disappearing disk space?

Message

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 6:48 am

I ran out of space on one of my drives, so I removed some files. However, that didn't seem to free up any space:

After deleting 12 gigs of data:

Code: Select all

$df -h --sync
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdd1             230G  217G  662M 100% /var/ftp

Any ideas?

frostschutz · Post by **frostschutz** » Sat Apr 21, 2007 6:56 am

Maybe you deleted sparse files which look like they take 12G space, but actually don't? Maybe the files were still in use somehow, so they get deleted only after they are not in use anymore? Filesystems may also have a root reserve, that can only be filled by root, as to prevent the system becoming unusable because some user took all available diskspace. Also check your filesystem for corruption of any kind (umount, fsck in read-only mode).

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 2:20 pm

Frostschutz,

Thanks for your feedback, that's really helpful. However I don't think any of those suggestions are the solution

The drive is a secondary device, so there's no system files (or root reserve) on it.

The files deleted were media files (movies, emails, etc). There's no way I only deleted 662MB of data, but that's all that is reported as free. The emails themselves were 5 GB. So, I don't think sparse files are the issue.

I've tried a ext3.fsck -f to no avail. I'll try again with the -n flag, which according to the man page opens the filesystem in read only mode. I'll let you know if that does it.

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 2:50 pm

fsck.ext3 -fn didn't work. Neither did a reboot

Although the amount used did change (see below).

However Used + Avail still doesn't equal size.

Any other ideas? I'm stumped.

Code: Select all

$df -h --sync
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdd1             230G  226G     0 100% /var/ftp

frostschutz · Post by **frostschutz** » Sat Apr 21, 2007 3:01 pm

Well, you could try and see what is supposed to be taking all the space; either with 'du' as a command line tool, or 'xdiskusage' for a graphical representation of occupied space of your directory tree. Maybe it's really full because some process or other writes stuff on it?

The case where the file system is displayed as full when it really isn't and the file system is not corrupt in some way is unknown to me. So either it's really full or something's really wrong with your fs.

I'm not sure wether Size and Used actually have to be the same number; every file system has some overhead. If this overhead is included in the 'Used' statistics of df, it should be the same, otherwise it will be lower; depending on size/type of files and overhead for that specific filesystem (may be considerably huge with many small files) the difference may be normal.

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 3:22 pm

I just don't get it though.

I can unmount the drive, which would indicate there aren't any processes writing to it.

When I delete more files on the drive, the space doesn't appear as available.

Before deleting 9GB of data:

Code: Select all

$df -h --sync
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdd1             230G  226G     0 100% /var/ftp

After deleting 9GB of data:

Code: Select all

$ df -h --sync
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdd1             230G  217G  767M 100% /var/ftp

The drive is STILL FULL

Ugh, I'm going crazy.

sonicbhoc · Post by **sonicbhoc** » Sat Apr 21, 2007 4:35 pm

emerge filelight. That will show you where the problems lay!

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 5:20 pm

It looks like firelight isn't in portage.

Regardless, according to the README it requires kdelibs. To install that, I would need to emerge 20 dependences and resolve 2 blocks. Not ideal

Any other ideas? How safe would it be for me to emerge kdelibs?

Code: Select all

$ emerge -pv kdelibs

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[blocks B     ] kde-base/kde-env (is blocking kde-base/kdelibs-3.5.5-r9)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-misc/util-macros-1.1.0)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/kbproto-1.0.3)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xextproto-7.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86bigfontproto-1.1.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/inputproto-1.3.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libX11-1.0.3)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xproto-7.0.7)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXau-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/bigreqsproto-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXdmcp-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xcmiscproto-1.1.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/xtrans-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libICE-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libSM-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXt-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86vidmodeproto-2.2.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXext-1.0.1)
[ebuild     U ] sys-apps/man-1.6d [1.6-r1] +nls 263 kB 
[ebuild  N    ] x11-misc/util-macros-1.1.0  -debug 42 kB 
[ebuild     U ] sys-devel/binutils-2.16.1-r3 [2.16.1] -multislot -multitarget +nls -test -vanilla 12,367 kB 
[ebuild  N    ] x11-proto/kbproto-1.0.3  -debug 56 kB 
[ebuild  N    ] x11-proto/xextproto-7.0.2  -debug 66 kB 
[ebuild  N    ] x11-proto/xf86bigfontproto-1.1.2  -debug 36 kB 
[ebuild  N    ] x11-proto/inputproto-1.3.2  -debug 45 kB 
[ebuild  N    ] x11-proto/xproto-7.0.7  -debug 130 kB 
[ebuild  N    ] x11-libs/libXau-1.0.2  -debug 214 kB 
[ebuild  N    ] x11-proto/bigreqsproto-1.0.2  -debug 35 kB 
[ebuild  N    ] x11-libs/libXdmcp-1.0.1  -debug 226 kB 
[ebuild  N    ] x11-proto/xcmiscproto-1.1.2  -debug 35 kB 
[ebuild  N    ] x11-libs/xtrans-1.0.1  -debug 89 kB 
[ebuild  N    ] x11-libs/libX11-1.0.3  -debug +ipv6 1,415 kB 
[ebuild  N    ] x11-libs/libICE-1.0.1  -debug +ipv6 248 kB 
[ebuild  N    ] x11-libs/libSM-1.0.1  -debug +ipv6 234 kB 
[ebuild  N    ] x11-libs/libXt-1.0.2  -debug 505 kB 
[ebuild  N    ] x11-proto/xf86vidmodeproto-2.2.2  -debug 38 kB 
[ebuild  NS   ] kde-base/arts-3.5.5  +alsa -artswrappersuid -debug (-elibc_FreeBSD) -esd -jack -kdeenablefinal +mp3 -nas +vorbis -xinerama 944 kB 
[ebuild  N    ] app-dicts/aspell-en-0.51.1  168 kB 
[ebuild  N    ] net-dns/libidn-0.6.5-r1  -doc (-elibc_FreeBSD) -emacs -java -mono +nls 2,078 kB 
[ebuild     U ] dev-libs/libxml2-2.6.27 [2.6.23] -debug -doc +ipv6 +python +readline -test 4,483 kB 
[ebuild  N    ] dev-libs/libgpg-error-1.0-r1  +nls 316 kB 
[ebuild  N    ] dev-libs/libgcrypt-1.2.2-r1  +nls 939 kB 
[ebuild  N    ] dev-libs/libxslt-1.1.17  +crypt -debug +python 1,865 kB 
[ebuild  N    ] media-libs/libart_lgpl-2.3.17  -debug 282 kB 
[ebuild  N    ] x11-libs/libXext-1.0.1  -debug 254 kB 
[ebuild  N    ] media-fonts/gnu-gs-fonts-std-8.11  3,664 kB 
[ebuild  N    ] app-text/ghostscript-gpl-8.54  +X -cjk +cups -djvu -emacs +gtk -jpeg2k 12,082 kB 
[ebuild  N    ] virtual/ghostscript-0  0 kB 
[ebuild  N    ] kde-base/kdelibs-3.5.5-r9  -acl +alsa +arts -avahi +cups -debug -doc (-elibc_FreeBSD) -fam -jpeg2k -kdeenablefinal -kdehiddenvisibility -kerberos +kernel_linux -legacyssl -linguas_he -lua -openexr +spell +ssl +tiff -utempter -xinerama -zeroconf 15,164 kB

terminal9 · Post by **terminal9** » Sat Apr 21, 2007 5:31 pm

lsof is your friend.

sonicbhoc · Post by **sonicbhoc** » Sat Apr 21, 2007 6:26 pm

FILELIGHT not FIRELIGHT. close though.

emerge filelight

odioworks_com · Post by **odioworks_com** » Sat Apr 21, 2007 10:15 pm

FILELIGHT not FIRELIGHT. close though.

Oops

It looks life filelight isn't much better, though:

Code: Select all

$ emerge -pv filelight

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-misc/util-macros-1.1.0)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/kbproto-1.0.3)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xextproto-7.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86bigfontproto-1.1.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/inputproto-1.3.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libX11-1.0.3)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xproto-7.0.7)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXau-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/bigreqsproto-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXdmcp-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xcmiscproto-1.1.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/xtrans-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libICE-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libSM-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXt-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86vidmodeproto-2.2.2)
[blocks B     ] kde-base/kde-env (is blocking kde-base/kdelibs-3.5.5-r9)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXext-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86miscproto-0.9.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXxf86misc-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libdmx-1.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/dmxproto-2.2.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/printproto-1.0.3)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXp-1.0.0)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/renderproto-0.9.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXrender-0.9.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xf86dgaproto-2.0.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-apps/xdpyinfo-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/xineramaproto-1.1.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXinerama-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXxf86vm-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXxf86dga-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXi-1.0.1)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-proto/recordproto-1.13.2)
[blocks B     ] <=x11-base/xorg-x11-6.9 (is blocking x11-libs/libXtst-1.0.1)
[ebuild     U ] sys-apps/man-1.6d [1.6-r1] +nls 263 kB 
[ebuild  N    ] x11-misc/util-macros-1.1.0  -debug 42 kB 
[ebuild     U ] sys-devel/binutils-2.16.1-r3 [2.16.1] -multislot -multitarget +nls -test -vanilla 12,367 kB 
[ebuild  N    ] x11-proto/kbproto-1.0.3  -debug 56 kB 
[ebuild  N    ] x11-proto/xextproto-7.0.2  -debug 66 kB 
[ebuild  N    ] x11-proto/xf86bigfontproto-1.1.2  -debug 36 kB 
[ebuild  N    ] x11-proto/inputproto-1.3.2  -debug 45 kB 
[ebuild  N    ] x11-proto/xproto-7.0.7  -debug 130 kB 
[ebuild  N    ] x11-libs/libXau-1.0.2  -debug 214 kB 
[ebuild  N    ] x11-proto/bigreqsproto-1.0.2  -debug 35 kB 
[ebuild  N    ] x11-libs/libXdmcp-1.0.1  -debug 226 kB 
[ebuild  N    ] x11-proto/xcmiscproto-1.1.2  -debug 35 kB 
[ebuild  N    ] x11-libs/xtrans-1.0.1  -debug 89 kB 
[ebuild  N    ] x11-libs/libX11-1.0.3  -debug +ipv6 1,415 kB 
[ebuild  N    ] x11-libs/libICE-1.0.1  -debug +ipv6 248 kB 
[ebuild  N    ] x11-libs/libSM-1.0.1  -debug +ipv6 234 kB 
[ebuild  N    ] x11-libs/libXt-1.0.2  -debug 505 kB 
[ebuild  N    ] x11-proto/xf86vidmodeproto-2.2.2  -debug 38 kB 
[ebuild  NS   ] kde-base/arts-3.5.5  +alsa -artswrappersuid -debug (-elibc_FreeBSD) -esd -jack -kdeenablefinal +mp3 -nas +vorbis -xinerama 944 kB 
[ebuild  N    ] app-dicts/aspell-en-0.51.1  168 kB 
[ebuild  N    ] net-dns/libidn-0.6.5-r1  -doc (-elibc_FreeBSD) -emacs -java -mono +nls 2,078 kB 
[ebuild     U ] dev-libs/libxml2-2.6.27 [2.6.23] -debug -doc +ipv6 +python +readline -test 4,483 kB 
[ebuild  N    ] dev-libs/libgpg-error-1.0-r1  +nls 316 kB 
[ebuild  N    ] dev-libs/libgcrypt-1.2.2-r1  +nls 939 kB 
[ebuild  N    ] dev-libs/libxslt-1.1.17  +crypt -debug +python 1,865 kB 
[ebuild  N    ] media-libs/libart_lgpl-2.3.17  -debug 282 kB 
[ebuild  N    ] x11-libs/libXext-1.0.1  -debug 254 kB 
[ebuild  N    ] media-fonts/gnu-gs-fonts-std-8.11  3,664 kB 
[ebuild  N    ] app-text/ghostscript-gpl-8.54  +X -cjk +cups -djvu -emacs +gtk -jpeg2k 12,082 kB 
[ebuild  N    ] virtual/ghostscript-0  0 kB 
[ebuild  N    ] kde-base/kdelibs-3.5.5-r9  -acl +alsa +arts -avahi +cups -debug -doc (-elibc_FreeBSD) -fam -jpeg2k -kdeenablefinal -kdehiddenvisibility -kerberos +kernel_linux -legacyssl -linguas_he -lua -openexr +spell +ssl +tiff -utempter -xinerama -zeroconf 15,164 kB 
[ebuild  N    ] x11-proto/xf86miscproto-0.9.2  -debug 37 kB 
[ebuild  N    ] x11-libs/libXxf86misc-1.0.1  -debug 217 kB 
[ebuild  N    ] x11-proto/dmxproto-2.2.2  -debug 38 kB 
[ebuild  N    ] x11-libs/libdmx-1.0.2  -debug 225 kB 
[ebuild  N    ] x11-proto/printproto-1.0.3  -debug 42 kB 
[ebuild  N    ] x11-libs/libXp-1.0.0  -debug 238 kB 
[ebuild  N    ] x11-proto/renderproto-0.9.2  -debug 39 kB 
[ebuild  N    ] x11-libs/libXrender-0.9.1  -debug 234 kB 
[ebuild  N    ] x11-proto/xf86dgaproto-2.0.2  -debug 39 kB 
[ebuild  N    ] x11-proto/xineramaproto-1.1.2  -debug 37 kB 
[ebuild  N    ] x11-libs/libXinerama-1.0.1  -debug 201 kB 
[ebuild  N    ] x11-libs/libXxf86vm-1.0.1  -debug 222 kB 
[ebuild  N    ] x11-libs/libXxf86dga-1.0.1  -debug 226 kB 
[ebuild  N    ] x11-libs/libXi-1.0.1  -debug 240 kB 
[ebuild  N    ] x11-proto/recordproto-1.13.2  -debug 38 kB 
[ebuild  N    ] x11-libs/libXtst-1.0.1  -debug 207 kB 
[ebuild  N    ] x11-apps/xdpyinfo-1.0.1  -debug 85 kB 
[ebuild  N    ] kde-misc/filelight-1.0_rc2  +arts -debug (-elibc_FreeBSD) -xinerama 518 kB

I'll try lsof.

sonicbhoc · Post by **sonicbhoc** » Sat Apr 21, 2007 11:55 pm

You're still running an old version of X? Why not upgrade to modular?

odioworks_com · Post by **odioworks_com** » Sun Apr 22, 2007 2:12 am

/dev/hdd1 is mounted on /var/ftp.

$lsof | grep "/var/ftp" returns nothing.

I haven't bothered with X because it's a server, and I only ssh into it. I didn't want to break anything by trying to upgrade it.

sonicbhoc · Post by **sonicbhoc** » Sun Apr 22, 2007 3:44 am

try unmounting the drive and doing a file system check on it. If that doesn't work... well, I'm out of ideas.

odioworks_com · Post by **odioworks_com** » Sun Apr 22, 2007 4:37 am

Yah, tried that several times, fsck.ext3 was a no go:

fsck.ext3 -fn didn't work. Neither did a reboot Although the amount used did change (see below).

However Used + Avail still doesn't equal size.

Any other ideas? I'm stumped.
Code: Select all
Code:
$df -h --sync
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdd1             230G  226G     0 100% /var/ftp 

The problem persists: If I delete 10 gigs of data using "rm -rf" - the space doesn't become "Available." I can't figure this out. I guess I might have to try wiping the drive.

terminal9 · Post by **terminal9** » Sun Apr 22, 2007 1:33 pm

I'll propose a number of dumb ideas, sometimes simplicity resolves:
1. Make sure the folders you claim to have deleted gigabytes of data from are REALLY on hdd1 and not some symlinks to other drive(s).
2. I assume you have an FTP server running on this machine (hint: /var/ftp) ? Could somebody be flooding incoming folder in /var/ftp? Try stopping ftp server. Actually, try to stop all other servers, if any, and observe the usage for sometime.
3. And -deviating from dumb mode- use lsof in repeat mode, e.g.

Code: Select all

lsof -r3 +D /var/ftp

That will print open files on /var/ftp and all its subdirs every 3 seconds, until you interrupt with Ctrl-C.

odioworks_com · Post by **odioworks_com** » Sun Apr 22, 2007 8:05 pm

I'll propose a number of dumb ideas, sometimes simplicity resolves:
1. Make sure the folders you claim to have deleted gigabytes of data from are REALLY on hdd1 and not some symlinks to other drive(s).
2. I assume you have an FTP server running on this machine (hint: /var/ftp) ? Could somebody be flooding incoming folder in /var/ftp? Try stopping ftp server. Actually, try to stop all other servers, if any, and observe the usage for sometime.
3. And -deviating from dumb mode- use lsof in repeat mode, e.g.
Code:
lsof -r3 +D /var/ftp

That will print open files on /var/ftp and all its subdirs every 3 seconds, until you interrupt with Ctrl-C.

No symlinks, this is a 300GB drive, the stuff couldn't fit anywhere else. The only other drive on the system is 60GB - and it's is only for the OS.

FTP isn't running, and I'm the only one who has file-level access to the machine.

lsof -r3 +D /var/ftp returns nothing

Any other ideas?

alex6z · Post by **alex6z** » Sun Apr 22, 2007 8:42 pm

It looks like someone filled the drive almost all the way up, and then changed the root reserve to a large %age. The drive can be 60% full but show 0 free space in df if the root reserve is set to 40%.

Also check that you are not out of inodes.

Try du -c /foo to see if things add up. Try using find to find large files too.

terminal9 · Post by **terminal9** » Sun Apr 22, 2007 8:44 pm

Check for bad blocks:

Code: Select all

dumpe2fs -b /dev/hdd1

If you get a badblock report, use badblocks command to get detailed analysis.
If no bad block, check the output of

Code: Select all

dumpe2fs -h /dev/hdd1

and observe the size of journal (or post it here, we can have a look together)