| View previous topic :: View next topic |
| Author |
Message |
Dr. Frankenbox
Apprentice
Joined: 16 Jul 2005
Posts: 170
Location: Iowa, USA
|
 Posted: Sun Apr 29, 2012 2:52 am Post subject: EXT4 Filesystem Full with "Ghost Data"? [SOLVED] |
 |
|
Not sure if this is the right forum for this, but. . .
My root filesystem (ext4) is nearly full, but I can't find the bulk of the data in any file. Compare the following df and du output:
| Code: | frankenbox / # df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/root 90G 83G 2.4G 98% /
frankenbox / # du -sh * --exclude=mnt
8.6M bin
47M boot
152K dev
4.4M etc
35G home
0 lib
5.3M lib32
158M lib64
16K lost+found
4.0K media
172M opt
du: cannot access `proc/17274/task/17274/fd/4': No such file or directory
du: cannot access `proc/17274/task/17274/fdinfo/4': No such file or directory
du: cannot access `proc/17274/fd/4': No such file or directory
du: cannot access `proc/17274/fdinfo/4': No such file or directory
0 proc
32M root
11M sbin
0 sys
68K tmp
13G usr
1.5G var
|
I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems.
The largest thing in that list is /home, at 35 GB. While that's healthy, there's no way the rest of those numbers add up to 48GB. I've heard that a Linux filesystem can build up "used" space that isn't actually in any file over time, and I've been using this one for some time (probably over a year; I haven't really been keeping track). Are there any utilities I can run to investigate this further and/or fix the problem if it is what I think?
Last edited by Dr. Frankenbox on Sun Apr 29, 2012 3:28 pm; edited 1 time in total |
|
| Back to top |
|
 |
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21624
|
| Posted: Sun Apr 29, 2012 3:51 am Post subject: |
|
|
| I am not aware of any Linux filesystems that leak space. Such behavior would be a bug that should be detected and fixed by the corresponding fsck. However, if you have deleted a file and not closed it, it will still consume space. |
|
| Back to top |
|
|
toralf
Developer
Joined: 01 Feb 2004
Posts: 3922
Location: Hamburg
|
| Posted: Sun Apr 29, 2012 6:02 am Post subject: |
|
|
| Hu wrote: | | I am not aware of any Linux filesystems that leak space. Such behavior would be a bug that should be detected and fixed by the corresponding fsck. However, if you have deleted a file and not closed it, it will still consume space. |
Right, the command "lsof" should point to such data. |
|
| Back to top |
|
|
Dr. Frankenbox
Apprentice
Joined: 16 Jul 2005
Posts: 170
Location: Iowa, USA
|
| Posted: Sun Apr 29, 2012 12:52 pm Post subject: |
|
|
But when I shut down and close the process(es) using those files, the space should be freed up - right? I certainly haven't deleted anything that large since I last booted (yesterday morning), and while I could believe I have some buggy processes on my system that are deleting files without closing them (or deleting each other's files without proper checking), I seriously doubt they could cause a problem like this in one day.
I will run lsof and see what it says, though. I just have to merge it first.
edit: Well, that was quick. There is a huge number of files open on my system (22,303), and many files have been opened by multiple processes, but the largest file in the list is only about 104MB. I seriously doubt it all adds up to 20-30GB, and most of these files probably haven't been deleted. Sill, is it normal for a running Gentoo desktop system with a GUI and a few applications open (I count 7 windowed apps, visible to the user) to have so many files open at a time? |
|
| Back to top |
|
|
Dont Panic
Guru
Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.
|
| Posted: Sun Apr 29, 2012 1:55 pm Post subject: Re: EXT4 Filesystem Full with "Ghost Data"? |
|
|
| Dr. Frankenbox wrote: | | I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems. |
Did you unmount the drives to see if you had any data that was being hidden by the mount point?
If you ever copied data to a directory in /mnt/ with nothing mounted at that point, and then mounted a partition on top of that point, it will hide the contents that were previously copied there unless you mounted with '-o bind'. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Apr 29, 2012 2:08 pm Post subject: |
|
|
| Dr. Frankenbox wrote: | | edit: Well, that was quick. There is a huge number of files open on my system (22,303), and many files have been opened by multiple processes, but the largest file in the list is only about 104MB. I seriously doubt it all adds up to 20-30GB, and most of these files probably haven't been deleted. Sill, is it normal for a running Gentoo desktop system with a GUI and a few applications open (I count 7 windowed apps, visible to the user) to have so many files open at a time? |
Looks normal to me:
| Code: | # lsof -n | wc -l
28915 |
|
|
| Back to top |
|
|
Dr. Frankenbox
Apprentice
Joined: 16 Jul 2005
Posts: 170
Location: Iowa, USA
|
| Posted: Sun Apr 29, 2012 3:28 pm Post subject: Re: EXT4 Filesystem Full with "Ghost Data"? |
|
|
| Dont Panic wrote: | | Dr. Frankenbox wrote: | | I excluded /mnt because I have other filesystems mounted there, and yes, I verified that the only data under /mnt is in those other filesystems. |
Did you unmount the drives to see if you had any data that was being hidden by the mount point?
If you ever copied data to a directory in /mnt/ with nothing mounted at that point, and then mounted a partition on top of that point, it will hide the contents that were previously copied there unless you mounted with '-o bind'. |
There it is!
| Code: | frankenbox mnt # mount
rootfs on / type rootfs (rw)
/dev/root on / type ext4 (rw,noatime,commit=0)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
rc-svcdir on /lib64/rc/init.d type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
udev on /dev type tmpfs (rw,nosuid,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda6 on /mnt/common type ext3 (rw,noatime,commit=0)
/dev/sdb1 on /mnt/bk type ext3 (rw,noatime,commit=0)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
/dev/sda2 on /mnt/windows type ntfs (rw,dmask=000,fmask=111,nls=utf8)
frankenbox mnt # umount windows/
frankenbox mnt # umount common/
frankenbox mnt # umount bk
frankenbox mnt # ls
bk cdimage cdrom common floppy gateway tmp windows
frankenbox mnt # du -sh *
32G bk
4.0K cdimage
4.0K cdrom
4.0K common
4.0K floppy
4.0K gateway
4.0K tmp
4.0K windows
|
Apparently at some point I had /mnt/bk unmounted while my backup scripts ran and happily saved my backups to my root partition instead of the separate drive I have reserved for them. Thank you for thinking of that, because I certainly wouldn't have.
So it turns out that I am really the fool here. My apologies to any ext4 developers I may have offended. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21624
|
| Posted: Sun Apr 29, 2012 4:36 pm Post subject: |
|
|
| As an alternate way to check for this problem, you can use bind mounts to expose directories that are normally hidden. In your case, you could have skipped the umount commands and instead run: mount --bind / /mnt/floppy && cd /mnt/floppy && du -sh * && umount /mnt/floppy. When a filesystem is bind-mounted into a new place, mounts beneath the original are not (by default) mounted in their corresponding place beneath the new place. Thus, the command I gave above would allow /mnt/floppy to show everything in /, even those things normally hidden by your active mounts. This is of marginal use in your case, but is very handy when the offending file is hiding under a popular mount point, such as the dev or var directories on root. Unmounting the filesystems that normally hide those directories is quite difficult in a running system, but exposing the contents via a bind mount is easy. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
