View previous topic :: View next topic |
Author |
Message |
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Fri Jun 21, 2013 5:05 am Post subject: Filesystems go missing |
|
|
My desktop computer recently developed a habit of losing access to its filesystems (physical, network, and virtual) a few hours after startup. The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem. The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths).
There are no relevant entries in dmesg or /var/log/messages, and the disappearance doesn't seem to be related to anything I do. Nothing changed around the time the problem developed, but since I don't reboot my computer very often, this could be long-delayed fallout from an upgrade as much as three months ago.
I'm running kernel 3.6.11; other version info is available on request.
Any ideas? |
|
Back to top |
|
|
audiodef Watchman
Joined: 06 Jul 2005 Posts: 6639 Location: The soundosphere
|
|
Back to top |
|
|
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Fri Jun 21, 2013 11:03 pm Post subject: |
|
|
I usually update once a week, but I'll often put off things like X or running applications. |
|
Back to top |
|
|
wcg Guru
Joined: 06 Jan 2009 Posts: 588
|
Posted: Sat Jun 22, 2013 12:03 am Post subject: |
|
|
At a guess, some corrupted data structure in the kernel's
dcache (directory cache) or inode cache. You can try to drop
caches when it happens and see if that fixes it.
Code: |
echo 3 > /proc/sys/vm/drop_caches
|
(See /usr/src/linux/Documentation/sysctl/vm.txt.)
If that works, you can change the cache flush to
Code: |
echo 2 > /proc/sys/vm/drop_caches
|
to skip freeing pagecache and only free the dcache and
inode cache and see if that works. _________________ TIA |
|
Back to top |
|
|
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Sat Jun 22, 2013 2:10 am Post subject: |
|
|
wcg wrote: | At a guess, some corrupted data structure in the kernel's
dcache (directory cache) or inode cache. You can try to drop
caches when it happens and see if that fixes it.
Code: |
echo 3 > /proc/sys/vm/drop_caches
|
|
Didn't fix it. |
|
Back to top |
|
|
wcg Guru
Joined: 06 Jan 2009 Posts: 588
|
Posted: Sat Jun 22, 2013 2:55 am Post subject: |
|
|
Well, the shell has a concept of "current working directory", but that
part seems to be working, because listing the contents of directories
relative to the current directory works. You can check it of course:
Code: |
pwd
env | grep `pwd`
|
But I expect that the results will be what you expect them to be.
The problem seems to be absolute paths (beginning with '/').
I do not really see how that could be other than an inode or
dcache problem.
(Is the shell mangling the first character of the pathname as
a command parameter? Then it seems like it would mangle
the first character of relative pathnames, too, and for commands
with more command line options, the shell would not know
which one was the pathname.)
What about something like emacs' dired (directory editing
mode), which will list the contents of a directory in an
editing window? (You may not have emacs. I expect that
vim has some command that will do the same thing, like
) nano apparently does not have that functionality by default.
"nano /tmp" simply reports that /tmp is a directory (meaning
not an editable text file).
Do you see the directory with an editor's directory listing
command when passing both absolute and relative paths
to the editor?
I guess it could be some filesystem bug. What kind of filesystem
is this? _________________ TIA |
|
Back to top |
|
|
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Sat Jun 22, 2013 3:31 am Post subject: |
|
|
wcg wrote: | I guess it could be some filesystem bug. What kind of filesystem
is this? |
All of 'em. "/" is NFSv2 remounted as NFSv3, /home is NFSv3, "/dev" is a devtmpfs, "/proc" is procfs, "/run" and "/tmp" are tmpfs, "/sys" is a sysfs, "/mnt/cdrom" is an ISO9660 filesystem, "/mnt/memcard" is FAT32, and so on. |
|
Back to top |
|
|
wcg Guru
Joined: 06 Jan 2009 Posts: 588
|
Posted: Sat Jun 22, 2013 12:36 pm Post subject: |
|
|
So probably not a filesystem bug. VFS at most.
This is not a problem that I see reported a lot. I suspect that
having root mounted over nfs has something to do with it,
although I do not know what exactly. Maybe nfs users will
have more insight or be able to suggest diagnostics. _________________ TIA
Last edited by wcg on Mon Jun 24, 2013 1:56 pm; edited 1 time in total |
|
Back to top |
|
|
TomWij Retired Dev
Joined: 04 Jul 2012 Posts: 1553
|
Posted: Sat Jun 22, 2013 8:11 pm Post subject: Re: Filesystems go missing |
|
|
Carnildo wrote: | The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem. The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths). |
Interesting, haven't heard about this, though it sounds like an udev or kernel bug.
Carnildo wrote: | I'm running kernel 3.6.11; other version info is available on request. |
Could you try a more recent kernel? The stable 3.8.13, the unstable 3.9.7 or the development 3.10-rc6? |
|
Back to top |
|
|
Carnildo Guru
Joined: 17 Jun 2004 Posts: 594
|
Posted: Sun Jun 23, 2013 9:13 pm Post subject: Re: Filesystems go missing |
|
|
TomWij wrote: | Carnildo wrote: | The root filesystem is still there, but everything else (/dev, /proc, /home, and so on) goes missing: the mountpoints show up as empty directories, or in the case of /dev, the static device nodes on the root filesystem. The filesystems still show up as mounted, and any terminal with a working directory in one of the filesystems can still see its contents (but only if accessed using relative paths). |
Interesting, haven't heard about this, though it sounds like an udev or kernel bug. |
That's the conclusion I came to as well, after having the problem with ever-more-minimal systems (I'd narrowed it down to one of udev, bash, init, md5sum, or the kernel, and I'm pretty sure that bash and md5sum running as an unprivileged user can't cause it).
Quote: | Carnildo wrote: | I'm running kernel 3.6.11; other version info is available on request. |
Could you try a more recent kernel? The stable 3.8.13, the unstable 3.9.7 or the development 3.10-rc6? |
3.8.13 has been running without trouble for a little over 24 hours, which is longer than 3.6.11 ever managed once things started going wrong. |
|
Back to top |
|
|
augury l33t
Joined: 22 May 2004 Posts: 722 Location: philadelphia
|
Posted: Tue Jun 25, 2013 7:28 am Post subject: |
|
|
Your disk may be going bad. It's a physical thing. Overheating, dirt.
What sort of spindles are you running? Any noise? Difficulty starting up? You can *usually* save the data but the reliability will decrease. The performance hit isn't to bad. |
|
Back to top |
|
|
|