Directories disappearing with NFS

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

I've recently noticed that occasionally some directories will become inaccessible/disappear on a NFS share. They will appear with question marks in the directory listing, like:

zeronullity · Tux's lil' helper Joined: 16 Oct 2010 Posts: 103

This is not a share of a share is it? As this would not be supported & not to mention be a security risk.
Also do you have this problem with all NFS clients you've tested? As this problem is often caused by
the client side & not server side. Try mount -a, along with exportfs -a next time you run into this issue
and see if it corrects it.

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

zeronullity · Tux's lil' helper Joined: 16 Oct 2010 Posts: 103

Try disabling any drive standby/power management on the server & see if the same problem occurs.

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

zeronullity · Tux's lil' helper Joined: 16 Oct 2010 Posts: 103

What does the file names look like.. standard? If in another language you may need to add UTF-8 support for that language.
As NFS can sometimes have issues with file names.

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

Just to exclude the possibility of it being either a hardware or a filesystem error I ran SMART self-tests on both hard drives and checked the XFS filesystem with `xfs_repair -n`. The "Extended offline" self-tests reported "Completed without error" for both sda and sdb, and xfs_repair found nothing wrong with the consistency of the filesystem.
I've also moved the offending directories into their own subdirectory and they continued to misbehave occasionally as before, so now I've made copies of those misbehaving directories and will report back if they continue to show the same behaviour.

Just to make clear once more, these problems are non-existent on the server itself - I can browse around just fine and if I `ls` the directories giving me trouble (trouble on the NFS clients that is) the problems disappear... for a while (maybe 30 minutes to an hour). It's only on the NFS shares mounted on the clients I have problems with certain directories - whenever the problems arise it seems to effect the same directories, hence why my last effort has been copying them in the hope of fixing whatever is wrong. Does NFS have a maximum entries per directory? Or a maximum file-depth I'm bumping against? I'm seriously lost.

PS: Samba has none of these problems, the Windows clients on my network using SMB have not suffered from these problems. It's solely the Linux clients (a Gentoo and an Ubuntu box) using the NFS share.
_________________
This post was created by millions of tiny cows jumping around on my keyboard.

zeronullity · Tux's lil' helper Joined: 16 Oct 2010 Posts: 103

Has NFSv4 had issues with large directories in the past YES, does it currently have with the version your using.. I'm not sure.
You can try setting your test directories using standard file names, similar sizes, permissions, etc. to see if same behavior occurs with small/large amount of files. If it only happens on certain directories that should be a red flag.

Approx. how many files are we talking about..?

Did you check your setting for fs.file-max/fs.file-nr?

The maximum file size over NFSv4 on ia32, when server runs XFS is 8 TB. This match to the maximal file size the local file system is able to create (on this architecture). So I don't think file size has anything to do with it. Number of files in a directory is more likely.

Have you tried sshfs any problems with that?

NFS may not like the name/path or # of files in your directory for what ever reason, I'd make more directories for testing to narrow the issue,
making sure to omit the directories you had issues with previously. Perhaps even making a COPY of the directory having all the issues and try
resetting file mode/permissions & file names recursively with different values.

I know on other NFS OS, similar issues are related to user permissions and if your using SYS_AUTH or kerberos, even having too many groups in a uid could cause issues if running older versions NFS /w newer versions of NFS on some particular systems.

There is a option also if I recall correctly to run v4 as v3 in dumbed down mode I don't recall what it is on Linux.. you may try that also.

Does the same problem occur with no_subtree_check disabled?

Ivion · n00b Joined: 23 Jan 2003 Posts: 45 Location: Amsterdam

zeronullity · Tux's lil' helper Joined: 16 Oct 2010 Posts: 103

I know people who've had similar issues in the past, downgrading to v3 NFS has usually fixed these issues.

Also the way VFS/NFS works it shouldn't make a difference what file system your using.. however if you have a spare drive it
might be worth trying a different FS other then XFS.

79 files per directory is no where near a lot.. and I wouldn't consider the size of the directory to be a issue.

Were these files/directories in question originally created on the server or copied from another system over the network or from another drive perhaps containing
another file system type? I've seen unusual issues in the past when copying directories from NTFS/FAT partitions to other file system types & vice versa.

This sounds like a caching issue on the server side either due to..

- bug
- uid/gid authentication issues between server/client
- lack of support in kernel, UTF, file system settings, etc.

Also check to see if you have /var/lib/nfs/v4recovery
If not try creating it. Also make sure it's readable/writable by the owner.
I'd try getting the issue below resolved before troubleshooting any further.