Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NFS4 much slower than NFS3 (when using subversion)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
Alan Turing
n00b
n00b


Joined: 10 Sep 2003
Posts: 53
Location: Freiburg, Baden (Germany)

PostPosted: Wed Mar 12, 2014 2:50 am    Post subject: NFS4 much slower than NFS3 (when using subversion) Reply with quote

I have a file server (using kernel 3.11.7-hardened-r1) on which an nfs server is running.
I have a desktop computer (using kernel 3.10.25-gentoo) which is mounting some folders which are exported by the server.

I am using subversion, the repository is somewhere in the internet, my local checkout is on the fileserver. This local checkout is also exported via nfs and sometimes mounted by the client.

Unfortunately this is very slow. An "svn up" done on the client takes around 135s. If I export the local checkout via samba, an "svn up" takes only around 20s.
Annoying: The "svn up" on the nfs filesystem seems to be slow for no reason. Neither on the client nor on the server I can see high activity with top or iotop.

However, mounting a filesystem via samba has some other disadvantages, and I live with this slow-svn-over-nfs problem for a few years now.


A few month ago, I switched from nfs3 to nfs4 and guess what? This made it even slower.
Meanwhile, I was too annoyed and made the following benchmarks.

First, I copied a 900MB file from the client to the nfs share.
With both, nfs3 and nfs4 this took around 9s. 100MB/s (over gigabit ethernet) - nothing to complain!

Furthermore, I copied the clients /usr/portage/dev-* (33102 files, together 100MB) to the nfs share.
With nfs3 this took 95s with nfs4 this took 135s.

Finally, I also compared the "svn up" on my local checkout. (For the case where there are no changes in the repository)
With nfs3 this took 135s with nfs4 this took 260s.

The /etc/exports on my server contains the line
Code:
/storage             192.168.0.42(rw,async,no_root_squash,no_subtree_check)


The /etc/fstab on my client contains the line
Code:
192.168.0.23:/storage  /mnt/nfs   nfs  noauto,nfsvers=4,retry=0,noatime,nocto,nolock,local_lock=all    0 0


I also tried different mount options and different export options. Using other options, the performance was slightly worse but there was always a big difference between nfs3 and nfs4.


Do you have any comments or suggestions for me?
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1241
Location: 42.68n 85.41w

PostPosted: Wed Mar 12, 2014 4:57 am    Post subject: Reply with quote

"Do you have any comments or suggestions for me?" suggestion... svn2git & git.... sftp & sshfs-fuse.... nfs, out the window, svn out the window... there are many ways to do things, and you're giving your self more means to the same ends.....
_________________
cat /etc/*-release
Funtoo Linux - baselayout 2.2.0
consider this warning no. 1
https://wiki.gentoo.org/index.php?title=Special:Contributions/666threesixes666&offset=&limit=500&target=666threesixes666


Last edited by 666threesixes666 on Wed Mar 12, 2014 9:55 am; edited 1 time in total
Back to top
View user's profile Send private message
Goverp
Guru
Guru


Joined: 07 Mar 2007
Posts: 345

PostPosted: Wed Mar 12, 2014 9:23 am    Post subject: Reply with quote

I'm in no way an NFS or SVN expert, but I've noticed that NFS can run slowly if you don't have the correct setup for handling ID mapping; I presume it's using some slow work-around in such circumstances. Hope that helps.
_________________
Greybeard
Back to top
View user's profile Send private message
Alan Turing
n00b
n00b


Joined: 10 Sep 2003
Posts: 53
Location: Freiburg, Baden (Germany)

PostPosted: Wed Mar 12, 2014 9:02 pm    Post subject: Reply with quote

I tested the sshfs fuse filesystem (using its default options)

Copying the 900MB file took 30s.
Three times as much as nfs3. But both cores of the servers CPU where at 100%.

Copying the 33102 files from /usr/portage/dev-* took 14m9s
Nine times as much as nfs3. But no CPU was used more than 10%.

Doing my "svn up" took 55s.
Faster than nfs3 !


Using git to access the svn repository?
That is an option than I am going to check. Might be complicated, because I want to use a client that is integrated in the eclipse IDE.

Problem with ID mapping?
Sounds reasonable, since with nfs4 there came this idmapd daemon. I am going to check this.
Back to top
View user's profile Send private message
krinn
Advocate
Advocate


Joined: 02 May 2003
Posts: 4294

PostPosted: Thu Mar 13, 2014 5:17 am    Post subject: Reply with quote

nfsv4 define a root in exports, if it's not there, you aren't doing it right. I'm not sure if newer implementations has simply drop this or not, but older server was accepting it , the v4 standard explicit the need of root, the implementation without one was out of the standard and all distros was doing what they want. Result was full of bugs.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1241
Location: 42.68n 85.41w

PostPosted: Thu Mar 13, 2014 6:00 am    Post subject: Reply with quote

svn2git, as in exporting your repo to git. then accessing git via git

Quote:
dev-vcs/svn2git
Tool for one-time conversion from svn to git.


so was sshfs fuse a good thing? im just throwing ideas out there. ssh is encrypted so certainly more secure than nfs. git by default uses ssh to push and pull commits.

https://www.eclipse.org/egit/

i guess its supported via egit? i dont use eclipse so its just a possibility being stated as a possibility.

its all moot to me i haven't looked into any bench marked comparisons.

gentoo's git wiki's starting to get pretty saucy... https://wiki.gentoo.org/wiki/Git

i like git because linus made it, and it revolves around diffs and is ultimately a patching system. i like git because it doesnt require svn repo upstream, you can git a folder on your desktop up and that is the upstream... git flow of git 2 git instead of client svn-srv
_________________
cat /etc/*-release
Funtoo Linux - baselayout 2.2.0
consider this warning no. 1
https://wiki.gentoo.org/index.php?title=Special:Contributions/666threesixes666&offset=&limit=500&target=666threesixes666
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 9053

PostPosted: Fri Mar 14, 2014 1:41 am    Post subject: Reply with quote

For converting a Subversion repository to Git, I prefer using git svn, which ships with core Git and is available with USE=subversion. However, OP may be happy with his legacy centralized VCS and prefer to solve the NFS slowness. Older Subversion was very hard on inodes due to how the shadow information was stored. Newer versions seem to be better, but may still be too heavy for use over NFS. It would be useful to know why Subversion performs so poorly here. OP, could you post a breakdown of number of each type of NFS request issued?
Back to top
View user's profile Send private message
Alan Turing
n00b
n00b


Joined: 10 Sep 2003
Posts: 53
Location: Freiburg, Baden (Germany)

PostPosted: Sun Mar 16, 2014 3:43 pm    Post subject: Reply with quote

I also defined a root in /etc/exports. I think the only difference is that you add fsid=0 to your root export in /etc/exports and use paths relative to this root in your mount options on the client.
So far I never noticed a difference between exporting a root via nfs or defining the exports in the old nfs3 style.

If the sshfs for me a good thing? I am indecisive. As written above, it was not always faster but it was also not always slower.

Converting the repository to GIT is not an option for me. I am neither the admin nor the only user of the repository.


I tried to obtain the NFS requests using nfsstat. Therefore, I rebootet the client to reset the counting of requests, did one "svn up" directly afterwards, and executed "nfsstat -l"

Using the following line in /etc/exports on the server
Code:
/storage             192.168.0.42(fsid=0,rw,async,no_root_squash,no_subtree_check)

and the following line in /etc/exports on the client
Code:
192.168.0.23:/  /mnt/nfs    nfs         noauto,nfsvers=4,retry=0,noatime,nocto,nolock,local_lock=all            0 0

I obtained the following output
Code:

bionseY ~ # nfsstat -l
nfs v4 client        total:    47171
------------- ------------- --------
nfs v4 client         read:    16551
nfs v4 client        write:       29
nfs v4 client       commit:        2
nfs v4 client         open:       17
nfs v4 client    open_conf:        1
nfs v4 client        close:        8
nfs v4 client       fsinfo:        5
nfs v4 client        renew:        3
nfs v4 client    setclntid:        3
nfs v4 client      confirm:        3
nfs v4 client         lock:     6594
nfs v4 client        locku:     6573
nfs v4 client       access:     3272
nfs v4 client      getattr:     3395
nfs v4 client       lookup:     3260
nfs v4 client  lookup_root:        2
nfs v4 client       remove:        7
nfs v4 client     pathconf:        3
nfs v4 client       statfs:        8
nfs v4 client      readdir:     4144
nfs v4 client  server_caps:        8
nfs v4 client  rel_lkowner:     3283


Afterwards I did the same with nfs3 and obtained the following output.
Code:

bionseY ~ # nfsstat -l
nfs v3 client        total:    30318
------------- ------------- --------
nfs v3 client      getattr:     3321
nfs v3 client       lookup:     3256
nfs v3 client       access:     3260
nfs v3 client         read:    16561
nfs v3 client        write:       27
nfs v3 client       create:        7
nfs v3 client       remove:        7
nfs v3 client  readdirplus:     3870
nfs v3 client       fsstat:        4
nfs v3 client       fsinfo:        2
nfs v3 client     pathconf:        1
nfs v3 client       commit:        2
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum