Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NFS completely fucked, misc. questions.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2253
Location: $TERM

PostPosted: Sat May 21, 2011 7:32 am    Post subject: NFS completely fucked, misc. questions. Reply with quote

On my side NFS is working just horribly.

1) If I set timeo to 3 seconds, the NFS client should retry requests every 3 seconds in case the server is too slow. In my case if I do any operation apart from a very heavy operation, the client applications utilizing the NFS shares hangs completely. Even if the second application utilizing the share is bash completetion. Yup, you got it right, dd on one side and bash completion on the other completely hangs the clients with a lot of -

Code:
...
nfs: server 192.168.1.3 not responding, still trying
nfs: server 192.168.1.3 OK
nfs: server 192.168.1.3 OK
nfs: server 192.168.1.3 not responding, still trying
...


Going on. And at times it's gonna hang everything except the kernel... even the ttys forcing a hard reboot.

2) HDD activity on the server too high -- I don't understand why a simple dd operation from the network share causes the HDD light to list as if the dd operation was going on locally, as a result I get 127 Kbps on a 100Mbps network and with a HDD with performance at 100MB/s. The performance of the dd operation is just good initially.

To top that off the dd operation refuses to die, even if I try to kill the process or umount the share.

All this happens regardless of the timeo value.

Once (recently) this happens with timeo=60

mount options -

rsize=200,wsize=200,async,hard,timeo=60,retrans=2,actimeo=900,retry=2,lookupcache=pos,nfsvers=4,nolock,intr,noacl,rdirplus
_________________
My blog
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15989

PostPosted: Sat May 21, 2011 6:33 pm    Post subject: Reply with quote

Those are really tiny read/write windows. Larger windows are generally better for performance. I mount a share with rw,rsize=524288,wsize=524288,timeo=600,retrans=2 (among other options) and can stream MythTV recordings over the NFS share with no stutter. A full read of a 1.6G file takes ~20 seconds:
Code:
time cat <test.mpg>/dev/null
real    0m20.549s
user    0m0.000s
sys     0m0.469s
Both machines are negotiated to 1000Mb/s, according to ethtool.
Back to top
View user's profile Send private message
whig
l33t
l33t


Joined: 27 Nov 2004
Posts: 972
Location: New Zealand

PostPosted: Sat May 21, 2011 11:52 pm    Post subject: Reply with quote

It may help to test the network speeds up/down eg net-misc/iperf. Ping loss perhaps?
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2253
Location: $TERM

PostPosted: Sun May 22, 2011 3:28 am    Post subject: Reply with quote

Hu wrote:
Those are really tiny read/write windows. Larger windows are generally better for performance. I mount a share with rw,rsize=524288,wsize=524288,timeo=600,retrans=2 (among other options) and can stream MythTV recordings over the NFS share with no stutter. A full read of a 1.6G file takes ~20 seconds:
Code:
time cat <test.mpg>/dev/null
real    0m20.549s
user    0m0.000s
sys     0m0.469s
Both machines are negotiated to 1000Mb/s, according to ethtool.


Wow. However in my case mounting with r/wsize 80000 does not work, and going above 200 will results in hangs on heavy file transfers (unsquahsfs for e.g)

If I don't specify r/wsize option the data transfer is a lot faster but still very slow, at best 4 MB/s with dd. However with other operations I get 88 Mb/s. But i/o on the server is still a problem, atleast with dd it's still a bottleneck with bandwidth. Does increasing the r/wsize help reduce i/o on server?

whig wrote:
It may help to test the network speeds up/down eg net-misc/iperf. Ping loss perhaps?


Nope, everything else works fine including X over ssh, which'll definitely trip if a bit goes wrong.
_________________
My blog
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2253
Location: $TERM

PostPosted: Sun May 22, 2011 4:15 am    Post subject: Reply with quote

Ok, my fault -- the r/wsize I thought was the packet size, thus I was suffering to the hardware's MTU, and initially set it to 1500. Now it's 1048576 and working well. This also solves the I/O issue on the server.

Thanks!!!
_________________
My blog
Back to top
View user's profile Send private message
blackwhite
Apprentice
Apprentice


Joined: 24 Jun 2004
Posts: 250

PostPosted: Mon May 23, 2011 2:31 pm    Post subject: Reply with quote

dE_logics wrote:
Ok, my fault -- the r/wsize I thought was the packet size, thus I was suffering to the hardware's MTU, and initially set it to 1500. Now it's 1048576 and working well. This also solves the I/O issue on the server.

Thanks!!!


Would you give me your system some details: kernel version, nfs-utils version? /etc/fstab nfs drive mount option, and /etc/conf.d/nfs?
In my case, several clients hangs up randomly on executing/copying files shared on NFS server, but the client still can copy other files on the NFS server.

Thanks.
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2253
Location: $TERM

PostPosted: Mon May 23, 2011 3:16 pm    Post subject: Reply with quote

blackwhite wrote:
dE_logics wrote:
Ok, my fault -- the r/wsize I thought was the packet size, thus I was suffering to the hardware's MTU, and initially set it to 1500. Now it's 1048576 and working well. This also solves the I/O issue on the server.

Thanks!!!


Would you give me your system some details: kernel version, nfs-utils version? /etc/fstab nfs drive mount option, and /etc/conf.d/nfs?
In my case, several clients hangs up randomly on executing/copying files shared on NFS server, but the client still can copy other files on the NFS server.

Thanks.


Client is Gentoo with nfs-utils version 1.2.3-r1 and kernel as 2.6.37-zen, mount options as follows -

mount -o rsize=1048576,wsize=1048576,async,hard,timeo=60,retrans=2,actimeo=900,retry=2,lookupcache=pos,nfsvers=4,nolock,intr,noacl,rdirplus

I do not use fstab for mounting, since I only mount sometimes.

Server is Debian testing with 2.6.32 using nfs-kernel-server v1.2.3, following are the export options -

anonuid=1000,anongid=1000,secure,subtree_check,rw,fsid=0
_________________
My blog
Back to top
View user's profile Send private message
blackwhite
Apprentice
Apprentice


Joined: 24 Jun 2004
Posts: 250

PostPosted: Mon May 23, 2011 4:37 pm    Post subject: Reply with quote

Thanks.
Would you give me emerge nfs-utils -pv outputs, thanks.
Quote:

emerge nfs-utils -pv
* Last emerge --sync was 41d 0h 59m 40s ago.

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R ] net-fs/nfs-utils-1.2.3-r1 USE="ipv6 nfsv3 nfsv4 tcpd -caps -kerberos" 0 kB
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum