Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NFSv3 really slow? (was: QEMU issue?)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Mon Sep 09, 2013 5:40 pm    Post subject: NFSv3 really slow? (was: QEMU issue?) Reply with quote

Anyone use QEMU (virtmanager) and NFS to the parent machine?

My VM mounts a NFS share on the host that runs that VM. For some reason that NFS share is excruciatingly slow, seems like I get only a few MB/min trying to rsync to it (trying to backup the VM disk). Any ideas how to speed it up?

Any better ways to backup virtual machine images?

EDIT:

I think I have ruled this down to NFSv3 with small files. There's probably some locking issue going on here.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?


Last edited by eccerr0r on Wed Sep 11, 2013 9:13 pm; edited 1 time in total
Back to top
View user's profile Send private message
azieba
n00b
n00b


Joined: 09 Sep 2013
Posts: 9

PostPosted: Tue Sep 10, 2013 5:00 pm    Post subject: Reply with quote

A little more information could help. What is the processor usage and the network usage on host and guest machine ?
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Tue Sep 10, 2013 5:45 pm    Post subject: Reply with quote

The (Physical) Network utilization is near 0 - after all this should be RAM-to-RAM copy. In virt-manager it reports very little network activity on the VM.

The guest/virtual machine is mostly in D-wait, otherwise it is idle. This implies it's a "host" issue.

The host machine... I never looked at, it's a quad core machine and its load average has always been less than 4.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
azieba
n00b
n00b


Joined: 09 Sep 2013
Posts: 9

PostPosted: Tue Sep 10, 2013 5:52 pm    Post subject: Reply with quote

Check just to be safe CPU usage on host machine by nfs process during the transfer (with top or/and htop ).
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Tue Sep 10, 2013 6:38 pm    Post subject: Reply with quote

Code:
vm:/nfs/host/tmp$ time dd if=/dev/zero of=temp bs=1M count=2K
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 102.517 s, 20.9 MB/s

real    1m42.620s
user    0m0.007s
sys     0m2.850s

Hmm. During this time, the host remained mostly idle but load average went up implying a lot of disk wait. NFSd was remained less than 3% of one core. qemu remains as #1 CPU utilizer but less than 50% of a core.

At 20MB/sec it's bad but not completely horrible (but much worse than when I had this as a physical machine), it's the small files that really kills, and I tend to work with many small files with this virtual machine... Backup over NFS...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
azieba
n00b
n00b


Joined: 09 Sep 2013
Posts: 9

PostPosted: Wed Sep 11, 2013 6:15 pm    Post subject: Reply with quote

I'm not certain but still it feels like something with CPU usage or disk waiting for me. Especially with lots of little files. Try running the backup and then watch CPU usage and disk waiting on (use top ) and maybe io operations on (use iotop) . The dd is not showing te real thing since its reading zeros and not from disk,
If it's IO problem you should notice high usage of CPU by IO operation and lots of waiting:
Look closely at third line (you should see something like below (it's in percents scale) )

%Cpu(s): 4,6 us, 1,0 sy, 0,3 ni, 93,5 id, 0,7 wa, 0,0 hi, 0,0 si, 0,0 st
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Wed Sep 11, 2013 9:06 pm    Post subject: Reply with quote

It almost seems like the server is sleeping between calls.

This was hard to grab a shot when nfsd actually showed up in 'top'. This is on the host machine:
Code:
top - 14:55:20 up 2 days, 14:04,  6 users,  load average: 1.71, 0.70, 0.34
Tasks: 209 total,   1 running, 208 sleeping,   0 stopped,   0 zombie
%Cpu0  :  3.9 us,  1.0 sy,  0.0 ni, 94.4 id,  0.3 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  :  0.7 us,  0.0 sy,  0.0 ni, 92.6 id,  6.7 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  3.0 us,  0.7 sy,  0.0 ni, 94.7 id,  1.7 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  1.0 us,  1.3 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   6104356 total,  4355868 used,  1748488 free,   322772 buffers
KiB Swap:  1048572 total,   178092 used,   870480 free,   988044 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU     TIME+ P   SWAP COMMAND
24358 fgu       20   0 1524764 302532  62120 S 3.322  66:14.99 3      0 firefox
 2784 qemu      20   0 2346580 783600   2052 S 2.990 350:18.70 2  28180 qemu-s+
 2373 root      20   0  671152 343504 208660 S 1.329 226:46.79 3   7800 X
 2478 root      20   0  876652   6740   3472 S 0.997  45:01.60 1   1568 libvir+
 3223 fgu       20   0  988680  74680  20260 S 0.997  33:48.43 2      0 python2
 2291 root      20   0       0      0      0 D 0.332   0:23.64 1      0 nfsd
 2764 qemu      20   0 1541456 130536   1676 S 0.332   6:01.87 1  96776 qemu-s+

The 2291 nfsd goes away fairly quickly, and another nfsd thread goes into D state.

This is what I did on the client (process 2784) (though I have another virtually idle qemu at 2764):
Code:

vm:~$ du -s /usr/include/
235548  /usr/include/
vm:~$ time cp -a /usr/include /nfs/host/tmp/test
(aborted after 20 minutes)


I just tested this on a physical machine and noticed it's just as slow. Hmm. This is a NFS issue and not QEMU.

NFS mount options: rw,nosuid,nodev,hard,intr,sloppy
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
azieba
n00b
n00b


Joined: 09 Sep 2013
Posts: 9

PostPosted: Thu Sep 12, 2013 7:57 pm    Post subject: Reply with quote

CPU and IO seems fine, hmm. From what I read NFS is always much slower with lots of small files, maybe some optimization may help,
modifying rsize and wsize maybe.
There should be something on NFS tuning here http://nfs.sourceforge.net/nfs-howto/ar01s05.html
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9645
Location: almost Mile High in the USA

PostPosted: Thu Sep 12, 2013 8:37 pm    Post subject: Reply with quote

I wonder if asynchronous writes are enabled default.

I suppose I don't have to worry about the nfs server machine going down in this QEMU case...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum