View previous topic :: View next topic |
Author |
Message |
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Mon Sep 09, 2013 5:40 pm Post subject: NFSv3 really slow? (was: QEMU issue?) |
|
|
Anyone use QEMU (virtmanager) and NFS to the parent machine?
My VM mounts a NFS share on the host that runs that VM. For some reason that NFS share is excruciatingly slow, seems like I get only a few MB/min trying to rsync to it (trying to backup the VM disk). Any ideas how to speed it up?
Any better ways to backup virtual machine images?
EDIT:
I think I have ruled this down to NFSv3 with small files. There's probably some locking issue going on here. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Last edited by eccerr0r on Wed Sep 11, 2013 9:13 pm; edited 1 time in total |
|
Back to top |
|
|
azieba n00b
Joined: 09 Sep 2013 Posts: 9
|
Posted: Tue Sep 10, 2013 5:00 pm Post subject: |
|
|
A little more information could help. What is the processor usage and the network usage on host and guest machine ? |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Tue Sep 10, 2013 5:45 pm Post subject: |
|
|
The (Physical) Network utilization is near 0 - after all this should be RAM-to-RAM copy. In virt-manager it reports very little network activity on the VM.
The guest/virtual machine is mostly in D-wait, otherwise it is idle. This implies it's a "host" issue.
The host machine... I never looked at, it's a quad core machine and its load average has always been less than 4. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
azieba n00b
Joined: 09 Sep 2013 Posts: 9
|
Posted: Tue Sep 10, 2013 5:52 pm Post subject: |
|
|
Check just to be safe CPU usage on host machine by nfs process during the transfer (with top or/and htop ). |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Tue Sep 10, 2013 6:38 pm Post subject: |
|
|
Code: | vm:/nfs/host/tmp$ time dd if=/dev/zero of=temp bs=1M count=2K
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 102.517 s, 20.9 MB/s
real 1m42.620s
user 0m0.007s
sys 0m2.850s
|
Hmm. During this time, the host remained mostly idle but load average went up implying a lot of disk wait. NFSd was remained less than 3% of one core. qemu remains as #1 CPU utilizer but less than 50% of a core.
At 20MB/sec it's bad but not completely horrible (but much worse than when I had this as a physical machine), it's the small files that really kills, and I tend to work with many small files with this virtual machine... Backup over NFS... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
azieba n00b
Joined: 09 Sep 2013 Posts: 9
|
Posted: Wed Sep 11, 2013 6:15 pm Post subject: |
|
|
I'm not certain but still it feels like something with CPU usage or disk waiting for me. Especially with lots of little files. Try running the backup and then watch CPU usage and disk waiting on (use top ) and maybe io operations on (use iotop) . The dd is not showing te real thing since its reading zeros and not from disk,
If it's IO problem you should notice high usage of CPU by IO operation and lots of waiting:
Look closely at third line (you should see something like below (it's in percents scale) )
%Cpu(s): 4,6 us, 1,0 sy, 0,3 ni, 93,5 id, 0,7 wa, 0,0 hi, 0,0 si, 0,0 st |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Wed Sep 11, 2013 9:06 pm Post subject: |
|
|
It almost seems like the server is sleeping between calls.
This was hard to grab a shot when nfsd actually showed up in 'top'. This is on the host machine:
Code: | top - 14:55:20 up 2 days, 14:04, 6 users, load average: 1.71, 0.70, 0.34
Tasks: 209 total, 1 running, 208 sleeping, 0 stopped, 0 zombie
%Cpu0 : 3.9 us, 1.0 sy, 0.0 ni, 94.4 id, 0.3 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu1 : 0.7 us, 0.0 sy, 0.0 ni, 92.6 id, 6.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 3.0 us, 0.7 sy, 0.0 ni, 94.7 id, 1.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 1.0 us, 1.3 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 6104356 total, 4355868 used, 1748488 free, 322772 buffers
KiB Swap: 1048572 total, 178092 used, 870480 free, 988044 cached
PID USER PR NI VIRT RES SHR S %CPU TIME+ P SWAP COMMAND
24358 fgu 20 0 1524764 302532 62120 S 3.322 66:14.99 3 0 firefox
2784 qemu 20 0 2346580 783600 2052 S 2.990 350:18.70 2 28180 qemu-s+
2373 root 20 0 671152 343504 208660 S 1.329 226:46.79 3 7800 X
2478 root 20 0 876652 6740 3472 S 0.997 45:01.60 1 1568 libvir+
3223 fgu 20 0 988680 74680 20260 S 0.997 33:48.43 2 0 python2
2291 root 20 0 0 0 0 D 0.332 0:23.64 1 0 nfsd
2764 qemu 20 0 1541456 130536 1676 S 0.332 6:01.87 1 96776 qemu-s+
|
The 2291 nfsd goes away fairly quickly, and another nfsd thread goes into D state.
This is what I did on the client (process 2784) (though I have another virtually idle qemu at 2764):
Code: |
vm:~$ du -s /usr/include/
235548 /usr/include/
vm:~$ time cp -a /usr/include /nfs/host/tmp/test
(aborted after 20 minutes)
|
I just tested this on a physical machine and noticed it's just as slow. Hmm. This is a NFS issue and not QEMU.
NFS mount options: rw,nosuid,nodev,hard,intr,sloppy _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
azieba n00b
Joined: 09 Sep 2013 Posts: 9
|
Posted: Thu Sep 12, 2013 7:57 pm Post subject: |
|
|
CPU and IO seems fine, hmm. From what I read NFS is always much slower with lots of small files, maybe some optimization may help,
modifying rsize and wsize maybe.
There should be something on NFS tuning here http://nfs.sourceforge.net/nfs-howto/ar01s05.html |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu Sep 12, 2013 8:37 pm Post subject: |
|
|
I wonder if asynchronous writes are enabled default.
I suppose I don't have to worry about the nfs server machine going down in this QEMU case... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|