Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NFS speed over virtual machines/qemu
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7274
Location: almost Mile High in the USA

PostPosted: Tue Sep 29, 2015 3:30 pm    Post subject: NFS speed over virtual machines/qemu Reply with quote

I wonder what people are seeing for NFS transaction speeds to virtual machines on the same host.

Setup: Core2 Quad Q9550S with 6GB RAM (DDR2/PC5300)
QEMU/KVM on host
Server disks is LVM on RAID5. All partitions except boot on LVM.
NFS server running on host to serve out LVM partitions

Networking: Software bridge, QEMU network via macvtap/macvlan. virtual machine using NFS.

Problem: NFS is extremely slow. For small files I'm seemingly getting IOPS of like 5 or so. When I get queued SMTP from remote hosts on my virtual machine, the IOPS is so slow that it can timeout when sendmail runs out of queue slots.

Interactive performance of shells is also very slow due to file open/close/lock speeds. I think after the files get opened/closed the performance is OK.

Any ideas how to speed things up? Should it be this slow? What kinds of speeds are you seeing? Perhaps LVM or RAID5 is what's causing issues?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1777

PostPosted: Tue Sep 29, 2015 6:02 pm    Post subject: Reply with quote

Quote:
For small files I'm seemingly getting IOPS of like 5 or so.
How you measure it so I could repeat it?

If it's getting slower for small files, I'd look at NFS itself first.
/etc/conf.d/nfs, the following line
# Number of servers to be started up by default
OPTS_RPC_NFSD="8"

This is the first thing that might need a big increase.
Another thing, is your IO blocking or parallel on the client side? (Does initializing session have a big impact?)

Quote:
starting machine testmachine with command
qemu-system-x86_64 -name testmachine -enable-kvm -m 2048 -smp cores=2 -net nic,model=rtl8139,macaddr=DE:AD:BE:EF:8F:2E,vlan=1 -net tap,ifname=testmachine-1,script=no,downscript=no,vlan=1

Well, it's an ancient image build without virtio, so it's pretty obvious where's the bottleneck, but let's see it anyway.
running "tar -c / | dd of=/dev/null" on qemu with squashfs over nfs root, so unfortunately it's one big file, but still:
iftop at host:
Code:

TX:             cum:   1.31GB   peak:   76.1Mb                                                                               rates:   74.6Mb  73.2Mb  73.2Mb
RX:                     250MB           14.2Mb                                                                                        13.9Mb  13.6Mb  13.6Mb
TOTAL:                 1.56GB           90.3Mb                                                                                        88.6Mb  86.8Mb  86.8Mb

a bit later
Code:
TX:             cum:   4.31GB   peak:   76.3Mb                                                                               rates:   55.9Mb  40.5Mb  49.5Mb
RX:                     816MB           14.2Mb                                                                                        8.55Mb  6.20Mb  8.08Mb
TOTAL:                 5.11GB           90.6Mb                                                                                        64.5Mb  46.7Mb  57.6Mb

and it keeps 2 nfsd bussy:
Code:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                 
12588 me        20   0 2761248 2.051g  34536 S 150.7 28.1  24:49.99 qemu-system-x86                                                                         
 4743 root      20   0       0      0      0 S  16.9  0.0   3:05.92 nfsd                                                                                     
 4742 root      20   0       0      0      0 R   9.9  0.0   1:44.86 nfsd                                                                                     
 7357 me        20   0 1369448 474248  92620 S   7.9  6.2 124:05.33 seamonkey                                                                               
 4174 root      20   0  305836  63568  34856 S   5.9  0.8  63:32.69 X                                                                                       


On the other hand, running tar against /usr/portage (without distfiles) gives ~4Mbps while running separate tar for every directory inside /usr/portage (skipping distfiles) gives between 15Mb and 40Mb
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7274
Location: almost Mile High in the USA

PostPosted: Tue Sep 29, 2015 6:21 pm    Post subject: Reply with quote

I suppose the perceived 5 IOPS more like a guess - I should look for a better benchmark but it needs to also test flocks. The impact is very visible at least to processing sendmail requests. I think it takes at least a minute to process a single mail from sendmail -> procmail -> write to disk, and if another sendmail request comes it, they seem to clash with each other, almost exponentially increasing time taken to complete. Come on, this is a small text email, shouldn't take a minute to procmail and write to disk... sometimes I get large batches of emails and sendmail seems to livelock - and I stop getting any mail.

I think just one of the nfsd's on the host/VM server is in D wait most of the time, but others are sleeping. It's clearly waiting on disk resources.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum