Strange kvm performance problem

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

Hi,
I hope somebody can help me. I have a machine (i7 3770, 16GB RAM) running Gentoo as kvm-hypervisor. I installed two machines both Centos 6.5. One with mysql as database server and one with zendserver as webserver. For testing and comparison I use some php-application to create a database.

Here my results:

1) Both Servers on their own physical machine -> database creation takes about 3 minutes
2) Both Servers on Virtualbox (Win7 as hypervisor) -> database creation takes about 3,5 minutes
3) Both Servers on the gentoo machine (internal disk 7200rpm) -> database creation takes about 32 minutes :!:

4) Both Servers on the gentoo machine (usb3 disk 7200rpm) -> database creation takes about 5 minutes

Various tests done on the 3rd configuration:
importing dump on the mysql server locally -> normal speed
importing dump remotely from the zendserver -> normal speed
network copy from zendserver to mysqlserver -> normal speed
iperf between host and mysql -> nearly 1GB
iperf between zendserver and mysqlserver -> nearly 1GB
local disk performance tests an the mysql shows disk write speeds between 40 and 70MB/s

Has anybody any idea why running from the internal disk is so incredibly slow ?
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

sk3l · Posted: Sun Mar 30, 2014 6:53 pm Post subject:

Presumably you are using the virtio block drivers? If not, that would the first thing I'd put in place. If yes, are you using the default I/O tuning parameters, or writethrough caching, etc? The I/O tuning parameters can have an impact on disk performance, AFAIK.

Also, what image format are you using, and are you using a file on the host system, LVM or a dedicated block device for the VM image?

If the DB creation is fast with VirtualBox, and using a USB drive, then odds are there is something you ~should~ be able to tune with your virtual disk properties on the Gentoo host to get the timing down closer to 3 minutes.

If you can figure out what is bottlenecking your virtual disk activity on the Gentoo host, I'd definitely recommend a) upgrading your Gentoo's disk device and/or b) use a dedicated block device to host the VM images, if possible. That should give you even more I/O performance.

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

Thanks for your reply,
I do use all virtio-drivers(net, block) and the virtual-discs are qcow2 files. I tried all tunings available in kvm(cache set to none, preallocation of metadata, elevator to noop).
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

sk3l · Posted: Sun Mar 30, 2014 9:53 pm Post subject:

You could try switching to a different image format to see what that yields. I use raw myself, after moving away from qcow2. However, that doesn't seem like it would make that large of a difference.

The only other thing is perhaps there is some parameter to tune mysql, letting it know that the underlying system is virtualized.

I run KVM on both my Gentoo desktop & OpenSUSE laptop, with a Windows 7 guest for my company's software dev. Obviously there is I/O latency, but not what you're seeing, an order of magnitude slower. The key detail seems to be (if I'm understanding this right) that if you use an external USB HDD attached to the guests, then the DB build is like 5 minutes, which is much better than 32. That tells me it is definitely something related to the virtual disk config & not the VM as a whole.

What about trying SCSI or SATA virtual disk interfaces instead of virtio to see if that makes any difference? Clearly it shouldn't be faster than virtio, but one never knows.

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

I tried all those options. The strange thing is that the same virtual machines are fast if the virtual disc files are stored on usb and slow if stored on the internal SATA-disc drive.
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

sk3l · Posted: Mon Mar 31, 2014 4:22 pm Post subject:

Maybe try a different guest, other than Cent OS, and see if that changes anything? It seems like you've tried everything already, so I'll bet you've probably done that as well.

I'm stumped. Sorry I couldn't be more help, but VM is merely a hobby for me, and a way to reconcile Linux environments w/ my job's Windows dev requirements.

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

No problem. I appreciate any input.

Maybe someone else got any idea what's going on here ?
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

miket · Posted: Wed Apr 02, 2014 3:01 am Post subject: qcow2 might the problem

Sometimes for testing with MSIE I fire up one VM image or another I have with Windows. I use qcow2 for the disk images, but run the VM's in snapshot mode. This suits my purposes because it lets me discard any changes made in the session. The other huge benefit, though, is that it runs hugely faster that way.

All this points out to me that qcow2, for all its nice attributes, imposes quite a speed penalty for writes. Actually, though I suspect it may bog down reads also, I have no actual numbers for you. All the same, I would suspect you'd get quite the performance boost if you switched to raw format, or as someone suggested, a physical or LVM volume.

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

Hi,
yes I read all those things. I would agree if the same machine running from usb disc would also be slow but that isn't the case.
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

Are the SATA and USB disks both using the same filesystem? Is the internal one full and causing the qemu images to badly fragment maybe? (try /usr/sbin/filefrag)

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

Both are ext4 filesystems 1TB, containing only the disk images
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

vaxbrat · l33t Joined: 05 Oct 2005 Posts: 731 Location: DC Burbs

When you switch out hypervisors, are they on the same drive on the same machine? You aren't clear about that. I'm wondering if your internal gentoo drive maybe has one or more sectors in the guest image on the verge of being locked out due to errors. What does a "smartctl -a" on that drive show?

FWIW I run raw format on btrfs and have never tried qcow2. btrfs lets me snapshot and rollback the vm, and will also clue me in if it's finding anything wrong from its own checksumming. I do run Centos 6.2 reference vm's on btrfs with virtio, cache and tuning defaulted, compression=lzo and autodefrag turned on.

I only turn off autodefrag for win7 corporate images which run Sophos A/V. That crap intercepts i/o's and writes so much it drives the brtrfs workers crazy and drags everything down as a result. Different rants for different folk I guess

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

I don't switch hypervisors. I have each virtualdisc on internal drive and an copy of it on the usb drive. I only replace the virtualdiscs in the config of the virtualmachine. It is always the same host machine using kvm.
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

mike155 · Posted: Fri Apr 04, 2014 7:22 pm Post subject:

Have you tried to turn ON disk write caching of your host system by setting "cache" to "unsafe"? For this test, your QEMU parameters should contain something like this:

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

Yes that's the performance I'm looking for

Just to be clear, the setting itself is not dangerous if the machine is running with an UPS and a RAID-Controller with BBU, correct ?
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

mike155 · Posted: Sat Apr 05, 2014 7:16 pm Post subject:

Nearly all databases frequently emit sync requests (write barriers) when writing data. That's the reason why your system is slow if your database runs in a VM and QEMU is configured without full write caching. If you run QEMU with full write caching, the sync requests are thrown away by the write cache and your system is much faster. VMs just are not optimized for databases. This may change in the future, but right now, I don't recommend to run databases in VMs. Run your database on the host system if you need speed.

If you have a battery backed RAID controller, then I'm a bit surprised of your results. Your battery backed RAID Controller should also be able to neutralize write barriers and you shouldn't see a huge slowdown. Are you sure that caching of your RAID controller works and that caching mode is set to writeback?

drizzt · Guru Joined: 21 Jul 2002 Posts: 428

At the moment I don't use a RAID-Controller. Since this is just a test system I'm okay with the potential risk of corrupting my virtual test machines.
If the database's write barriers is the culprit, then why is the USB disc so fast ? Is this disc always write cached ?
_________________
People don't have to earn my respect. I offer my respect to them, but be careful to lose my respect...

mike155 · Posted: Sat Apr 05, 2014 7:47 pm Post subject: