Strange kvm performance problem

[drizzt]

Hi,
I hope somebody can help me. I have a machine (i7 3770, 16GB RAM) running Gentoo as kvm-hypervisor. I installed two machines both Centos 6.5. One with mysql as database server and one with zendserver as webserver. For testing and comparison I use some php-application to create a database.

Here my results:

1) Both Servers on their own physical machine -> database creation takes about 3 minutes
2) Both Servers on Virtualbox (Win7 as hypervisor) -> database creation takes about 3,5 minutes
3) Both Servers on the gentoo machine (internal disk 7200rpm) -> database creation takes about 32 minutes
4) Both Servers on the gentoo machine (usb3 disk 7200rpm) -> database creation takes about 5 minutes

Various tests done on the 3rd configuration:
importing dump on the mysql server locally -> normal speed
importing dump remotely from the zendserver -> normal speed
network copy from zendserver to mysqlserver -> normal speed
iperf between host and mysql -> nearly 1GB
iperf between zendserver and mysqlserver -> nearly 1GB
local disk performance tests an the mysql shows disk write speeds between 40 and 70MB/s

Has anybody any idea why running from the internal disk is so incredibly slow ?

[sk3l]

Presumably you are using the virtio block drivers? If not, that would the first thing I'd put in place. If yes, are you using the default I/O tuning parameters, or writethrough caching, etc? The I/O tuning parameters can have an impact on disk performance, AFAIK.

Also, what image format are you using, and are you using a file on the host system, LVM or a dedicated block device for the VM image?

If the DB creation is fast with VirtualBox, and using a USB drive, then odds are there is something you ~should~ be able to tune with your virtual disk properties on the Gentoo host to get the timing down closer to 3 minutes.

If you can figure out what is bottlenecking your virtual disk activity on the Gentoo host, I'd definitely recommend a) upgrading your Gentoo's disk device and/or b) use a dedicated block device to host the VM images, if possible. That should give you even more I/O performance.

[drizzt]

Thanks for your reply,
I do use all virtio-drivers(net, block) and the virtual-discs are qcow2 files. I tried all tunings available in kvm(cache set to none, preallocation of metadata, elevator to noop).

[sk3l]

You could try switching to a different image format to see what that yields. I use raw myself, after moving away from qcow2. However, that doesn't seem like it would make that large of a difference.

The only other thing is perhaps there is some parameter to tune mysql, letting it know that the underlying system is virtualized.

I run KVM on both my Gentoo desktop & OpenSUSE laptop, with a Windows 7 guest for my company's software dev. Obviously there is I/O latency, but not what you're seeing, an order of magnitude slower. The key detail seems to be (if I'm understanding this right) that if you use an external USB HDD attached to the guests, then the DB build is like 5 minutes, which is much better than 32. That tells me it is definitely something related to the virtual disk config & not the VM as a whole.

What about trying SCSI or SATA virtual disk interfaces instead of virtio to see if that makes any difference? Clearly it shouldn't be faster than virtio, but one never knows.

[drizzt]

I tried all those options. The strange thing is that the same virtual machines are fast if the virtual disc files are stored on usb and slow if stored on the internal SATA-disc drive.

[sk3l]

Maybe try a different guest, other than Cent OS, and see if that changes anything? It seems like you've tried everything already, so I'll bet you've probably done that as well.

I'm stumped. Sorry I couldn't be more help, but VM is merely a hobby for me, and a way to reconcile Linux environments w/ my job's Windows dev requirements.

[drizzt]

No problem. I appreciate any input.

Maybe someone else got any idea what's going on here ?

[miket]

Sometimes for testing with MSIE I fire up one VM image or another I have with Windows. I use qcow2 for the disk images, but run the VM's in snapshot mode. This suits my purposes because it lets me discard any changes made in the session. The other huge benefit, though, is that it runs hugely faster that way.

All this points out to me that qcow2, for all its nice attributes, imposes quite a speed penalty for writes. Actually, though I suspect it may bog down reads also, I have no actual numbers for you. All the same, I would suspect you'd get quite the performance boost if you switched to raw format, or as someone suggested, a physical or LVM volume.

[drizzt]

Hi,
yes I read all those things. I would agree if the same machine running from usb disc would also be slow but that isn't the case.

[Ant P.]

Are the SATA and USB disks both using the same filesystem? Is the internal one full and causing the qemu images to badly fragment maybe? (try /usr/sbin/filefrag)

[drizzt]

Both are ext4 filesystems 1TB, containing only the disk images

[vaxbrat]

When you switch out hypervisors, are they on the same drive on the same machine? You aren't clear about that. I'm wondering if your internal gentoo drive maybe has one or more sectors in the guest image on the verge of being locked out due to errors. What does a "smartctl -a" on that drive show?

FWIW I run raw format on btrfs and have never tried qcow2. btrfs lets me snapshot and rollback the vm, and will also clue me in if it's finding anything wrong from its own checksumming. I do run Centos 6.2 reference vm's on btrfs with virtio, cache and tuning defaulted, compression=lzo and autodefrag turned on.

I only turn off autodefrag for win7 corporate images which run Sophos A/V. That crap intercepts i/o's and writes so much it drives the brtrfs workers crazy and drags everything down as a result. Different rants for different folk I guess

[drizzt]

I don't switch hypervisors. I have each virtualdisc on internal drive and an copy of it on the usb drive. I only replace the virtualdiscs in the config of the virtualmachine. It is always the same host machine using kvm.

[mike155]

Have you tried to turn ON disk write caching of your host system by setting "cache" to "unsafe"? For this test, your QEMU parameters should contain something like this:

Code: Select all

-drive file="disk.img",index=0,media=disk,cache=unsafe

I'm almost sure your internal disk will be faster than any of the tests you have performed so far...

WARNING: cache=unsafe is dangerous and may corrupt your QCOW image if your host system crashes or if you experience a power loss. Use this setting only for a speed test.

[drizzt]

Yes that's the performance I'm looking for
Just to be clear, the setting itself is not dangerous if the machine is running with an UPS and a RAID-Controller with BBU, correct ?

[mike155]

Nearly all databases frequently emit sync requests (write barriers) when writing data. That's the reason why your system is slow if your database runs in a VM and QEMU is configured without full write caching. If you run QEMU with full write caching, the sync requests are thrown away by the write cache and your system is much faster. VMs just are not optimized for databases. This may change in the future, but right now, I don't recommend to run databases in VMs. Run your database on the host system if you need speed.

If you have a battery backed RAID controller, then I'm a bit surprised of your results. Your battery backed RAID Controller should also be able to neutralize write barriers and you shouldn't see a huge slowdown. Are you sure that caching of your RAID controller works and that caching mode is set to writeback?

Just to be clear, the settings itself is not dangerous if the machine is running with an UPS and a RAID-Controller with BBU, correct ?

Well, I wouldn't run QEMU with "cache=unsafe" even on a machine with UPS and battery backed write cache. It might work, but if anythings goes wrong (think about bugs in QEMU or your OS or even hardware bugs) you might get disk image corruptions. The problem is that you won't even notice that your image is corrupted - and then, after a while (maybe months) you will get all kind of strange errors or even wrong results of your programs. It's just not worth it.

[drizzt]

At the moment I don't use a RAID-Controller. Since this is just a test system I'm okay with the potential risk of corrupting my virtual test machines.
If the database's write barriers is the culprit, then why is the USB disc so fast ? Is this disc always write cached ?

[mike155]

At the moment I don't use a RAID-Controller.

OK, then your test results make sense. If you run tests on your production server, please also post results for the configurations below:

• server with battery backed RAID controller and QEMU with "cache=unsafe"
• server with battery backed RAID controller and QEMU with "cache=writeback".

I guess that both variants will be fast. If that's true you should use "cache=writeback" on your production server.

why is the USB disc so fast

I have no idea. Maybe some kind of caching is involved.

BTW: before I install a database on a new system, I run a small test program: it opens a file, writes a byte and emits fsync. The program does this in a loop and counts (and prints) the number of cycles within 10 seconds.
The program returns the rpm of the drive divided by 6 if it writes to a file on a rotating disk. If the result is lower, something is wrong in the data path. If the result is higher, some kind of caching is involved.