XEN in VMWare: unusable slow

neonknight · Posted: Sun Oct 05, 2008 9:30 am Post subject: XEN in VMWare: unusable slow

I started playing around with XEN as a rather an academic approach to get some experience with it. As all my AMD64 machines are in productive use, I decided to try XEN in a VMWare virtual machine. This workes quite well, except for one thing: It is extremely slow! By slow I mean: It already takes more than one hour to boot dom0!

Let's see the setup:
Hardware: Athlon 64 X2 with 2x2.2Ghz S939 (so no hardware virtualization is supported by the CPU), 2GB RAM, SATA RAID-1 with 2 disks
Host: Ubuntu (yeah, I know...) 32bit, VMWare Server 1.0.5
Guest: Hardened Gentoo 64bit, xen-sources 2.6.18, 768MB RAM, 2 processors, all packages up to date, CFLAG -mno-tls-direct-seg-refs is set and world has been recompiled

The facts: Booting the system using the live-cd or a non-xen kernel runs at expected speed. Booting takes around 1 minute, everything seems to be fine. But when I boot XEN and the XEN-kernel, it rather feels like having a 5Mhz machine. Booting lasts forever (around 1h to bring up the system... bringing up eth0 alone takes more than 10 minutes!). In the meantime, the both CPU-cores of the host are used at 100%. The interesting thing is, that only the userland seems to be slow. The hypervisor and kernel itself seem to boot at equal speed as the non-xen-kernel. Of course I tried all three versions of XEN that are in portage.

Searching the web did not help me, as this is of course a very rare situation. I found some people reporting successful usage of XEN in VMWare. So: what went wrong here? Is using a 64bit dom0/XEN a bad idea? Do I need special kernel-features which i missed to use? I strictly followed the Gentoo XEN manual from http://www.gentoo.org/doc/en/xen-guide.xml. Any idea?

As I mentioned before, this is rather an academic experiment, but I intend to rebuild a server using XEN, so a bit of experience and preparation might be a good idea.

ukavi · Posted: Sun Oct 05, 2008 10:26 am Post subject:

i don't really think this is a problem you can fix. you're virtualizing in two layers, both without hardware support. i'd honestly expect such ridiculous boot times. your cpu isn't too fast anyway either, you don't have that much ram, you have weird -mi-pwn-so-i-use-this-flag flags in your CFLAGS. let's just say i'd recommend you to run the xen thing on a real hardware device.
_________________
English is not my native language, please notify me of any mistakes.

neonknight · Posted: Sun Oct 05, 2008 11:05 am Post subject:

tulcod, did you never look at the XEN-howto? It is explicitely recommended to set this "-mi-pwn-so-i-use-this-flag" USE flag due to a TLS bug in glibc. Take a look at: http://gentoo-wiki.com/Xen#TLS_and_CFLAGS

Maybe someone else with some knowledge of the matter will respond as well...

bbgermany · Posted: Sun Oct 05, 2008 11:12 am Post subject:

Hi,

i dont think you will get this faster, coz every system call from the Xen environment to the hardware needs to go through the virtual machine manager of vmware. since Xen likes to have access to the hardware directly, you wont run a fast system at all in this case.

bb
_________________
Desktop: Ryzen 5 5600G, 32GB, 2TB, RX7600
Notebook: Dell XPS 13 9370, 16GB, 1TB
Server #1: Ryzen 5 Pro 4650G, 64GB, 16.5TB
Server #2: Ryzen 4800H, 32GB, 22TB

gimpel · Posted: Sun Oct 05, 2008 11:33 am Post subject:

I tried something like this once. But that was with a virtual OpenSUSE 10.3 Xen host inside an ESX Cluster with all-in-all 48GHz CPUs and 64GB RAM (8 of it for that OpenSUSE VM). And all 32bit. It was somewhat usable.

Looking at your setup, you are hypervirtualizing (64bit guest on 32bit host) a Gentoo VM on a host machine that doesn't even support AMD-V...
While this doesn't really have to be the reason for the slowness, I wouldn't be wondering about it.
_________________
http://proaudio.tuxfamily.org/wiki - pro-audio software overlay

neonknight · Posted: Sun Oct 05, 2008 2:24 pm Post subject:

It seems indeed to be a problem of the 64bit hypervirtualization. A 32bit eisxen installation runs as smooth as expected. So my next step will be the installation of a 32bit Gentoo with XEN.

neonknight · Posted: Mon Oct 06, 2008 11:31 am Post subject:

OK, 32bit Gentoo with XEN is also running at reasonable speed. So it indeed seems to be a problem of hypervirtualizing XEN.

I will keep on trying, next experiment will be on a 64bit host. But you will have to wait for the results as I don't intend to re-set up my desktop machine within the next few days