Xen ebuilds - discussion and directions

[trad511]

The idea behind this topic is to begin or continue discussions around where to go with the Xen ebuilds in Gentoo. Currently our ebuilds are for only the mainline Xen release, but discussions of supporting unstable, or Red Hat patches come up again and again in both bug reports and the forums - chiefly when a version bump is requested. The bug that spawned this discussion is [bug=]179412[/bug] which is a version bump request from xen-3.0.4 to xen-3.1.0.

As a brief aside, I am not a Gentoo dev. I am simply a "power" user of Gentoo Xen on production servers since version 2 and on both production servers and everyday on my work laptop since version 3. I will say that the Gentoo Xen devs have done a great job. The Xen development line is historically far behind the mainline kernel and I know the devs get alot of pressure to support Xen for kernel versions that the main Xen has barely, if ever, touched. Additionally, Xen requires a pretty hefty skill set to be a dev, and problems that arise due to bad builds can and have destroyed data. As a final point, the Xen mailing lists leave something to be desired with respect to organization - it is very difficult to see what the direction of Xen is, who the players are or to find if your question has been answered in the past. Please don't criticize or think I'm criticizing the the Gentoo Xen devs. I simply see this support topic come up over and over and would like the community to have an open discussion.

A request was made in the current version bump bug [bug=]179412[/bug] to create an Xen ebuild which leverage the Fedora 2.6.20 patches or the (I hear) very beta 2.6.21 OpenSuSE patches. I personally am beginning to get into hangup-ville since my ipw3945 drivers have changed significantly from my currently supported 2.6.16 xen kernel: the ipw drivers/ebuild now are compatible with and require the ieee80211 stack in kernel 2.6.18 and later. My current solution: mask out ebuild updates. I see this starting to happen now that gentoo main is getting further and further away from the supported Xen kernels. As a consequence I'd love to see more recent kernels supported.

However, I see support of "third party" patching away from the Xen mainline as a risk, both from a data standpoint and from an ebuild maintenance standpoint. During the migration from Xen 3.0 to 3.0.2 (which then was eventually released as 3.0.4_p1) someone released ebuilds against the 2.6.18 kernel (sounds soooo old now doesn't it?) for the 2.6.16 xen patches. This worked for several people, but one person used backend exported hardware (a SATA drive I believe) to a domU and really borked his data. For a dev this isn't realistic for several reasons. First, if these patches were designed for 2.6.16 and used on 2.6.18, any reports upstream to Xen will simply be discarded because it's not supported upstream. Second, no dev wants to be responsible for maintaining an ebuild that seriously trashes data.

A similar thing may happen if we decide to support Fedora Xen patches. The beauty of the open source community is that bugs are reported upwards towards the source. But if we report FC-2.6.20 bugs to the Gentoo Xen devs where do they try to get them resolved? Fedora won't support Gentoo's use of it's patches...

All this puts us in a difficult situation that has, as I see it, fractionalized the Gentoo Xen community. The outcome of this same discussion during the 3.0 to 3.0.2/4 version bump request was the creation of several external, non-mainline, overlays. While there's nothing inherently wrong with overlays, I think in this case that we now have several communities, each allied with the kernel version of Xen supported by the overlay ebuilds. Doesn't really sound so good in the end to me since (that I know of) none of these communities share with each other.

What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds. Obviously this requires some thought and planning with regard to support and communication, but that's what I wanted this forum topic to be about. Comments? Questions? Suggestions?

[smiler.se]

What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds. Obviously this requires some thought and planning with regard to support and communication, but that's what I wanted this forum topic to be about. Comments? Questions? Suggestions?

I see a valid and good point in the problem with maintaining a "bleeding edge" official build of Xen. I think this overlay idea is a good solution if someone is up for really maintaining it.
For me, I need a recent kernel due to driver support/improvements, not because it's new. How does the Xen-people handle this issue? Alot has happend since 2.6.18, especially when it comes to ATA/SATA development.

[roock]

I am also using Gentoo + Xen in a production environment. And I would like to see xen with official patches in portage and one overlay for unstable xen. If i recall correctly, there were nearyl 3 or 4 overlays, when 3.0.4 went stable and 3.0.2 was in portage. So one official overlay with bleeding edge version would be very nice, in particular xen's official kernel is atm very much behind official kernel versions. And of course many thanks to the gentoo devs.. I would like to help more.. but spare time is rare...

[tokka]

What I'd like to suggest is putting mainline Xen (this means release + kernel version) in portage and the creation of an official Xen overlay for "buyer beware", unstable or third party (ie. FC) Xen ebuilds.

That sounds like the way to go to me.

I've got a production server that has been running Xen 2 for ages, it has has no problems and so I've left it alone. Two weeks ago I decided to build a new Xen server, what was in Portage was ancient, gentoo-wiki.com pointed me towards two overlays with 3.03 and 3.04. I tried 3.04 and it seems to be ok bar networking issues covered in this forum - then to my surprise the 3.04 ebuild seemed to get stuffed into Portage without the known issues being addressed...

So currently a Gentoo newcomer to Xen using the ebuilds in Portage will end up with a real headache trying to get networking working - whereas other major distributions have pretty slick Xen implementations - before the 3.1 release I found myself seriously considering using a CentOS domain 0 with Gentoo VM's:(

I think a start would be to have working ebuilds in Portage, and then yes an official overlay, there are already people posting in this forum having issues with the mechanics of installing an ebuild from that bug thread.

[CpuID]

Im all for official releases in portage, and bleeding edge/experimental stuff in an overlay. I just gave the OpenSUSE patches a test against current gentoo-sources-2.6.21*, and its rather nasty, lets just say that much The xen3-patch-2.6.21 has no chance of applying in its current state, not to mention the mach-xen asm dirs that dont exist for some archs, im considering giving the redhat patches a try next against 2.6.20 or something... Let me know if anyone starts something to collaborate on progress in different areas etc, maybe a xen overlay on overlays.gentoo.org or something? marineam might like to push his progress over that way maybe... either that or make more use of the VPS overlay I guess...

[sdrik]

I've just published my overlay containing my 2.6.20 Xen kernel, based on Fedora patches :
http://cedric.gabriello.fr/gentoo/layman.xml
http://cedric.gabriello.fr/gentoo/sdrik-xen.tar.bz2

Only DomU/i386 has been tested for now as my Dom0 is an Ubuntu system, but I use nearly the same patchset for my Ubuntu packages (http://cedric.gabriello.fr/ubuntu)

[tomekki]

Hi,
I just gave it a try and compiled a dom0 kernel on my x86_64 architecture.
It does not work:

Code: Select all

linux-2.6.20.12-xen # make
  
  ...

  CC      drivers/xen/util.o
  LD      drivers/xen/privcmd/built-in.o
  CC      drivers/xen/xenbus/xenbus_xs.o
  CC      drivers/xen/xenbus/xenbus_probe.o
  CC      drivers/xen/xenbus/xenbus_backend_client.o
  CC      drivers/xen/xenbus/xenbus_probe_backend.o
  CC      drivers/xen/xenbus/xenbus_dev.o
  LD      drivers/xen/xenbus/xenbus_be.o
  LD      drivers/xen/xenbus/built-in.o
  LD      drivers/xen/built-in.o
  LD      drivers/built-in.o
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
arch/x86_64/kernel/built-in.o: In function `intel_bugs':
early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'
make: *** [.tmp_vmlinux1] Error 1

As told, it was not tested completely ...

[CpuID]

hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64 anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

[zoltak]

hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64 anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

You will need the SMP fix if you want your virtual machines to access more then 1 VCPU's. Thats all I found so far.

[sdrik]

hey sdrik, I just used the linux-2.6-xen.patch out of the 2.6.20 2925 srpm for fedora core 7, no other patches. patched against gentoo-sources 2.6.20-r8 and it built and booted first time with x86_64 anything else your patchset has that I dont know about? only thing left to sort out is my missing phys_to_machine symbol for nvidia drivers, which shouldnt be too hard

patch 01 is the core xen patch
patches 10-13 are things in fedora's patchset which touches xen bits and looked interesting
patches 20-25 contain a fix for a nasty problem which triggers when using dhcp beetween DomU and Dom0
patches 50-51 are random fixes necessary to build (but may be needed only with some specific configs)
patch 90 is needed to build with binutils-2.16.1 (the problem reported in bug #179412 comment #22)

I've only tried with vanillia sources, so maybe some patches are not needed anymore with gentoo-sources

[sdrik]

Hi,
I just gave it a try and compiled a dom0 kernel on my x86_64 architecture.
It does not work:

Code: Select all

linux-2.6.20.12-xen # make
  
  ...

  CC      drivers/xen/util.o
  LD      drivers/xen/privcmd/built-in.o
  CC      drivers/xen/xenbus/xenbus_xs.o
  CC      drivers/xen/xenbus/xenbus_probe.o
  CC      drivers/xen/xenbus/xenbus_backend_client.o
  CC      drivers/xen/xenbus/xenbus_probe_backend.o
  CC      drivers/xen/xenbus/xenbus_dev.o
  LD      drivers/xen/xenbus/xenbus_be.o
  LD      drivers/xen/xenbus/built-in.o
  LD      drivers/xen/built-in.o
  LD      drivers/built-in.o
  GEN     .version
  CHK     include/linux/compile.h
  UPD     include/linux/compile.h
  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
arch/x86_64/kernel/built-in.o: In function `intel_bugs':
early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'
make: *** [.tmp_vmlinux1] Error 1

As told, it was not tested completely ...

I cannot reproduce your problem here, it must be specific to your .config
Can you send me your .config privately ? I don't think this topic is the good place to further debug my overlay (where can this take place ? I'm new to Gentoo hacking)

[zoltak]

Is anyone having problems using images via

Code: Select all

tap:aio:/

I get an error:

Code: Select all

XENBUS: Timeout connecting to device: device/vbd/51712 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51715 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51714 (state 3)

File method works fine

Code: Select all

file:/

The images boot on a centos dom0 running 2.6.18

[CpuID]

theres an updated overlay at http://www.nightsys.net/xen-3.1.0-2.6.2 ... ay.tar.bz2 , which includes sdriks patches (thanks sdrik, they all look pretty good ) for the kernel, mixed with gentoo-sources genpatches 10 (had to exclude the 2.6.20.y patches from genpatches to avoid duplication but otherwise they work fine), also included is xen-3.1 and xen-tools-3.1 (thanks gentoo@soit.de), and the nvidia-drivers-9755-r2 patched to work with xen (thanks Guy Baconniere). hope everyone likes, im typing this from a dom0 using the above, with the nvidia drivers working fine I attempted to boot a winxp pro hvm last night, it got into setup fine, did the first restart, but hung right on the end of installing devices..., but i think thats most likely a driver issue with xp setup than xen if anything (it hung halfway through installing devices at first, turned off pae and it got to almost the end)...

[zoltak]

Is anyone having problems using images via

Code: Select all

tap:aio:/

I get an error:

Code: Select all

XENBUS: Timeout connecting to device: device/vbd/51712 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51715 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51714 (state 3)

File method works fine

Code: Select all

file:/

The images boot on a centos dom0 running 2.6.18

I don't need kernel 2.6.20 on Dom0 so I installed 2.6.18 and the problem above goes away. I still use the patched 2.6.20 for domU's

Has anyone else overcome this problem running 2.6.20 on Dom0 with tap:aio devices?

[sdrik]

Is anyone having problems using images via

Code: Select all

tap:aio:/

I get an error:

Code: Select all

XENBUS: Timeout connecting to device: device/vbd/51712 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51715 (state 3)
XENBUS: Timeout connecting to device: device/vbd/51714 (state 3)

File method works fine

Code: Select all

file:/

The images boot on a centos dom0 running 2.6.18

I don't need kernel 2.6.20 on Dom0 so I installed 2.6.18 and the problem above goes away. I still use the patched 2.6.20 for domU's

Has anyone else overcome this problem running 2.6.20 on Dom0 with tap:aio devices?

Kernels built from upstream xen tarballs (this his the case for the 2.6.18 contained in my overlay) contain a patch (blktap-aio-16_03_06.patch) which has been vetoed upstream. Fedora decided not to include it so they implented another patch against xen tools as a workaround (xen-blktap-no-aio-epoll.patch)
(http://lists.xensource.com/archives/htm ... 00331.html)

I will work on integrating both xen-blktap-no-aio-epoll.patch and blktap-aio-16_03_06.patch in my xen-sources and (upcoming) xen ebuilds and let the user select between the two methods by a USE flag.

[zoltak]

Could someone send me a working .config for 2.6.20 privatley or post a link.

Thanks

[tomekki]

Hi,
I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.

Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:

Code: Select all

arch/x86_64/kernel/built-in.o: In function `intel_bugs':
early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'
make: *** [.tmp_vmlinux1] Error 1 

The option which lets the compilation fail is:

Code: Select all

Bus options (PCI etc.)  ---> 
[*] PCI support

At least I can make my very first steps with Xen
However, I would like to have PCI available in my future domU machine.

Any Idea what might went wrong with the the PCI support?

[zoltak]

Is it possible to specify 2 ip's on eth0 in dom0. When I do this it fails to bring eth0 up? 1 works fine?

Any suggestions?

[Mescalito]

Hey,

I made an ebuild and some layman stuff for it.

emerge -uv layman
layman -kfo http://thestonertree.com/layman.txt -a mescalito
emerge -av xen xen-tools xen-sources

[CpuID]

Hi,
I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.

Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:

Code: Select all

arch/x86_64/kernel/built-in.o: In function `intel_bugs':
early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'
make: *** [.tmp_vmlinux1] Error 1 

The option which lets the compilation fail is:

Code: Select all

Bus options (PCI etc.)  ---> 
[*] PCI support

At least I can make my very first steps with Xen
However, I would like to have PCI available in my future domU machine.

Any Idea what might went wrong with the the PCI support?

To be honest, I just hashed this quirk out, as I dont have the hardware it relates to (according to the comments in the src) But yea, from what I can tell the quirk exists in i386 but wasnt replicated to amd64, but im reluctant to do so as im not experienced enough to know if its safe/what the implications are in the case you do have that hardware

[Mescalito]

Hi,
I still can't compile a (domU) Kernel. I tried sdrik's and CpuID's patched version without success.

Fortunately, I was able to isolate the kernel option responsible for my previous error. Which was:

Code: Select all

arch/x86_64/kernel/built-in.o: In function `intel_bugs':
early-quirks.c:(.text+0xaf97): undefined reference to `quirk_intel_irqbalance'
make: *** [.tmp_vmlinux1] Error 1 

The option which lets the compilation fail is:

Code: Select all

Bus options (PCI etc.)  ---> 
[*] PCI support

At least I can make my very first steps with Xen :)
However, I would like to have PCI available in my future domU machine.

Any Idea what might went wrong with the the PCI support?

To be honest, I just hashed this quirk out, as I dont have the hardware it relates to (according to the comments in the src) :) But yea, from what I can tell the quirk exists in i386 but wasnt replicated to amd64, but im reluctant to do so as im not experienced enough to know if its safe/what the implications are in the case you do have that hardware :P

The problem with disabling [*] PCI support is you disable support for other kernel options, and to be honest it's more likely that one of these dependant options is to blame than PCI support itself.

I put the 2.6.20 kernel patches in my repository and am running WinXP inside it now with nvidia drivers on linux. Everything seems to work amazingly.

[tomekki]

Hi Mescalito,
i just tried your xen overlay and this time the compilation of a kernel 2.6.18 went smoothly.
Thanks, now I can start to use pci devices in domU.

You mentioned that you checked in kernel 2.6.20 patches in your repository.
I could find those under 'svn://thestonertree.com/mescalito/sys-kernel/xen-sources'. Could you give me a hint where to find the patch?

I agree, I would be also afraid that missing PCI support would bring some unexpected side effects.

Btw. CpuID, I hope that Santa is going to give you a nice 64 bit machine

Greetings

[CpuID]

tomekki, I already have 3 amd64 boxen a c2d t7200 mediapc (1gb ram, 32bit gentoo only atm though), c2d e6600 home desktop (with 4gb ram, 64bit gentoo), and an opteron 148 work desktop (with 4gb ram, 64bit gentoo) hehe.

[Mescalito]

Hi Mescalito,
i just tried your xen overlay and this time the compilation of a kernel 2.6.18 went smoothly.
Thanks, now I can start to use pci devices in domU. :)

You mentioned that you checked in kernel 2.6.20 patches in your repository.
I could find those under 'svn://thestonertree.com/mescalito/sys-kernel/xen-sources'. Could you give me a hint where to find the patch?

I agree, I would be also afraid that missing PCI support would bring some unexpected side effects.

Btw. CpuID, I hope that Santa is going to give you a nice 64 bit machine ;)

Greetings

Do a `svn update' or use `layman -s mescalito'

Cheers, James

[tomekki]

At the risk of doing wrong ...

I just found an ebuild for kernel 2.6.18:

Code: Select all

svn co svn://thestonertree.com/mescalito
[...]
A    mescalito/app-emulation/xen/xen-3.1.0.ebuild
A    mescalito/sys-kernel
A    mescalito/sys-kernel/xen-sources
A    mescalito/sys-kernel/xen-sources/metadata.xml
A    mescalito/sys-kernel/xen-sources/files
A    mescalito/sys-kernel/xen-sources/files/digest-xen-sources-2.6.18
A    mescalito/sys-kernel/xen-sources/files/patch-2.6.18_to_xen-3.1.0.bz2
A    mescalito/sys-kernel/xen-sources/Manifest
A    mescalito/sys-kernel/xen-sources/ChangeLog
A    mescalito/sys-kernel/xen-sources/xen-sources-2.6.18.ebuild
Checked out revision 5.

Might it be that your patches for kernel 2.6.20 are not accessible for anonymous svn users?

But, no stress ... I live already quite well with the 2.6.18 kernel

Greetings, Tomek