Xen 2.6.25 dom0 kernel ebuild

[andylyon]

Hmm, I've just noticed that there is a new suse rpm kernel-source-2.6.27.5-2.1.src.rpm, when I first downloaded the suse kernel source rpm I got it from http://download.opensuse.org/factory/re ... /suse/src/ but when they updated to .4-10.2 the kernel-source-2.6.27.4-10.2.src.rpm was never available there (other kernel rpms were there, but no source), but I was able to locate it in http://download.opensuse.org/repositori ... ctory/src/ instead, I assumed that the latter location was the correct place to get the latest kernel source rpm in future and I stopped checking the former, today I happened to check both folders and noticed that this time the latest version is in http://download.opensuse.org/factory/re ... /suse/src/

Anyway, I've had a quick look and as usual there are some new xen patches, and quite a lot of changes to the existing ones, including those that I had to modify to apply cleanly to vanilla 2.6.27, this release coincides nicely with gentoo-sources-2.6.27-r3.ebuild which was released yesterday, tomorrow I will update the ebuild with genpatches 5 and kernel-source-2.6.27.5-2.1.src.rpm

I will test before release and include a list of my tests.

Andy

[drescherjm]

Can you put your .config somwhere I can get it? id rather you didnt post it into the forum as it takes up so much space, but i need the config to look into this problem.

http://home.comcast.net/~drescherjm/domU.config

Just to refresh. The following is the error when building the domU kernel:

Code: Select all

  CC      arch/x86/ia32/ia32_aout.o
  LD      arch/x86/ia32/built-in.o
  CC      kernel/sched.o
  CC      kernel/fork.o
  CC      kernel/exec_domain.o
  CC      kernel/panic.o
  CC      kernel/printk.o
  CC      kernel/cpu.o
/usr/src/linux-2.6.27-xen-r2/kernel/cpu.c: In function 'disable_nonboot_cpus':
/usr/src/linux-2.6.27-xen-r2/kernel/cpu.c:410: error: implicit declaration of function '_cpu_down'
make[3]: *** [kernel/cpu.o] Error 1
make[2]: *** [kernel] Error 2
make[1]: *** [sub-make] Error 2
make: *** [all] Error 2

[BlackEye]

I've the same error btw

edit: I was able to compile the kernel by adding

Code: Select all

CONFIG_HOTPLUG_CPU=y

to my config

[uprooter]

did you pass console=xvc0 to the domU kernel ? I had the exact same problem until I informed the kernel of the correct console to use.

Andy

Good news:
xencons=tty did the trick.
I also had that _cpu_down issue, i started with a fresh config and it was solved. i guess something is messed it up.

Bad news:
xen 3.3 screwed up my firewall in domU because of this bug: http://bugzilla.xensource.com/bugzilla/ ... gi?id=1340
something is wrong with the PCI assignment and it gives me the "must be co-assigned to the same guest" error.
I spent too much time googling on this XEN crap...

[bbgermany]

Bad news:
xen 3.3 screwed up my firewall in domU because of this bug: http://bugzilla.xensource.com/bugzilla/ ... gi?id=1340
something is wrong with the PCI assignment and it gives me the "must be co-assigned to the same guest" error.
I spent too much time googling on this XEN crap...

can you post your output of /proc/interrupts. maybe the devices are sharing the same irq.

bb

[drescherjm]

I've the same error btw

edit: I was able to compile the kernel by adding

Code: Select all

CONFIG_HOTPLUG_CPU=y

to my config

Thanks. That fixed my issue with the domU config.

[uprooter]

can you post your output of /proc/interrupts. maybe the devices are sharing the same irq.

bb

Code: Select all

           CPU0       CPU1
  1:          0          0  Phys-irq-level     i8042
  4:        902          0  Phys-irq-level     serial
  8:          0          0  Phys-irq-level     rtc0
  9:          0          0  Phys-irq-level     acpi
 14:         55          0  Phys-irq-level     ata_piix
 15:          0          0  Phys-irq-level     ata_piix
 16:    1386017          0  Phys-irq-level     uhci_hcd:usb4, eth1
 17:   29839253          0  Phys-irq-level     eth0
 18:    7814313          0  Phys-irq-level     uhci_hcd:usb3, eth2
 19:    5271536          0  Phys-irq-level     ahci, uhci_hcd:usb2
 23:         87          0  Phys-irq-level     uhci_hcd:usb1, ehci_hcd:usb5
512:   30060425          0  Dynamic-irq-level     timer0
513:     731843          0  Dynamic-irq-level     resched0
514:          0          0  Dynamic-irq-level     callfunc0
515:       3326          0  Dynamic-irq-level     call1func0
516:      89237          0  Dynamic-irq-level     spinlock0
517:          0    1947565  Dynamic-irq-level     resched1
518:          0          0  Dynamic-irq-level     callfunc1
519:          0       3854  Dynamic-irq-level     call1func1
520:          0    1813248  Dynamic-irq-level     spinlock1
521:          0   86410560  Dynamic-irq-level     timer1
522:      29519          0  Dynamic-irq-level     xenbus
523:          0          0  Dynamic-irq-level     console
524:     606214          0  Dynamic-irq-level     blkif-backend
525:          2          0  Dynamic-irq-level     blkif-backend
526:      31718          0  Dynamic-irq-level     blkif-backend
527:    3321207          0  Dynamic-irq-level     vif13.0
528:   12323167          0  Dynamic-irq-level     vif13.1
529:    3716448          0  Dynamic-irq-level     vif13.2
530:     576489          0  Dynamic-irq-level     blkif-backend
531:      32953          0  Dynamic-irq-level     blkif-backend
532:   11355885          0  Dynamic-irq-level     vif14.0
533:      51426          0  Dynamic-irq-level     blkif-backend
534:         11          0  Dynamic-irq-level     blkif-backend
535:      51044          0  Dynamic-irq-level     vif16.0
536:     847245          0  Dynamic-irq-level     blkif-backend
537:      27734          0  Dynamic-irq-level     blkif-backend
538:    3640924          0  Dynamic-irq-level     vif17.0
539:     305655          0  Dynamic-irq-level     blkif-backend
540:      14529          0  Dynamic-irq-level     vif18.0
NMI:          0          0   Non-maskable interrupts
RES:     731843    1947565   Rescheduling interrupts
CAL:       3326       3854   function call interrupts

And here's lspci

Code: Select all

00:00.0 Host bridge: Intel Corporation E7230/3000/3010 Memory Controller Hub (rev c0)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)
00:1f.2 SATA controller: Intel Corporation 82801GR/GH (ICH7 Family) SATA AHCI Controller (rev 01)
03:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)
04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI Express (rev 21)
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
0a:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)

as you can see this machine has 3 nics.
the 2 realteks of the same type are used by the dom0 since I couldn't pass one of them to the domU with pciback.hide=(0a:01.0)

[bbgermany]

hmm, seems i need to revive my other xen installation and try this out. i havent updated that system for years now.

bb

[bbgermany]

Ok, got my xen up and running again, but it seems i dont have enough recources bond to each other to check for this problem

i was able to pass my pci tv tuner card to a debian 4 domU:

Code: Select all

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1221     2     r-----   1880.3
debian.4-0                                  30   256     1     -b----     19.2
windowsxp                                   24   512     1     -b----   2382.7
herkules ~ # xm console 30

Debian GNU/Linux 4.0 debian-pristine tty1

debian-pristine login: root
Password:
Linux debian-pristine 2.6.21-xen #7 SMP Tue Oct 28 20:59:22 CET 2008 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
debian-pristine:~# lspci
01:09.0 Multimedia video controller: Internext Compression Inc iTVC16 (CX23416) MPEG-2 Encoder (rev 01)
debian-pristine:~#

host pci devices:

Code: Select all

herkules ~ # lspci -tv
-[0000:00]-+-00.0  nVidia Corporation MCP67 Memory Controller
           +-01.0  nVidia Corporation MCP67 ISA Bridge
           +-01.1  nVidia Corporation MCP67 SMBus
           +-01.2  nVidia Corporation MCP67 Memory Controller
           +-02.0  nVidia Corporation MCP67 OHCI USB 1.1 Controller
           +-02.1  nVidia Corporation MCP67 EHCI USB 2.0 Controller
           +-04.0  nVidia Corporation MCP67 OHCI USB 1.1 Controller
           +-04.1  nVidia Corporation MCP67 EHCI USB 2.0 Controller
           +-06.0  nVidia Corporation MCP67 IDE Controller
           +-07.0  nVidia Corporation MCP67 High Definition Audio
           +-08.0-[0000:01]----09.0  Internext Compression Inc iTVC16 (CX23416) MPEG-2 Encoder
           +-09.0  nVidia Corporation MCP67 AHCI Controller
           +-0a.0  nVidia Corporation MCP67 Ethernet
           +-0b.0-[0000:02]----00.0  nVidia Corporation G71 [GeForce 7300 GS]
           +-0c.0-[0000:03]--
           +-0d.0-[0000:04]--
           +-0e.0-[0000:05]--
           +-0f.0-[0000:06]--
           +-10.0-[0000:07]--
           +-11.0-[0000:08]--
           +-18.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
           +-18.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
           +-18.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
           \-18.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
herkules ~ #

bb

[trikolon]

any progress with the new patches andy?

[andylyon]

any progress with the new patches andy?

Yes, I have just uploaded 2.6.27-r4 ebuild + patches to http://code.google.com/p/gentoo-xen-ker ... loads/list

It is kernel 2.6.27.7 (genpatches base v6) with openSUSE patches from kernel-source-2.6.27.7-6.1.src.rpm, I have tested both the 32 and 64 bit dom0 and pv domU, everything seems to work perfectly.

Andy

[trikolon]

great! many thanks! compiling right now..

how did you get pv domus booting? mine ones dont boot.. and xen 3.3.0 misses pygrub, doesnt it?

[andylyon]

great! many thanks! compiling right now..

how did you get pv domus booting? mine ones don't boot.. and xen 3.3.0 misses pygrub, doesnt it?

I don't use pvgrub, I load the kernel in the config file:

kernel = "/root/gentoovm/vmlinuz-2.6.27-xen-domU"
memory = 256
name = "gentoovm"
vif = [ 'bridge=eth0' ]
disk = ['phy:/dev/raptor/gentoovm,sda1,w']
root = "/dev/sda1 ro"
extra = "gentoo=nodevfs console=xvc0"
vcpus=2
vnc=1


Andy

[Aonoa]

@andylyon:

Have you tried patching vanilla 2.6.28 sources with Xen? If I wanted to do that, would you have any pointers?
I want to try current ext4 code on a domU. I know there is ext4dev support in 2.6.27, but that's not "stable" ext4 as found in 2.6.28.

Thanks for your work.

[andylyon]

@andylyon:

Have you tried patching vanilla 2.6.28 sources with Xen? If I wanted to do that, would you have any pointers?
I want to try current ext4 code on a domU. I know there is ext4dev support in 2.6.27, but that's not "stable" ext4 as found in 2.6.28.

Thanks for your work.

You don't have to use my ebuild for domU, the mainline kernel has pv_ops domU support so you should be able to use vanilla 2.6.28-rc7 pv_ops dom0 is still a work in progress but domU should be fairly stable. Devices have different names and you need to choose different config options, have a look at http://wiki.xensource.com/xenwiki/XenParavirtOps for instructions.

I noticed today that openSUSE has a new kernel, when I get time I will make 2.6.27-r5 ebuild which will use genpatches 7 for 2.6.27.8 and kernel-source-2.6.27.8-1.1.src.rpm for Xen patches, .8 includes a lot of ext4 fixes so perhaps it might have what you need?

It would be a massive amount of work to get the patches to apply to 2.6.28, when I make the ebuild I have to modify some of the Xen patches because the openSUSE kernel applies 800+ other patches before patches.xen and I do not include them, there are always a few Xen patches that fail and have to be amended or I have to pull patches from other sources, I would not attempt to rebase the Xen patches to a different major release.

Andy

[Aonoa]

You don't have to use my ebuild for domU, the mainline kernel has pv_ops domU support so you should be able to use vanilla 2.6.28-rc7 pv_ops dom0 is still a work in progress but domU should be fairly stable. Devices have different names and you need to choose different config options, have a look at http://wiki.xensource.com/xenwiki/XenParavirtOps for instructions.

Aha, I did not know that. This is my first time setting up a Xen box and I have not had that much time to read up on the details.
Since the Gentoo docs on how to install Xen used xen-sources for both the dom0 and domU, I thought that was necessary.

I noticed today that openSUSE has a new kernel, when I get time I will make 2.6.27-r5 ebuild which will use genpatches 7 for 2.6.27.8 and kernel-source-2.6.27.8-1.1.src.rpm for Xen patches, .8 includes a lot of ext4 fixes so perhaps it might have what you need?

Yes, that sounds awesome. I will be looking forward to testing the results of your work on this.

Thanks for your quick reply.

[bbgermany]

Me too, since i have several issues booting the dom0 kernel on my Intel Atom system. System crashes shortly after loading 2.6.27-r2 with some xen errors which i wasnt able to copy and paste up to now.

bb

[Aonoa]

@andylyon:

I'm having trouble booting your sources as dom0 too.
I get an error about a kernel bug, referring to hypervisor and SMP.
If I get the same error on my next try without SMP support, I'll write it down.

[_markd]

I am compiling 2.6.27-r4 on a 32bit (pentium4) and a 64bit (xeon 5150) machine right now.
I got two compile issues:
On 32bit: It seems like it only compiles with CONFIG_CC_OPTIMIZE_FOR_SIZE=y. Otherwise I geht this error:

Code: Select all

CC      arch/x86/mm/fault.o
{standard input}: Assembler messages:
{standard input}:1507: Error: bad register name `%sil'
make[1]: *** [arch/x86/mm/fault.o] Error 1
make: *** [arch/x86/mm] Error 2
make: *** Waiting for unfinished jobs....

On 64bit: Compilation fails when I don't set CONFIG_IA32_EMULATION. Error is:

Code: Select all

 
CC      arch/x86/kernel/cpu/common_64.o
arch/x86/kernel/cpu/common_64-xen.c: In function 'syscall_init':
arch/x86/kernel/cpu/common_64-xen.c:610: error: variable 'cstar' has initializer but incomplete type
arch/x86/kernel/cpu/common_64-xen.c:611: error: unknown field 'type' specified in initializer
arch/x86/kernel/cpu/common_64-xen.c:611: error: 'CALLBACKTYPE_syscall32' undeclared (first use in this function)
arch/x86/kernel/cpu/common_64-xen.c:611: error: (Each undeclared identifier is reported only once
arch/x86/kernel/cpu/common_64-xen.c:611: error: for each function it appears in.)
arch/x86/kernel/cpu/common_64-xen.c:611: warning: excess elements in struct initializer
arch/x86/kernel/cpu/common_64-xen.c:611: warning: (near initialization for 'cstar')
arch/x86/kernel/cpu/common_64-xen.c:612: error: unknown field 'address' specified in initializer
arch/x86/kernel/cpu/common_64-xen.c:613: warning: excess elements in struct initializer
arch/x86/kernel/cpu/common_64-xen.c:613: warning: (near initialization for 'cstar')
arch/x86/kernel/cpu/common_64-xen.c:610: error: storage size of 'cstar' isn't known
arch/x86/kernel/cpu/common_64-xen.c:615: error: 'CALLBACKOP_register' undeclared (first use in this function)
arch/x86/kernel/cpu/common_64-xen.c:616: error: 'KERN_WARN' undeclared (first use in this function)
arch/x86/kernel/cpu/common_64-xen.c:616: error: expected ')' before string constant
arch/x86/kernel/cpu/common_64-xen.c:610: warning: unused variable 'cstar'
make[2]: *** [arch/x86/kernel/cpu/common_64.o] Error 1
make[1]: *** [arch/x86/kernel/cpu] Error 2

I use the gentoo stable gcc 4.1.2 on both systems.
I can finish compilation if I set those options. On the 32bit system, this kernel won't boot. I don't get an error message - the system reboots right after the xen init is finished.
Can't boot the kernel on the 64bit machine since it is a production system.

[Aonoa]

Well, I ended up on a xen-sources 2.6.18-r12 for my dom0 which boots perfectly. I'm trying to create a domU now, based on vanilla-sources 2.6.28-rc7, but it will not load. I get this error when running xm create config -c:

Code: Select all

Using config file "./config".
Error: (2, 'Invalid kernel', 'elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images')

Any ideas?

EDIT: I solved it. I must be blind, I needed to select Xen guest support in the paravirtualization kernel option.

[Btoo]

I have been running Andy's 2.6.27-r4 Dom0 kernel for a few weeks. I can happily say no problems from the kernel, I think.

I can not seem to get usb to pass through to an XP HVm DomU. My fix for that is a pci usb card for the XP DomU, which I will hide from the DomO if necessary.
I could not get the DVD drive to be recognized until I used 'phy:/dev/sr0,hdc:cdrom,r' in the domu config file.

Another issue is that sometimes on startup there is no video or console after grub, like the machine just goes lala before X starts. I am not sure if this is the Xen kernel or the hardware support with any kernel combination. Regardless of which the problem goes away with a reboot. Also, it does not happen every time.


If anyone is experiencing similar problems please post about it.

This is my first attempt at Xen and so far it has been a challenge, but not impossible.
I am looking forward to the 2.6.28 kernel for the GEM support. This is needed for intel video on my Shuttle SG45H7 to run compiz.

Thanks to Andy for bringing us up to date!

[drescherjm]

I managed to get myself in a condition that xend will not start with 2.6.27-xen-r2.

Code: Select all

[2009-01-07 12:26:47 15821] INFO (SrvDaemon:331) Xend Daemon started
[2009-01-07 12:26:47 15821] INFO (SrvDaemon:335) Xend changeset: unavailable.
[2009-01-07 12:26:47 15821] ERROR (SrvDaemon:347) Exception starting xend ((13, 'Permission denied'))
Traceback (most recent call last):
  File "usr/lib64/python2.5/site-packages/xen/xend/server/SrvDaemon.py", line 339, in run
    servers = SrvServer.create()
  File "usr/lib64/python2.5/site-packages/xen/xend/server/SrvServer.py", line 251, in create
    root.putChild('xend', SrvRoot())
  File "usr/lib64/python2.5/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "usr/lib64/python2.5/site-packages/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "usr/lib64/python2.5/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "usr/lib64/python2.5/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "usr/lib64/python2.5/site-packages/xen/xend/XendNode.py", line 752, in instance
    inst = XendNode()
  File "usr/lib64/python2.5/site-packages/xen/xend/XendNode.py", line 87, in __init__
    self.other_config["xen_pagesize"] = self.xeninfo_dict()["xen_pagesize"]
  File "usr/lib64/python2.5/site-packages/xen/xend/XendNode.py", line 741, in xeninfo_dict
    return dict(self.xeninfo())
  File "usr/lib64/python2.5/site-packages/xen/xend/XendNode.py", line 685, in xeninfo
    info['xen_scheduler'] = self.xenschedinfo()
  File "usr/lib64/python2.5/site-packages/xen/xend/XendNode.py", line 675, in xenschedinfo
    sched_id = self.xc.sched_id_get()
Error: (13, 'Permission denied')
[2009-01-07 12:26:47 15820] INFO (SrvDaemon:219) Xend exited with status 1.

I am not sure what caused that. It may have been upgrading python or glibc (2.9.X).

For the time being I had to move my PDC and web server to a openvz server and my postfix server is down so no more email notifications when a server goes down or when my backup system wants me to insert a new tape in the autochanger...

[fr3akX]

Could not compile 2.6.27-xen-r5.

Code: Select all

# uname -a
Linux bart 2.6.21-xen #7 SMP Tue Jan 13 21:28:43 EET 2009 x86_64 Intel(R) Xeon(R) CPU E5405 @ 2.00GHz GenuineIntel GNU/Linux
# gcc-config -l
 [1] x86_64-pc-linux-gnu-3.4.6 *
 [2] x86_64-pc-linux-gnu-3.4.6-hardenednopie
 [3] x86_64-pc-linux-gnu-3.4.6-hardenednopiessp
 [4] x86_64-pc-linux-gnu-3.4.6-hardenednossp
 [5] x86_64-pc-linux-gnu-3.4.6-vanilla

make fails with:

Code: Select all

  GEN     usr/initramfs_data.cpio.gz
  AS      usr/initramfs_data.o
  LD      usr/built-in.o
  CC      arch/x86/kernel/process_64.o
include/asm/mach-xen/asm/hypercall_64.h: In function `xen_load_gs_index':
include/asm/mach-xen/asm/hypercall_64.h:53: warning: asm operand 3 probably doesn't match constraints
include/asm/mach-xen/asm/hypercall_64.h:53: error: impossible constraint in `asm'
include/asm/mach-xen/asm/hypercall_64.h:53: warning: '__res' might be used uninitialized in this function
make[1]: *** [arch/x86/kernel/process_64.o] Error 1
make: *** [arch/x86/kernel] Error 2

kernel config: http://chown.lv/dom0.config

[drescherjm]

It may be that gcc-3.4.6 is too old to compile newer xen kernels.


I have successfully built these patched kernels with 4.2.4 and 4.3.2.

[_markd]

I have 2.6.27-r5 runnung on x86 and x86_64 machines for some time now without issues. I had to play around with the kernel config to make it compile though. I used gcc 4.1.2 for compilation since it's marked stable.