Xen 2.6.25 dom0 kernel ebuild

[Mathijs]

True, because until now i see no other solution.
You do?

[Scorpion265]

Did you get your updates into portage?

[Scorpion265]

Also, since we're all xen users... has anyone had any problems upgrading glibc? My issue here: http://forums.gentoo.org/viewtopic-p-61 ... ml#6168649

it only happens in the domUs. dom0 built glibc fine.

[meyerm]

Hi,

first let me say a big thank you for providing newer xen-kernels for Gentoo. I'll wait for a 2.6.32-version (since it is said this will be supported longer and stabilized by Suse and Redhat) before trying to go productive but am toying around with your current kernels at the moment.

Just to add for those who had problems with 2.6.31-xen-r10: I experienced the crashes under load, too. Now with 2.6.31-xen-r11 the problems seem to be gone. I just started a "cat /dev/urandom > /dev/null" on each CPU and will let it run at least over the night to be really sure. For now it already runs for ca. 2 hours stable.

[andylyon]

Hi,

first let me say a big thank you for providing newer xen-kernels for Gentoo. I'll wait for a 2.6.32-version (since it is said this will be supported longer and stabilized by Suse and Redhat) before trying to go productive but am toying around with your current kernels at the moment.

2.6.31 is the kernel used for openSUSE 11.2 so it will be supported for a while longer, but you are right that .32 will be the longest supported as it is used for SLE11SP1, next week I will rebase the 2.6.32 patches and upload a ebuild.

Just to add for those who had problems with 2.6.31-xen-r10: I experienced the crashes under load, too. Now with 2.6.31-xen-r11 the problems seem to be gone. I just started a "cat /dev/urandom > /dev/null" on each CPU and will let it run at least over the night to be really sure. For now it already runs for ca. 2 hours stable.

I've not had any crashes with -r10, but yes -r11 should be very good now, I am running it on several production systems and I've had no problems at all.

Andy

[meyerm]

2.6.31 is the kernel used for openSUSE 11.2 so it will be supported for a while longer, but you are right that .32 will be the longest supported as it is used for SLE11SP1, next week I will rebase the 2.6.32 patches and upload a ebuild.

Great!

Just to add for those who had problems with 2.6.31-xen-r10: I experienced the crashes under load, too. Now with 2.6.31-xen-r11 the problems seem to be gone. I just started a "cat /dev/urandom > /dev/null" on each CPU and will let it run at least over the night to be really sure. For now it already runs for ca. 2 hours stable.

I just reached the 24h runtime. I guess, I'll now switch the default entry in grub from 2.6.29 to the 2.6.31-r11.

[Mathijs]

I just tested 2.6.33 and both dom0 and domU compile fine and work.
Live migration is also working.
I did not see any bad behavior yet, when i do i'll let you know here.

Thank you for the ebuild.

[stof]

Any news on stability of xen-sources-2.6.31-r11?

[meyerm]

Well, I'm not yet very happy. Since this is a development machine I didn't need to install any domU until now. But I won't do that and wait for 2.6.32 because of the follwing problems with xen 3.4.2-r1:

Code: Select all

Mar 14 18:29:34 endor kernel: [1649126.870036] BUG: soft lockup - CPU#1 stuck for 224s! [swapper:0]
Mar 14 18:29:34 endor kernel: [1649126.870036] Modules linked in: ipv6 ipt_LOG xt_limit xt_state iptable_filter ipt_REDIRECT xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_mangle ip_tables x_tables scsi_wait_scan
Mar 14 18:29:34 endor kernel: [1649126.870036] CPU 1:
Mar 14 18:29:34 endor kernel: [1649126.870036] Modules linked in: ipv6 ipt_LOG xt_limit xt_state iptable_filter ipt_REDIRECT xt_tcpudp iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_mangle ip_tables x_tables scsi_wait_scan
Mar 14 18:29:34 endor kernel: [1649126.870036] Pid: 0, comm: swapper Not tainted 2.6.31-xen-r11 #1 Sun Fire V40z
Mar 14 18:29:34 endor kernel: [1649126.870036] RIP: e030:[<ffffffff802063aa>]  [<ffffffff802063aa>] 0xffffffff802063aa
Mar 14 18:29:34 endor kernel: [1649126.870036] RSP: e02b:ffff8803e8483eb8  EFLAGS: 00000246
Mar 14 18:29:34 endor kernel: [1649126.870036] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff802063aa
Mar 14 18:29:34 endor kernel: [1649126.870036] RDX: ffff8803e8483ed0 RSI: 0000000000000000 RDI: 0000000000000001
Mar 14 18:29:34 endor kernel: [1649126.870036] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8803e8498000
Mar 14 18:29:34 endor kernel: [1649126.870036] R10: 86634d5592bb9e06 R11: 0000000000000246 R12: 0000000000000000
Mar 14 18:29:34 endor kernel: [1649126.870036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Mar 14 18:29:34 endor kernel: [1649126.870036] FS:  00007f9bd973e6f0(0000) GS:ffffc90000010000(0000) knlGS:0000000000000000
Mar 14 18:29:34 endor kernel: [1649126.870036] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Mar 14 18:29:34 endor kernel: [1649126.870036] CR2: 00000000006d4ae4 CR3: 00000003e764d000 CR4: 0000000000000660
Mar 14 18:29:34 endor kernel: [1649126.870036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 14 18:29:34 endor kernel: [1649126.870036] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 14 18:29:34 endor kernel: [1649126.870036] Call Trace:
Mar 14 18:29:34 endor kernel: [1649126.870036]  [<ffffffff8020caec>] ? xen_safe_halt+0xb2/0xdd
Mar 14 18:29:34 endor kernel: [1649126.870036]  [<ffffffff80210e3e>] ? xen_idle+0x6e/0xe1
Mar 14 18:29:34 endor kernel: [1649126.870036]  [<ffffffff80208c20>] ? cpu_idle+0xa1/0xdd

I do have a swap-partition on a LVM-VG, but because of the not disposed RAM of 16GB it's still at 0kb usage. All the machine is basically doing is syncing, compiling new packages for itself and logging all those script kiddies / bot net attacks using iptables.

All CPUs are shown in htop with 0 load (all idle). If anybody can use these small traces to search for the problem I'm happy to help with this machine. But the problem did not appear from the beginning on. I guess, as soon as I reboot it, the problem will be gone for some time (which is of course not good enough for a production use, but bad enough for debugging ).

[meyerm]

I just switched to 2.6.32-xen-r1 and this is what I see in the kernel log right after booting:

Code: Select all

[   27.186394] EXT3 FS on md1, internal journal
[   28.156511] Adding 2097144k swap on /dev/mapper/vg_swap-dom0.  Priority:-1 extents:1 across:2097144k 
[   36.602542] ip_tables: (C) 2000-2006 Netfilter Core Team
[   36.670856] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   36.671096] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
[   36.671098] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
[   36.671100] sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
[   37.859260] tg3 0000:02:02.0: firmware: using built-in firmware tigon/tg3_tso.bin
[   39.728844] tg3: eth0: Link is up at 100 Mbps, full duplex.
[   39.728851] tg3: eth0: Flow control is off for TX and off for RX.
[   42.599496] tg3 0000:02:03.0: firmware: using built-in firmware tigon/tg3_tso.bin
[   44.490092] tg3: eth1: Link is up at 100 Mbps, full duplex.
[   44.490099] tg3: eth1: Flow control is off for TX and off for RX.
[  142.789967] BUG: soft lockup - CPU#1 stuck for 114s! [swapper:0]
[  142.789967] Modules linked in: iptable_mangle ipt_REDIRECT iptable_nat nf_nat xt_tcpudp ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables scsi_wait_scan
[  142.789967] CPU 1:
[  142.789967] Modules linked in: iptable_mangle ipt_REDIRECT iptable_nat nf_nat xt_tcpudp ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables scsi_wait_scan
[  142.789967] Pid: 0, comm: swapper Not tainted 2.6.32-xen-r1 #1 Sun Fire V40z
[  142.789967] RIP: e030:[<ffffffff802063aa>]  [<ffffffff802063aa>] 0xffffffff802063aa
[  142.789967] RSP: e02b:ffff8803e8483ee8  EFLAGS: 00000246
[  142.789967] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff802063aa
[  142.789967] RDX: ffff8803e8483f00 RSI: 0000000000000000 RDI: 0000000000000001
[  142.789967] RBP: ffffffff80757df0 R08: 0000000000000000 R09: ffff88000101bc30
[  142.789967] R10: 0000000000009c00 R11: 0000000000000246 R12: 0000000000000000
[  142.789967] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  142.789967] FS:  00007f481b29b6f0(0000) GS:ffff880001026000(0000) knlGS:0000000000000000
[  142.789967] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
[  142.789967] CR2: 00007f481ad77d2e CR3: 00000003e6a95000 CR4: 0000000000000660
[  142.789967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  142.789967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  142.789967] Call Trace:
[  142.789967]  [<ffffffff8020c475>] ? xen_safe_halt+0xdf/0x10f
[  142.789967]  [<ffffffff8020f1e3>] ? xen_idle+0x25/0x5a
[  142.789967]  [<ffffffff80208835>] ? cpu_idle+0x98/0xcb
[  142.789967]  [<ffffffff803ad85a>] ? vgacon_cursor+0x0/0x17d
[  142.791231] BUG: soft lockup - CPU#3 stuck for 114s! [swapper:0]
[  142.791231] Modules linked in: iptable_mangle ipt_REDIRECT iptable_nat nf_nat xt_tcpudp ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables scsi_wait_scan
[  142.791231] CPU 3:
[  142.791231] Modules linked in: iptable_mangle ipt_REDIRECT iptable_nat nf_nat xt_tcpudp ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables scsi_wait_scan
[  142.791231] Pid: 0, comm: swapper Not tainted 2.6.32-xen-r1 #1 Sun Fire V40z
[  142.791231] RIP: e030:[<ffffffff802063aa>]  [<ffffffff802063aa>] 0xffffffff802063aa
[  142.791231] RSP: e02b:ffff8803e8487ee8  EFLAGS: 00000246
[  142.791231] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffffffff802063aa
[  142.791231] RDX: ffff8803e8487f00 RSI: 0000000000000000 RDI: 0000000000000001
[  142.791231] RBP: ffffffff80757df0 R08: ffff8803e8487ed0 R09: 0000000000000040
[  142.791231] R10: 000000013fffa42f R11: 0000000000000246 R12: 0000000000000000
[  142.791231] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  142.791231] FS:  00007fc917ce66f0(0000) GS:ffff88000104a000(0000) knlGS:0000000000000000
[  142.791231] CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
[  142.791231] CR2: 00007fc9177c2d2e CR3: 00000003e60ef000 CR4: 0000000000000660
[  142.791231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  142.791231] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  142.791231] Call Trace:
[  142.791231]  [<ffffffff8020c475>] ? xen_safe_halt+0xdf/0x10f
[  142.791231]  [<ffffffff8020f1e3>] ? xen_idle+0x25/0x5a
[  142.791231]  [<ffffffff80208835>] ? cpu_idle+0x98/0xcb

Is nobody else seeing this?

EDIT: I opened an issue at the project website's bugtracker.

[meyerm]

Anybody already using xen-4.0? Will you/andylyon provide new xen-sources for xen-4.0?

[2bbionic]

@meyerm: I use the new XEN 4 for about three days now with the 2.6.32-xen-r1 kernel on a 32bit system. No problems so far...

[vr13]

2meyerm: seems to work stable with 2.6.32-xen-r1@x86_64:

Code: Select all

# xm dmesg
 __  __            _  _    ___   ___  
 \ \/ /___ _ __   | || |  / _ \ / _ \ 
  \  // _ \ '_ \  | || |_| | | | | | |
  /  \  __/ | | | |__   _| |_| | |_| |
 /_/\_\___|_| |_|    |_|(_)___(_)___/ 
                                      
(XEN) Xen version 4.0.0 (@local) (gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5) ) Sun May  9 10:29:43 MSD 2010
...
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0x1000000 -> 0x1506000
...

i'm using both 32 and 64bit pv domUs and 32bit hvm domUs (winxp, freebsd). the only difference from app-emulation/xen-tools-3.4.2 ebuild is `ioemu' use flag: one have to be specified explicitly

[meyerm]

Mh, thank you both for answering. I tried the xen 4.0 and xen-sources 2.6.32-r1 - both from the portage tree. But even though it starts, I still have the problem mentioned above. Did anybody of you compile in the soft lockup detection into the kernel? If not, does anybody of you have a possibility to play around with that setting and check if you see the same problem?

Perhaps it has to do something with my Serial-Over-LAN-setting of the server which forwards the serial console to a service processor? I can't see xen when booting but only the kernel and userspace on the serial console (or just xen and the kernel but no userspace...) - but this problem ist probably just a misconfiguration by me (see below). I just mention it, if it is connected to the soft lockup (vgacon_cursor).

Code: Select all

kernel /boot/xen.gz console=vga
module /boot/vmlinuz-xen root=/dev/ram0 real_root=/dev/md1 console=tty0 console=ttyS0,38400

[meyerm]

Guys, I guess I found the problem. I do not understand the real reason behind it. But I now know how to prevent the soft lockup messages...

Since I had bad experiences with Xen at the beginning of crashing and freezing my computer, I decided to continue step by step. So my first step was setting up a nice, working envorinment with gentoo-sources. After that I added Xen and started xen-sources on it. I did NOT start xend or did any other configurations within Xen until this point.

Since nobody was able to help me with my soft lockup problems, I decided to start creating a xen config for now and did hope for a later version of xen-sources solving my problems. And after starting xend (which takes a lot of time) these messages no longer appear in my dmesg... I do not yet understand the reason why the kernel has these problems when running without xend. But for now I'm happy

So, short for people coming over search engines: Starting a dom0 on top of the Xen-HV without starting xend seems to lead to soft lockup errors in the kernel.

[amissus]

I have strange issue with HVM mode.
When I sometimes hardly shutdown HVM domain (xm destroy) it causes "xen stops working". Domain 0 is still functional, no kernel panic or freezing, but it isn't possible to create U domains until reboot domain 0 (no matter if PVM or HVM). When xm create is initiated it waits a few minutes and there write something about error in xen hotplug script. Have someone experienced this behaviour too?

[lopb]

The only compression I seem to have is gzip (so yes, it's vmlinuz file instead of bzImage file). I cannot pick bz2 or something. I can try to disable it, but that won't make a lot of difference I guess, as bz2 works fine for my normal kernel.

For some reason I can now boot using the Xen.gz kernel, but it hangs during boot

You'll have to excuse my blurry images, I've used my cell phone to take them. Basicly, it hangs at async/1

The last message: "async/1 used greatest stack depth"

http://www.flickr.com/photos/tuinslak/3720354353/
http://www.flickr.com/photos/tuinslak/3720353783/

I've left the server on for over an hour, and it hasn't moved a bit.

The kernel without xen.gz still gives me error 13.

kernel sizes:

Code: Select all

four boot # du -sh kern*
2.7M	kernel-2.6.29
3.2M	kernel-2.6.30

one being bz2, other gzip. Seem like normal sizes to me.

I got the same problem
Here is the solution
https://sites.google.com/site/lopezjp/c ... 01092010.3

[Mathijs]

And you know for sure the domU does not come up? Did you try pinging it from another machine?
Because that's exactly how it looks when your console is not set up right in your domU config.

[Scorpion265]

Is there any chance of getting the cleancache patches merged in so we can have tmem support on a gentoo dom0?