Well -
That was rough!
Xen 4.19.2 boots! And I can shell in via ssh.
The Tigon eth cards would not come up under xen. I would get:
Code: Select all
1 # RTNETLINK answers: cannot allocate memory
2 # SIOCSIFFLAGS cannot allocate memory
Number 1 showed up the boot output. Number 2 returned from any variation of
ifconfig eth0 up. Trying to modprobe tg3 just made a mess.
xl dmesg shows four changes - two of which are just due to boot timing. The two change indicators are:
- (XEN) Intel VT-d Dom0 DMA Passthrough enabled.
(XEN) - Dom0 mode: Passthrough
These are from adding:
to xen's command line. The command line is the same in xen.cfg, and refind_xen.conf. That has the effect of obfuscating the root cause. YMMV.
My command line is:
Code: Select all
loader xen.efi ucode=ucode.cpio iommu=dom0-passthrough dom0_mem=4096M kernel=vmlinuz-6.12.31-gentoo root=/dev/sda3 net.ifnames=0 ipv6.autoconf=0
The initrd was generated with dracut - on bare metal. It gets the machine booted. It may be worth trimming in the future. The firmware is built into the kernel. There is no way to do that for Xen.
dmesg shows mainly boot timimg differences, and the Tigon cards successfully coming online. The best way to fix memory allocation issues as a system guy is to look for missing components!
The Kernel! I'm pretty sure I would not have gotten here by the handbook, or the Xen documentation! I would suggest that "making it easier" to install the kernel is a disservice to the community! We need these skills to innovate! </rant>
So - Here is my diff from kernel to kernel. This was the solution! The files are kernels/six.12.31-e
AND kernels/six.12.31-f.
Code: Select all
1066a1067
> CONFIG_DEVICE_MIGRATION=y
1108c1109,1111
< # CONFIG_ZONE_DEVICE is not set
---
> CONFIG_ZONE_DEVICE=y
> CONFIG_GET_FREE_REGION=y
> CONFIG_DEVICE_PRIVATE=y
1128c1131
< # CONFIG_NUMA_EMU is not set
---
> CONFIG_NUMA_EMU=y
1714a1724
> # CONFIG_PCI_P2PDMA is not set
4474c4483
< # CONFIG_USB_XEN_HCD is not set
---
> CONFIG_USB_XEN_HCD=m
4986a4996
> CONFIG_XEN_PRIVCMD_EVENTFD=y
4996c5006,5010
< # CONFIG_XEN_VIRTIO is not set
---
> CONFIG_XEN_UNPOPULATED_ALLOC=y
> CONFIG_XEN_GRANT_DMA_IOMMU=y
> CONFIG_XEN_GRANT_DMA_OPS=y
> CONFIG_XEN_VIRTIO=y
> CONFIG_XEN_VIRTIO_FORCE_GRANT=y
5929a5944,5946
> CONFIG_ND_PFN=m
> CONFIG_NVDIMM_PFN=y
> CONFIG_NVDIMM_DAX=y
You are not supposed to edit .config by hand! But you can read it! I use
make menuconfig, because it works in ssh in a terminal. nconfig requires X, and a mouse. Anyone get heavy fingers? Menuconfig has a search function accessed by "/". Just like the man pages! Or any text you can get into less(1). The text after "CONFIG_" will usually bring up a discription of that option that is quite helpful. If it brings up more - *BONUS*! In this case I searched for "xen". That showed me more than the help text ever will, in this case:
XEN_VIRTIO! Dom0's need all of Xen!
Some of the changes are automatic. We are looking for xen to add memory management. Xen manages multiple processors differently than the kernel. Numa emulation is needed! ZONE_DEVICE was a dependency for a xen option. USB_XEN_HCD is for future use. Some just disappear when not selected! They are critical adds to the Xen documentation. I've been in the wiki - it's a mess!
The patchset is small, and it doesn't apply smoothly. In any case, it was mostly for tools, so it did not help to get the server fully booted!
The 30 MB boot file booted every time, and loaded the kernel, every time. FWIW, I moved to rEFInd because my lappy boots xen with that. I may try Grub in the future.
I'm off to build VMs! I hope this helps someone!
