CPU ISA level is lower than required (32bit cross-compiled)

Message

sliwowitz · Post by **sliwowitz** » Fri Nov 22, 2024 9:03 pm

I have a x86-64 machine that builds binpkgs for a 32bit Pentium 4 machine. The build machine tyr-cross-master is a 64bit gentoo installation in a VM. When doing updates, I:

Log in, update the packages in the VM
Do a bunch of bind mounts and switch to a 32bit chroot with linux32 chroot /usr/i686-pc-linux-gnu /bin/bash --login
Run emerge world -DNu in the chroot. This builds binpkgs and stores them on a NFS mount.
Fire up tyr-staging VM which runs a 32bit system
Update the staging system from the binpkgs on the NFS mount
Check that everything works
Log in to the physical 32bit Pentium 4 machine tyr
Update the physical machine via the same binpkgs

So far this worked fine, but today, glibc-2.40-r5 was built which doesn't work on the 32bit hardware

In the 32bit build chroot, I have:

Code: Select all

(chroot)tyr-cross-master / # emerge --info
Portage 3.0.66.1 (python 3.13.0-final-0, default/linux/x86/23.0/i686, gcc-14, glibc-2.40-r5, 6.6.62-gentoo-kvm i686)
=================================================================
System uname: Linux-6.6.62-gentoo-kvm-i686-Intel_Xeon_Processor_-Skylake,_IBRS-with-glibc2.40
...
CBUILD="i686-pc-linux-gnu"
CFLAGS="-m32 -march=pentium4 -O3 -pipe -fomit-frame-pointer -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=auto -fuse-linker-plugin"
CHOST="i686-pc-linux-gnu"

On this 64bit VM in 32bit chroot, the new glibc-2.40-r5 works.

In the staging VM:

Code: Select all

tyr-staging ~ # emerge --info
Portage 3.0.66.1 (python 3.13.0-final-0, default/linux/x86/23.0/i686, gcc-13, glibc-2.40-r5, 6.6.62-gentoo-tyr-staging i686)
=================================================================
System uname: Linux-6.6.62-gentoo-tyr-staging-i686-Intel_Xeon_Processor_-Skylake,_IBRS-with-glibc2.40
...
CBUILD="i686-pc-linux-gnu"
CFLAGS="-m32 -O3 -march=pentium4 -pipe -fomit-frame-pointer -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin"
CHOST="i686-pc-linux-gnu"

On this 32bit VM it still works (maybe because the physical CPU actually supports that ISA)

The real Pentium 4 hardware, after glibc-2.40-r5 was installed:

Code: Select all

tyr /home/mame # emerge --info
Portage 3.0.65 (python 3.12.6-final-0, default/linux/x86/23.0/i686, gcc-13, glibc-2.40-r5, 6.6.52-gentoo-mame i686)
=================================================================
System uname: Linux-6.6.52-gentoo-mame-i686-with-glibc2.40
...
CBUILD="i686-pc-linux-gnu"
CFLAGS="-m32 -O3 -march=pentium4 -pipe -fomit-frame-pointer -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=1 -fuse-linker-plugin"
CHOST="i686-pc-linux-gnu"

Here it fails spectaculary. After emerge updates glibc, almost everything starts throwing CPU ISA level is lower than required error. I'm stuck in a terminal where I can't do e.g. ls (though I could run emerge --info, so maybe python is yet unaffected, but I can't emerge packages due to failing basic commands like find).

Checking ld.so --help:

Code: Select all

# ld.so --help
...

This program interpreter self-identifies as: /lib/ld-linux.so.2

Shared library search path:
  (libraries located via /etc/ld.so.cache)
  /lib (system search path)
  /usr/lib (system search path)

No subdirectories of glibc-hwcaps directories are searched.

Same on all machines.

Looking at libc.so.6:

Code: Select all

(chroot)tyr-cross-master / # readelf -n /lib/libc.so.6

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

The older libc.so.6 from glibc-2.39-r9 shows the exact same readelf output, yet it has worked fine on the physical machine since August.

What's wrong with my glibc-2.40-r5 build? How do I build it properly so that it would work on the 32bit hardware as did the earlier versions?

Post by **sam_** » Sat Nov 23, 2024 12:04 am

I can't look at this more right this moment, but let me dump some notes to help you (and maybe anyone else here).

There's two things to keep in mind when it comes to ISA markers:

The ISA markers can be a pain when building binaries even without -march=native, see https://wiki.gentoo.org/wiki/User:Sam/I ... rtup_files. I don't think this is the case for you here.
We had a bug a few months ago where this marker was wrongly applied in some cases for x86: https://bugs.gentoo.org/933764[b][/b].

But I don't think either of those things are your issue - just noting them to hopefully avoid red herrings, and also to give a better understanding of the mechanism at play here.

silowitz wrote: The older libc.so.6 from glibc-2.39-r9 shows the exact same readelf output, yet it has worked fine on the physical machine since August.

This part makes it even more mysterious.

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 12:33 am

Yeah, I've seen that bug report, and after a while figured it probably doesn't apply to me exactly (also the other bug report about livecds that was about glibc-2.39-r9 which worked for me). IM[/quote]HO the -march=pentium4 should do, as that machine is an actual Pentium 4. That also means it has SSE2, so many other reports where this error came on x86 due to missing SSE2 don't apply to me either.

One think that changed between the last update (cca a month ago, though the glibc-2.39-r9 package is even older, built on Aug 16) and this one is that this time there was a gcc update, so before everything was built with gcc-13, now I rebuilt with gcc-14 (did an emerge -e world). I've come across a couple recent reports that also used gcc-14, though it might be just a coincidence (and their other parameters were also different from my situation).

sam_ wrote:I can't look at this more right this moment, but let me dump some notes to help you (and maybe anyone else here).
silowitz wrote: The older libc.so.6 from glibc-2.39-r9 shows the exact same readelf output, yet it has worked fine on the physical machine since August.
This part makes it even more mysterious.

Yes this is puzzling me too.

Post by **sam_** » Sat Nov 23, 2024 12:42 am

Can you diff the objdump output for each of the binaries installed by both glibcs? I assume you have binpkgs of them both available, possibly to share if necessary.

It's possible that one of the other files does differ (like ld-linux-* or one of the startup files).

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 12:49 am

sam_ wrote:Can you diff the objdump output for each of the binaries installed by both glibcs? I assume you have binpkgs of them both available, possibly to share if necessary.

Sure. I have both binpkgs (don't know where to upload them for sharing in the forum - they're 15MB each).

Which objdump options should I run?

Post by **sam_** » Sat Nov 23, 2024 1:08 am

Start with just readelf -n, actually, then can look at the proper binaries if needed (can email them or something) later on (easier than trying to do it all via proxy).

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 1:38 am

sam_ wrote:Start with just readelf -n, actually, then can look at the proper binaries if needed (can email them or something) later on (easier than trying to do it all via proxy).

readelf shows the same info:

Code: Select all

(chroot)tyr-cross-master ~ # readelf -n glibc-2.39-r9/lib/libc.so.6 

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0


(chroot)tyr-cross-master ~ # readelf -n glibc-2.39-r9/lib/ld-linux.so.2 

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000018	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature used: x86, FXSR, XSAVE, XSAVEC
	x86 ISA used: x86-64-baseline, x86-64-v3


(chroot)tyr-cross-master ~ # readelf -n glibc-2.40-r5/lib/libc.so.6 

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0


(chroot)tyr-cross-master ~ # readelf -n glibc-2.40-r5/lib/ld-linux.so.2 

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000018	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature used: x86, FXSR, XSAVE, XSAVEC
	x86 ISA used: x86-64-baseline, x86-64-v3

Sent you a PM with links to the binpkg files.

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 1:43 am

Since the physical Pentium 4 machine's OS won't boot anymore, I have to access it via a LiveUSB stick. From there I can only copy files over to the hard drive because chroot won't work either. Just to be sure - if I try to make a new glibc binpkg, can I then just extract the contents of the binpkg image to the root of the target machine filesystem, and expect it to work or is there more to it?

Post by **sam_** » Sat Nov 23, 2024 1:48 am

You can raw untar the image. It's not great to do and it's better to use emerge --root= from outside but in a pinch, untarring is alright, just it might make a mess and obviously won't remove any old files which got left behind by the previous version (so you might have dead symlinks or similar).

Will check the link in a bit, thanks.

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 3:32 am

I got into the machine, and replaced the glibc, but it didn't help. Then I noticed (in /var/log/portage/elog) that emerge actually died in the prerm phase of emerging coreutils which came right after glibc. So that's where the issue might have been actually coming from. I'm investigating now, trying to replace coreutils with an old binpkg too.

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 5:07 am

OK. For some reason, coreutils and ncurses were broken. I emerged both again (and the glibc too) on the build machine (this is imho weird since the earlier emerge -e world should have done that), and again on the physical machine from the new binpkgs, they started working.

Then went to I install gcc-14 on the physical machine, and found out its binaries also started throwing the ISA error - though here it somehow made sense since earlier I actually ran emerge -e world --exclude=sys-devel/gcc, (when a new gcc is out, I always upgrade gcc itself first, then rebuild everything except the gcc because compiling gcc takes about half of the time of the full upgrade). After rebuilding gcc-14 on the build machine, and installing the new binpkg on the physical machine, it started working.

Then I tried adding some random nonessential packages to see if I'd have to rebuild everything once more. Some worked (ncdu) some didn't (pciutils), so to be on the safe side, I'm just rebuilding everything again with the newly rebuilt gcc-14 now.

sliwowitz · Post by **sliwowitz** » Sat Nov 23, 2024 5:55 am

Yeah. Seems the issue was gcc after all. Originally, I:

logged in to a VM with gcc-13 and glibc-2.39-r9
built gcc-14 with glibc-2.39-r9,
built glibc-2.40-r5, with gcc-14,
then built all remaining packages with gcc-14 and glibc-2.40-r5.

This gcc-14 (and some other packages) were bad - they worked in the VM, but not on the real Pentium 4.

Then I rebuilt gcc-14 (with glibc-2.40-r5 already installed), and it started working on the real Pentium 4 too.

readelf shows the gcc binaries are differnt

Code: Select all

strazce@odin ~/tmp $ readelf -n gcc-14.2.1_p20241116-bad/image/usr/i686-pc-linux-gnu/gcc-bin/14/gcc

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline, x86-64-v2, x86-64-v3, x86-64-v4
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0


strazce@odin ~/tmp $ readelf -n gcc-14.2.1_p20241116/image/usr/i686-pc-linux-gnu/gcc-bin/14/gcc

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

This carried over to other packages built by the "bad" gcc (though not into glibc itself even though it was built with the "bad" gcc too).

Code: Select all

strazce@odin ~/tmp $ readelf -n vim-9.1.0366-bad/image/usr/bin/vim

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline, x86-64-v2, x86-64-v3, x86-64-v4
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0


strazce@odin ~/tmp $ readelf -n vim-9.1.0366/image/usr/bin/vim

Displaying notes found in: .note.gnu.property
  Owner                Data size 	Description
  GNU                  0x00000024	NT_GNU_PROPERTY_TYPE_0
      Properties: x86 ISA needed: x86-64-baseline
	x86 feature used: x86, x87, XMM
	x86 ISA used: x86-64-baseline

Displaying notes found in: .note.ABI-tag
  Owner                Data size 	Description
  GNU                  0x00000010	NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

The solution seems to be (assuming you first built gcc, then glibc): after building glibc2.40-r5 with gcc-14, rebuild gcc-14 with glibc2.40-r5, then rebuild the remaining packaes.