glibc fails when building for bin client [solved]

[DaggyStyle]

Greetings,

I have a bin host and client, the host is ryxen X2700 and the client is intel based.
I compile all the bins for the client inside a chroot with shared cflags between the host and the client.
everything works great when the client cpu was g4560.
now I've upgraded it to i7-7700 and updated the shared cflags and cpu flags to this:

Code: Select all

CFLAGS="-O2 -pipe -march=skylake -mabm --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=8192 -mno-sgx -mno-rtm -mno-hle -mno-mwaitx -mno-clzero -mno-sse4a -mno-sha"
CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3"

cpu's compile line can be seen at: http://dpaste.com/6QQL7GN5M
I've recompiled all pkgs and the kernel, one failure, glibc.
the error is

Code: Select all

x86_64-pc-linux-gnu-gcc -m64 -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wl,-O1 -Wl,--as-needed ../sysdeps/unix/sysv/linux/x86/elision-lock.c -c -std=gnu11 -fgnu89-inline  -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wall -Wwrite-strings -Wundef -fmerge-all-constants -frounding-math -fstack-protector-strong -Wstrict-prototypes -Wold-style-definition -fmath-errno    -mrtm  -ftls-model=initial-exec   -U_FORTIFY_SOURCE -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha  -I../include -I/var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl  -I/var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl  -I../sysdeps/unix/sysv/linux/x86_64/64  -I../sysdeps/unix/sysv/linux/x86_64  -I../sysdeps/unix/sysv/linux/x86/include -I../sysdeps/unix/sysv/linux/x86  -I../sysdeps/x86/nptl  -I../sysdeps/unix/sysv/linux/wordsize-64  -I../sysdeps/x86_64/nptl  -I../sysdeps/unix/sysv/linux/include -I../sysdeps/unix/sysv/linux  -I../sysdeps/nptl  -I../sysdeps/pthread  -I../sysdeps/gnu  -I../sysdeps/unix/inet  -I../sysdeps/unix/sysv  -I../sysdeps/unix/x86_64  -I../sysdeps/unix  -I../sysdeps/posix  -I../sysdeps/x86_64/64  -I../sysdeps/x86_64/fpu/multiarch  -I../sysdeps/x86_64/fpu  -I../sysdeps/x86/fpu  -I../sysdeps/x86_64/multiarch  -I../sysdeps/x86_64  -I../sysdeps/x86  -I../sysdeps/ieee754/float128  -I../sysdeps/ieee754/ldbl-96/include -I../sysdeps/ieee754/ldbl-96  -I../sysdeps/ieee754/dbl-64/wordsize-64  -I../sysdeps/ieee754/dbl-64  -I../sysdeps/ieee754/flt-32  -I../sysdeps/wordsize-64  -I../sysdeps/ieee754  -I../sysdeps/generic  -I.. -I../libio -I. -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include-fixed -isystem /usr/include -D_LIBC_REENTRANT -include /var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/libc-modules.h -DMODULE_NAME=libpthread -include ../include/libc-symbols.h  -DPIC     -DTOP_NAMESPACE=glibc -o /var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl/elision-lock.o -MD -MP -MF /var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl/elision-lock.o.dt -MT /var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl/elision-lock.o
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/immintrin.h:115,
                 from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/x86intrin.h:32,
                 from ../sysdeps/unix/sysv/linux/x86/hle.h:5,
                 from ../sysdeps/unix/sysv/linux/x86/elision-lock.c:22:
../sysdeps/unix/sysv/linux/x86/elision-lock.c: In function ‘__lll_lock_elision’:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:50:1: error: inlining failed in call to ‘always_inline’ ‘_xbegin’: target specific option mismatch
   50 | _xbegin (void)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:62:18: note: called from here
   62 |    if ((status = _xbegin()) == _XBEGIN_STARTED)
      |                  ^~~~~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/immintrin.h:115,
                 from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/x86intrin.h:32,
                 from ../sysdeps/unix/sysv/linux/x86/hle.h:5,
                 from ../sysdeps/unix/sysv/linux/x86/elision-lock.c:22:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:71:1: error: inlining failed in call to ‘always_inline’ ‘_xabort’: target specific option mismatch
   71 | _xabort (const unsigned int __imm)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:70:8: note: called from here
   70 |        _xabort (_ABORT_LOCK_BUSY);

full log at https://c.gmx.com/@562924463562169039/T ... YBkwsxYtuw

ideas?

I don't think it is a compiler issue

[eccerr0r]

Still seems like a compiler issue to me or at least what you're telling the compiler to do with CFLAGS, since it will build with simpler CFLAGS ?

Another weirdness is your CFLAGS=-no-sse4a but you have CPU_FLAGS_X86 of sse4a - seems like a conflict here (not sure what gcc does with it -- though it does seem to be a mismatch!!!)

[Anon-E-moose]

The SSE4a instruction group was introduced in AMD's Barcelona microarchitecture. These instructions are not available in Intel processors. Support is indicated via the CPUID.80000001H:ECX.SSE4A[Bit 6] flag

[DaggyStyle]

Still seems like a compiler issue to me or at least what you're telling the compiler to do with CFLAGS, since it will build with simpler CFLAGS ?

Another weirdness is your CFLAGS=-no-sse4a but you have CPU_FLAGS_X86 of sse4a - seems like a conflict here (not sure what gcc does with it -- though it does seem to be a mismatch!!!)

good catch regarding the latter, fixed.
the issue still persists.

should I upgrade gcc to 10.3? the main system uses gcc 10.3 and there is no issue there

[Anon-E-moose]

Code: Select all

../sysdeps/unix/sysv/linux/x86/elision-lock.c: In function ‘__lll_lock_elision’:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:50:1: error: inlining failed in call to ‘always_inline’ ‘_xbegin’: target specific option mismatch
   50 | _xbegin (void)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:62:18: note: called from here
   62 |    if ((status = _xbegin()) == _XBEGIN_STARTED)
      |                  ^~~~~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/immintrin.h:115,
                 from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/x86intrin.h:32,
                 from ../sysdeps/unix/sysv/linux/x86/hle.h:5,
                 from ../sysdeps/unix/sysv/linux/x86/elision-lock.c:22:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:71:1: error: inlining failed in call to ‘always_inline’ ‘_xabort’: target specific option mismatch
   71 | _xabort (const unsigned int __imm)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:70:8: note: called from here
   70 |        _xabort (_ABORT_LOCK_BUSY);
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~

I'm not sure that going to 10.3 will fix this.

[eccerr0r]

I'm not clear still - if you have no cflags at all, it works?

[DaggyStyle]

Code: Select all

../sysdeps/unix/sysv/linux/x86/elision-lock.c: In function ‘__lll_lock_elision’:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:50:1: error: inlining failed in call to ‘always_inline’ ‘_xbegin’: target specific option mismatch
   50 | _xbegin (void)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:62:18: note: called from here
   62 |    if ((status = _xbegin()) == _XBEGIN_STARTED)
      |                  ^~~~~~~~~
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/immintrin.h:115,
                 from /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/x86intrin.h:32,
                 from ../sysdeps/unix/sysv/linux/x86/hle.h:5,
                 from ../sysdeps/unix/sysv/linux/x86/elision-lock.c:22:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/rtmintrin.h:71:1: error: inlining failed in call to ‘always_inline’ ‘_xabort’: target specific option mismatch
   71 | _xabort (const unsigned int __imm)
      | ^~~~~~~
../sysdeps/unix/sysv/linux/x86/elision-lock.c:70:8: note: called from here
   70 |        _xabort (_ABORT_LOCK_BUSY);
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~

I'm not sure that going to 10.3 will fix this.

what Ifind weird is that Idefine -mno-rtm but gcc adds it regardless.

[Anon-E-moose]

if you go to the new machine (i7) and run

gcc -march=native -E -v - </dev/null 2>&1 | sed -n 's/.* -v - //p'

what does it show?

Edit to add: from my i3 7100u (kabylake but compiled as skylake)

Code: Select all

gcc version 9.3.0 (Gentoo 9.3.0 p2) 
COLLECT_GCC_OPTIONS='-march=native' '-E' '-v'
 /usr/libexec/gcc/x86_64-pc-linux-gnu/9.3.0/cc1 -E -quiet -v - -march=skylake -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -msgx -mbmi2 -mno-pconfig -mno-wbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx -mno-clzero -mno-pku -mno-rdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg -mno-movdiri -mno-movdir64b -mno-waitpkg -mno-cldemote -mno-ptwrite --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=3072 -mtune=skylake

[DaggyStyle]

Code: Select all

-march=skylake -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -msgx -mbmi2 -mno-pconfig -mno-wbnoinvd -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-avx5124fmaps -mno-avx5124vnniw -mno-clwb -mno-mwaitx -mno-clzero -mno-pku -mno-rdpid -mno-gfni -mno-shstk -mno-avx512vbmi2 -mno-avx512vnni -mno-vaes -mno-vpclmulqdq -mno-avx512bitalg -mno-avx512vpopcntdq -mno-movdiri -mno-movdir64b -mno-waitpkg -mno-cldemote -mno-ptwrite -mno-avx512bf16 -mno-enqcmd -mno-avx512vp2intersect --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -mtune=skylake

the host returns

Code: Select all

-march=znver1 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mno-vpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mno-clwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mno-prefetchwt1 -mprfchw -mno-ptwrite -mno-rdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mno-shstk -mno-tbm -mno-tsxldtrk -mno-vaes -mno-waitpkg -mno-wbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=znver1 -dumpbase -

[Anon-E-moose]

Well, you're (more than likely) getting the processor mismatch because it knows it's on a ryzen trying to do something that it expects to do on an intel chip, (target specific option mismatch)

So there's probably a flag in gcc that controls that, just not sure what it is off the top of my head.

Edit to add: add -no-rdpid to your flags (ryzen has, intel doesn't)

[DaggyStyle]

thought of that bug that raises the question, why it worked with the previous cpu?

[DaggyStyle]

Well, you're (more than likely) getting the processor mismatch because it knows it's on a ryzen trying to do something that it expects to do on an intel chip, (target specific option mismatch)

So there's probably a flag in gcc that controls that, just not sure what it is off the top of my head.

Edit to add: add -no-rdpid to your flags (ryzen has, intel doesn't)

fixed thanks. still no go

[DaggyStyle]

I see this in the log:

Code: Select all

 * strip-flags: CFLAGS: changed '-O2 -pipe -march=skylake -mabm --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=8192 -mno-sgx -mno-rtm -mno-hle -mno-mwaitx -mno-clzero -mno-sse4a -mno-sha -mno-rdpid' to '-O2 -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha'

why -mno-sgx was removed? ryzen doesn't have sgx

[eccerr0r]

None of this should be a ryzen vs intel problem as it doesn't even work on the build machine.

There is a conflict between two options on the build machine, like saying you want chocolate chip cookies but not allowed to use chocolate.

[Anon-E-moose]

The inlining failure reason target specific option mismatch means that inlining failed because the program calls an always-inline function with a specific target attribute from another function which does not support this target. This is really something that is not supportable: the compiler cannot both compile a function to use certain micro-architecture features (the always-inline function) and not use them (the function into which is inlined).

Why you're getting it, I don't know.

Edit to add: I would also add -mtune=skylake to the CFLAGS

[Anon-E-moose]

x86_64-pc-linux-gnu-gcc -m64 -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wl,-O1 -Wl,--as-needed ../sysdeps/unix/sysv/linux/x86/elision-lock.c -c -std=gnu11 -fgnu89-inline -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wall -Wwrite-strings -Wundef -fmerge-all-constants -frounding-math -fstack-protector-strong -Wstrict-prototypes -Wold-style-definition -fmath-errno -mrtm -ftls-model=initial-exec -U_FORTIFY_SOURCE -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -I../include -I/var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl -I/

Figure out where the hell -mrtm is coming from and that problem should be solved.

Edit to add: it comes from inside the glibc tarball sysdeps/unix/sysv/linux/x86/Makefile, still not sure why it's being pulled in, probably some define.

[DaggyStyle]

x86_64-pc-linux-gnu-gcc -m64 -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wl,-O1 -Wl,--as-needed ../sysdeps/unix/sysv/linux/x86/elision-lock.c -c -std=gnu11 -fgnu89-inline -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -O2 -Wall -Wwrite-strings -Wundef -fmerge-all-constants -frounding-math -fstack-protector-strong -Wstrict-prototypes -Wold-style-definition -fmath-errno -mrtm -ftls-model=initial-exec -U_FORTIFY_SOURCE -pipe -march=skylake -mno-rtm -mno-hle -mno-sse4a -mno-sha -I../include -I/var/tmp/portage/sys-libs/glibc-2.32-r7/work/build-amd64-x86_64-pc-linux-gnu-nptl/nptl -I/

Figure out where the hell -mrtm is coming from and that problem should be solved.

Edit to add: it comes from inside the glibc tarball sysdeps/unix/sysv/linux/x86/Makefile, still not sure why it's being pulled in, probably some define.

here: https://sourceware.org/git/?p=glibc.git ... b=HEAD#l14
now I wonder, why x86 and not x86_64?[/code]

[Anon-E-moose]

Do you have another compiler (9.3 for example) and if so does glibc compile with that?

[DaggyStyle]

Do you have another compiler (9.3 for example) and if so does glibc compile with that?

no but I can emerge one

[Anon-E-moose]

I tried with gcc 9.3 and didn't have a problem with compiling (used your version of glibc) and used skylake flags, etc.

[DaggyStyle]

I tried with gcc 9.3 and didn't have a problem with compiling (used your version of glibc) and used skylake flags, etc.

based on this and my main machine, upgrading to 10.3 should fix the issue

[DaggyStyle]

Solved!
it seems that I needed to add -mno-hle but not -mno-rtm.
I had gentoo updated glibc to 2.33 so I've looked into the build log and saw that -mrtm is used, so I've removed it and kept -mno-hle and all worked ok,