| View previous topic :: View next topic |
| Author |
Message |
kgdrenefort
Apprentice
Joined: 19 Sep 2023
Posts: 200
Location: Somewhere in the 77
|
 Posted: Fri Apr 05, 2024 12:46 pm Post subject: Binhosting: CPU instruction set for 'silvermont' vs 'znver1' |
 |
|
Hello,
Re-doing a badly made binhost and trying to take care of every details, I have this first question:
The binhost (Mephistopheles) have these values:
| Code: | root@Mephistopheles ~ # resolve-march-native
-march=znver1 --param=l1-cache-line-size=64 --param=l1-cache-size=32 --param=l2-cache-size=512 |
| Code: | root@Mephistopheles ~ # cpuid2cpuflags
CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 |
For this CPU: 12 × AMD Ryzen 5 2600 Six-Core Processor
While client (lilith) have:
| Code: | (chroot) root@livecd / # cpuid2cpuflags
CPU_FLAGS_X86: mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3 |
| Code: | (chroot) root@livecd / # resolve-march-native
-march=silvermont --param=l1-cache-line-size=64 --param=l1-cache-size=24 --param=l2-cache-size=1024 |
For this CPU: Intel(R) Celeron(R) CPU N2840 @ 2.16GHz
---
I'm pretty sure to have already asked, months ago, this question on #gentoo@libera.chat, but it's lost in the oblivion.
After reading a bit of this page: https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
I found that 'silvermont' (client) has not these instructions set from 'znver1':
(from the GCC page:)
| Code: | | SAHF, FXSR, PREFETCHW |
(from difference between direct outputed value between both cpuid2cpuflags command)
| Code: | | aes, avx, avx2, fl16c, sha, sse4 |
First, I do not understand why the command do not exactly report as the GCC page. I guess that is normal and I should probably refers to what actual systems are outputing.
---
I read from the wiki's page about setting up a binhost:
| a great wiki wrote: | | The CFLAGS and CXXFLAGS variables used to build the binary packages must be compatible with all clients. |
| a great wiki wrote: | | USE flags for processor specific instruction set features (like MMX, SSE, etc.) must be carefully selected; all clients must support them. |
Binary package guide - General prerequisites
And my guess would be that 'znver1' (the binhost, Mephistopheles) can only use these CPU flags instruction sets to build compatible binary for 'silvermont' (the client, lilith):
| Code: | | mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3 |
1/ So every flags from silvermont are into znver1, and I should keep the values as above on binhost. And it won't make any problem because all these flags are already in the sets of znver1. Correct ?
2/ In this case, "USE flags" are referring to the CPUs instructions sets, not to be confuse with GLOBAL and LOCAL (USE) flags sets for the system-wide or a specifical package, both are USE flags but not for the same purpose and has not to be mixed around. The CPU USE Flag has NOT to be sets into USE variables from make.conf, they are sets elsewhere in make.conf and portage (CFLAGS, 00cpu-flags file and such). Correct ?
3/ So the binhost, Mephistopheles, has to be set for CFLAGS values according to the target, 'silvermont' (actually installing from live USB). Otherwise, the source code will be build with potentially no-compliant CPU instructions set, would fail and torture me. Is that correct ?
---
Just in case, the output of lscpu on both:
The binhost (Mephistopheles): https://bpa.st/4R3Q
The client (lilith): https://bpa.st/IQBA
Regards,
GASPARD DE RENEFORT Kévin
_________________
«Gentoo does not have problems, only learning opportunities.» - NeddySeagoon
«If your Gentoo installation isn't valuable to you, feel free to continue to ignore the instructions.» - figueroa |
|
| Back to top |
|
 |
kgdrenefort
Apprentice
Joined: 19 Sep 2023
Posts: 200
Location: Somewhere in the 77
|
| Posted: Fri Apr 05, 2024 1:14 pm Post subject: |
|
|
This is the answer I got on #gentoo@libera.chat:
| tirnanog wrote: | | mostly correct. 3 isn't quite correct (it would be more accurate to say that you must not specify any flags that result in binaries containing opcodes that the binhost CPU, itself, is not aware of). as far as CFLAGS is concerned, -march=x86-64-v2 is your simplest option by far. |
So, from this reply, the set:
| Code: | | mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3 |
is valid from binhost to client. And should not make any troubles.
From this point, I could have all that is needed to keep going, I think.
---
EDIT: After much talking about it:
| Code: | # Define a convenient helper function
flags() { gcc -march="$1" -Q --help=target; }
# Show the exact differences between two targets (x86-64-v2 and silvermont in this example)
diff -u0 <(flags x86-64-v2) <(flags silvermont) | less -R |
Which output for silvermont:
| Code: | --- /dev/fd/63 2024-04-05 15:18:17.090086607 -0000
+++ /dev/fd/62 2024-04-05 15:18:17.090086607 -0000
@@ -13 +13 @@
- -maccumulate-outgoing-args [disabled]
+ -maccumulate-outgoing-args [enabled]
@@ -29 +29 @@
- -march= x86-64-v2
+ -march= silvermont
@@ -124 +124 @@
- -mmovbe [disabled]
+ -mmovbe [enabled]
@@ -145 +145 @@
- -mpclmul [disabled]
+ -mpclmul [enabled]
@@ -155 +155 @@
- -mprfchw [disabled]
+ -mprfchw [enabled]
@@ -160 +160 @@
- -mrdrnd [disabled]
+ -mrdrnd [enabled]
@@ -204 +204 @@
- -mtune= generic
+ -mtune= silvermont |
And for znver1:
| Code: |
--- /dev/fd/63 2024-04-05 15:24:09.350090570 -0000
+++ /dev/fd/62 2024-04-05 15:24:09.350090570 -0000
@@ -12 +12 @@
- -mabm [disabled]
+ -mabm [enabled]
@@ -15,2 +15,2 @@
- -madx [disabled]
- -maes [disabled]
+ -madx [enabled]
+ -maes [enabled]
@@ -29 +29 @@
- -march= x86-64-v2
+ -march= znver1
@@ -31,2 +31,2 @@
- -mavx [disabled]
- -mavx2 [disabled]
+ -mavx [enabled]
+ -mavx2 [enabled]
@@ -34 +34 @@
- -mavx256-split-unaligned-store [disabled]
+ -mavx256-split-unaligned-store [enabled]
@@ -58,2 +58,2 @@
- -mbmi [disabled]
- -mbmi2 [disabled]
+ -mbmi [enabled]
+ -mbmi2 [enabled]
@@ -65 +65 @@
- -mclflushopt [disabled]
+ -mclflushopt [enabled]
@@ -67 +67 @@
- -mclzero [disabled]
+ -mclzero [enabled]
@@ -78 +78 @@
- -mf16c [disabled]
+ -mf16c [enabled]
@@ -83 +83 @@
- -mfma [disabled]
+ -mfma [enabled]
@@ -89 +89 @@
- -mfsgsbase [disabled]
+ -mfsgsbase [enabled]
@@ -118 +118 @@
- -mlzcnt [disabled]
+ -mlzcnt [enabled]
@@ -124 +124 @@
- -mmovbe [disabled]
+ -mmovbe [enabled]
@@ -127 +127 @@
- -mmove-max= 128
+ -mmove-max= 256
@@ -132 +132 @@
- -mmwaitx [disabled]
+ -mmwaitx [enabled]
@@ -145 +145 @@
- -mpclmul [disabled]
+ -mpclmul [enabled]
@@ -151 +151 @@
- -mprefer-vector-width= none
+ -mprefer-vector-width= 128
@@ -155 +155 @@
- -mprfchw [disabled]
+ -mprfchw [enabled]
@@ -160,2 +160,2 @@
- -mrdrnd [disabled]
- -mrdseed [disabled]
+ -mrdrnd [enabled]
+ -mrdseed [enabled]
@@ -175 +175 @@
- -msha [disabled]
+ -msha [enabled]
@@ -186 +186 @@
- -msse4a [disabled]
+ -msse4a [enabled]
@@ -196 +196 @@
- -mstore-max= 128
+ -mstore-max= 256
@@ -204 +204 @@
- -mtune= generic
+ -mtune= znver1
@@ -218,4 +218,4 @@
- -mxsave [disabled]
- -mxsavec [disabled]
- -mxsaveopt [disabled]
- -mxsaves [disabled]
+ -mxsave [enabled]
+ -mxsavec [enabled]
+ -mxsaveopt [enabled]
+ -mxsaves [enabled]
|
From these both, we can say that this is the safe way:
| Code: | | CFLAGS="-march=x86-64-v2 -mmovbe -mpclmul -mprfchw -mdrdnd" |
For the binhost.
The client would use -march=native.
Regards,
GASPARD DE RENEFORT Kévin
_________________
«Gentoo does not have problems, only learning opportunities.» - NeddySeagoon
«If your Gentoo installation isn't valuable to you, feel free to continue to ignore the instructions.» - figueroa |
|
| Back to top |
|
|
kgdrenefort
Apprentice
Joined: 19 Sep 2023
Posts: 200
Location: Somewhere in the 77
|
| Posted: Fri Apr 05, 2024 1:47 pm Post subject: |
|
|
To add more precision, this is what should looks like my make.conf for the binhost:
| Code: |
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
COMMON_FLAGS="-march=x86-64-v2 -O2 -pipe -mmovbe -mpclmul -mprfchw -mrdrnd"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
# NOTE: This stage was built with the bindist Use flag enabled
# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C.utf8 |
And the client should only differ by having:
| Code: | | COMMON_FLAGS="-march=native -O2 -pipe" |
Regards,
GASPARD DE RENEFORT Kévin
_________________
«Gentoo does not have problems, only learning opportunities.» - NeddySeagoon
«If your Gentoo installation isn't valuable to you, feel free to continue to ignore the instructions.» - figueroa |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 4186
Location: Bavaria
|
| Posted: Fri Apr 05, 2024 2:02 pm Post subject: |
|
|
Moved from Installing Gentoo to Documentation, Tips & Tricks.
I think our readers will find this document easier here.
_________________
https://wiki.gentoo.org/wiki/User:Pietinger |
|
| Back to top |
|
|
kgdrenefort
Apprentice
Joined: 19 Sep 2023
Posts: 200
Location: Somewhere in the 77
|
| Posted: Fri Apr 05, 2024 3:49 pm Post subject: |
|
|
| pietinger wrote: | Moved from Installing Gentoo to Documentation, Tips & Tricks.
I think our readers will find this document easier here. |
If you think it's better like this :) !
It was not the purpose, at first I hoped an answer from the forum, but it comes from the IRC channel and just edited it for me as for others.
_________________
«Gentoo does not have problems, only learning opportunities.» - NeddySeagoon
«If your Gentoo installation isn't valuable to you, feel free to continue to ignore the instructions.» - figueroa |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54276
Location: 56N 3W
|
| Posted: Fri Apr 05, 2024 4:36 pm Post subject: |
|
|
kgdrenefort,
There are two problems to deal with.
The built host can emit code for any amd64 CPU bunt may not be able to execute it. Keep that thought.
There are two places where that matters.
1. In the toolchain doing the building. You don not a want a broken toolchain.
2. In build systems that build code that happens to contain instructions that the build host cannot execute, the the build system executes that code as a part of the build process.
Depending on how much time you want to put into setting up the binhost, there are two solutions.
Find some common CPU flags, or per package CPU flage for the problem packages.
See What Goes Wrong.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
kgdrenefort
Apprentice
Joined: 19 Sep 2023
Posts: 200
Location: Somewhere in the 77
|
| Posted: Sat Apr 06, 2024 9:48 am Post subject: |
|
|
Thanks for you reply Neddy.
So far, it seems to works.
I have been able to finish the installation, and made it with most of 95% binary packages from the binhost I made.
I have now rebooted and have to finish the setup. But the system rebooted with a kernel binary I compiled on the binhost, and it boots, as for grub etc.
So I think the guidance of tirnanog was really useful to me and maybe avoided me some troubles.
| NeddySeagoon wrote: | There are two problems to deal with.
The built host can emit code for any amd64 CPU bunt may not be able to execute it. Keep that thought.
There are two places where that matters.
1. In the toolchain doing the building. You don not a want a broken toolchain.
2. In build systems that build code that happens to contain instructions that the build host cannot execute, the the build system executes that code as a part of the build process. |
It seems that any instructions from silvermont is into the flags set of znver1, so if I understood, that won’t be a problem.
For example I was able to compile lighttpd with the CPU flag sets from silvermont on znver1 and it works flawlessly. Maybe that is not the best example, maybe lighttpd was not a problem but some other tools will have some.
I’ll update the topic about how it worked, if I missed something regarding the binhost-compliance with the client.
Regards,
GASPARD DE RENEFORT Kévin
_________________
«Gentoo does not have problems, only learning opportunities.» - NeddySeagoon
«If your Gentoo installation isn't valuable to you, feel free to continue to ignore the instructions.» - figueroa |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54276
Location: 56N 3W
|
| Posted: Sat Apr 06, 2024 10:13 am Post subject: |
|
|
kgdrenefort,
| Quote: | | For example I was able to compile lighttpd with the CPU flag sets from silvermont on znver1 and it works flawlessly. |
I would expect anything that builds on the build host for the target to run correctly on the target.
Problems arise when code built for the target is run on the build host,
That can happen either deliberately, with something like gcc, or as a build product, say if the lighttpd builds something for the target in the course of the build, then tries to run it on the build host.
This sort of thing stands out like a sore thumb when cross compiling because trying to run arm code on amd64 fails very quickly.
Its more subtle in the case of a chroot same arch build. The build system is not compelled to use the entire instruction set for the target. When it chooses a subset that the host can execute everything works.
Thank you for for your signature. I missed that wee gem by figueroa :)
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21661
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
