Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] LLVM_TARGETS not respected?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
smartding
Tux's lil' helper
Tux's lil' helper


Joined: 22 Jan 2021
Posts: 129

PostPosted: Mon Mar 07, 2022 3:01 pm    Post subject: [solved] LLVM_TARGETS not respected? Reply with quote

I have LLVM_TARGETS="X86 BPF" in my /etc/portage/make.conf

Today, after upgrading to llvm 13.0.1, running llc --version shows:

Code:

llc --version
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: tigerlake
 
  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc32le    - PowerPC 32 LE
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore


Ilvm now supports all those targets and my global LLVM_TARGETS setting is not respected.

Also, running equery u llvm now shows:

Code:

equery u llvm
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for sys-devel/llvm-13.0.1:
 U I
 - - abi_x86_32      : 32-bit (x86) libraries
 + + binutils-plugin : Build the binutils plugin
 - - debug           : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
 - - doc             : Build and install the HTML documentation and regenerate the man pages
 - - exegesis        : Enable performance counter support for llvm-exegesis tool that can be used to measure host machine instruction characteristics
 - - libedit         : Use the libedit library (replacement for readline)
 + + libffi          : Enable support for Foreign Function Interface library
 + + ncurses         : Support querying terminal properties using ncurses' terminfo
 - - test            : Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently)
 - - xar             : Support dumping LLVM bitcode sections in Mach-O files (uses app-arch/xar)
 - - xml             : Add support for XML files
 - - z3              : Enable support for sci-mathematics/z3 constraint solver


I think I used to have flags such as llvm_targets_AArch64, llvm_targets_ARM, they are no longer there.

[Moderator edit: changed [quote] tags to [code] tags to preserve output layout. -Hu]


Last edited by smartding on Tue Mar 08, 2022 2:50 am; edited 1 time in total
Back to top
View user's profile Send private message
Ionen
Developer
Developer


Joined: 06 Dec 2018
Posts: 2874

PostPosted: Mon Mar 07, 2022 3:59 pm    Post subject: Reply with quote

Given disabling targets breaks ABI, they're now forced by default as a precaution (i.e. could break clang/rust/lld and -U/-N to force rebuild on LLVM_TARGETS changes doesn't always work out so well). Packages like zig also need all targets enabled and don't support these being arbitrarily disabled.

If you /really/ want to disable them, you'll need to unforce it through /etc/portage/profile/package.use.force by adding e.g. */* -llvm_targets_AArch64 and so on at your own risks (will at least need to ensure clang was rebuilt after llvm -- if before it'll be broken, and manually rebuild lld -- if using system-llvm on rust it'll likely be broken too if rebuilt before, any other package using llvm libraries may also have problems even if they don't use LLVM_TARGETS at all).
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22778

PostPosted: Mon Mar 07, 2022 4:13 pm    Post subject: Reply with quote

Ionen addressed the immediate problem. As a general tip, when reporting that flags are not what you want, you should show the Portage output confirming that. In this case, emerge --verbose --info sys-devel/llvm would have been nice to have.
Back to top
View user's profile Send private message
sdauth
l33t
l33t


Joined: 19 Sep 2018
Posts: 659
Location: Ásgarðr

PostPosted: Mon Mar 07, 2022 4:28 pm    Post subject: Reply with quote

Thanks Ionen for the explanation. I was having the same issue as OP. It would be nice to add a note to eselect news to warn users about it.
Back to top
View user's profile Send private message
figueroa
Advocate
Advocate


Joined: 14 Aug 2005
Posts: 3007
Location: Edge of marsh USA

PostPosted: Mon Mar 07, 2022 6:09 pm    Post subject: Reply with quote

+1 -- I came here this morning specifically looking for this thread.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3446
Location: Canada

PostPosted: Mon Mar 07, 2022 6:57 pm    Post subject: Reply with quote

Same as other people, a bit froze seeing all the targets in the today's update
Back to top
View user's profile Send private message
Ionen
Developer
Developer


Joined: 06 Dec 2018
Posts: 2874

PostPosted: Mon Mar 07, 2022 7:15 pm    Post subject: Reply with quote

dmpogo wrote:
Same as other people, a bit froze seeing all the targets in the today's update

The forcing technically been done early last November, but was set to be only active starting >=llvm-13.0.1_rc, formerly unkeyworded, then non-rc got keyworded, and now llvm-13.0.1 was stabilized today so it's what you're newly seeing.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 3446
Location: Canada

PostPosted: Mon Mar 07, 2022 7:30 pm    Post subject: Reply with quote

Ionen wrote:
dmpogo wrote:
Same as other people, a bit froze seeing all the targets in the today's update

The forcing technically been done early last November, but was set to be only active starting >=llvm-13.0.1_rc, formerly unkeyworded, then non-rc got keyworded, and now llvm-13.0.1 was stabilized today so it's what you're newly seeing.


I understand, now that prompts me once again to check whether I can get rid of llvm on my laptop. Only mesa needs it, but I am with intel driver, and though use gallium, do not use opencl. So should check if I can disable llvm, and what will be the outcome.
Back to top
View user's profile Send private message
smartding
Tux's lil' helper
Tux's lil' helper


Joined: 22 Jan 2021
Posts: 129

PostPosted: Tue Mar 08, 2022 2:49 am    Post subject: Reply with quote

Mystery solved.

It would be better if there's some info about this change at the end of emerge.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22778

PostPosted: Tue Mar 08, 2022 3:59 pm    Post subject: Reply with quote

I would think that most people who notice this at all would notice it before building LLVM, so a notice logged at the end of the llvm emerge would be too late. As I hinted above, the output of emerge --pretend --verbose should show the flags as forced.
Back to top
View user's profile Send private message
nagora
n00b
n00b


Joined: 08 Jul 2014
Posts: 11

PostPosted: Wed Mar 09, 2022 9:27 pm    Post subject: Reply with quote

Yeah, mystery solved but what a terrible solution! And a huge waste of electricity and processing time compiling targets that I will never use or need.

So, does LLVM_TARGETS just do nothing now?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22778

PostPosted: Wed Mar 09, 2022 10:15 pm    Post subject: Reply with quote

As Ionen described, this was an unfortunate necessity due to other packages breaking in response to custom LLVM_TARGETS values. If those breaks were fixed, the flag could probably be made tunable again.
Back to top
View user's profile Send private message
sam_
Developer
Developer


Joined: 14 Aug 2020
Posts: 2008

PostPosted: Thu Mar 10, 2022 3:04 am    Post subject: Reply with quote

nagora wrote:
Yeah, mystery solved but what a terrible solution! And a huge waste of electricity and processing time compiling targets that I will never use or need.

So, does LLVM_TARGETS just do nothing now?


They also don't do nothing. You can unforce the targets if you wish in /etc/portage. See bug 767700.

As for news, maybe, although there's a tendency to prefer them for actionable things. In most cases here, no action is actually suggested. But I think it probably would've been worth it on balance because of confusion if nothing else. We still could but someone would need to write it up.
Back to top
View user's profile Send private message
figueroa
Advocate
Advocate


Joined: 14 Aug 2005
Posts: 3007
Location: Edge of marsh USA

PostPosted: Thu Mar 10, 2022 4:15 am    Post subject: Reply with quote

Sam, how about: "To prevent breakage of other core packages, a change has been made to now stable sys-devel/llvm- 13.0.1 to force ON all LLVM_TARGETS. Users' custom LLVM_TARGETS are ignored. No action is required on the part of the user. For further information, users may refer to the discussion in the following forum thread:
https://forums.gentoo.org/viewtopic.php?p=8694331
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi


Last edited by figueroa on Wed Jul 27, 2022 3:52 pm; edited 1 time in total
Back to top
View user's profile Send private message
xavier10
Guru
Guru


Joined: 19 Jan 2004
Posts: 485
Location: Paris, France

PostPosted: Wed Jul 27, 2022 9:57 am    Post subject: Reply with quote

Useful discussion, I was looking just for this information.

I was looking for ideas to cut down llvm/clang compilation time and initially thought reducing the list of targets might be a good idea, but I see it is not; is there any other way to reduce the emerge time that I could pursue ?
Back to top
View user's profile Send private message
rogge
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2006
Posts: 135
Location: Erfurt

PostPosted: Sat Jul 30, 2022 2:09 pm    Post subject: Reply with quote

Thanks a lot for explanation!
Back to top
View user's profile Send private message
njsg
Tux's lil' helper
Tux's lil' helper


Joined: 17 Dec 2005
Posts: 88

PostPosted: Mon Jul 24, 2023 1:28 pm    Post subject: Reply with quote

Is rust-bin also affected by this ABI incompatibility?
Back to top
View user's profile Send private message
RIA77
Guru
Guru


Joined: 24 Feb 2016
Posts: 391

PostPosted: Tue Aug 15, 2023 10:17 am    Post subject: Reply with quote

Why there is no LLM targets respected ? I have still the same problem.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22778

PostPosted: Tue Aug 15, 2023 12:25 pm    Post subject: Reply with quote

RIA77 wrote:
Why there is no LLM targets respected ? I have still the same problem.
Your question is answered by multiple people just a few posts up. In short, because respecting LLVM_TARGETS caused other packages to exhibit bad failure modes, so the targets are now forced enabled to avoid that. As I speculated last year when this was asked, that force might be removed if the downstream packages were changed to better handle changes in LLVM_TARGETS.
Back to top
View user's profile Send private message
Ionen
Developer
Developer


Joined: 06 Dec 2018
Posts: 2874

PostPosted: Tue Aug 15, 2023 1:51 pm    Post subject: Reply with quote

Can note that the ABI issues are still valid today and people that un-force LLVM_TARGETS sometime still end up with a broken llvm toolchain (incl. rust if USE=system-llvm) depending on ordering, not that this is a problem if you can fallback to gcc+binutils and rebuild the broken bits (doing this on a llvm profile without gcc could lead to a broken system that will need binpkgs to recover).

If toolchain is broken, it will manifest itself as undefined symbols in clang or lld themselves, and things like firefox[clang] will fail to build. You just need to rebuild lld and clang in that situation, and maybe rust.

But after that's sorted, it should never come back as an issue and you can safely enjoy quicker llvm builds (fine to have less targets on rust too, or USE=system-llvm is fine as well after it is rebuilt -- albeit may as well use rust-bin if goal is to shorten build time)

If understand the risks and in case process is still not clear -- essentially need to revert what /var/db/repos/gentoo/profiles/base/package.use.force does for llvm targets which can be done using /etc/portage/profile/package.use.force (note that this is under the profile/ directory, not top level), aka with:
Code:
*/* LLVM_TARGETS: -AArch64 -AMDGPU -ARM -AVR -BPF -Hexagon -Lanai -LoongArch -MSP430 -Mips -NVPTX -PowerPC -RISCV -Sparc -SystemZ -VE -WebAssembly -XCore
(this may need updates as new targets are added, X86 is left forced as it'd be a bad idea for most)

After that you can set what you need using /etc/portage/package.use normally, and note you can use -* there (unlike in package.use.force) e.g.:
Code:
*/* LLVM_TARGETS: -* AMDGPU NVPTX
If you use NVIDIA, may want to enable NVPTX given can be used for some cuda/ffmpeg stuff, and AMDGPU is currently hard required by mesa's ebuild with USE=llvm even if you don't use an amd gpu (note USE=-llvm typically means gimped mesa, albeit with nvidia-drivers it'd at worst just means losing llvmpipe software rendering, unsure for consequences with Intel gpus or Nouveau but I don't think it should be disabled given they can use gallium and llvm is used with it).

If you ever find yourself needing dev-lang/zig which need most llvm targets to be enabled, you can always fallback to dev-lang/zig-bin that comes with its own llvm. Some packages need BPF to be enabled too, and there's chances there's missing dependencies going around given not many test with targets disabled anymore.


Last edited by Ionen on Tue Aug 15, 2023 10:49 pm; edited 2 times in total
Back to top
View user's profile Send private message
RIA77
Guru
Guru


Joined: 24 Feb 2016
Posts: 391

PostPosted: Tue Aug 15, 2023 2:29 pm    Post subject: Reply with quote

Quote:
Mystery solved.

It would be better if there's some info about this change at the end of emerge.

Оbviously it's not mystery solved. That's reason why I asked.
And also I didn't know if it's safe to force disabling targets or not. Thank you Lonen for clarifying.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum