How to make temporary change to package.env?

[ipic]

This question arises for me because I want to experiment with building the kernel with LLVM, with a view to experimenting with a rust build in the future.
I have scripted the kernel build process, and that part is straight forward.
I checked that the LLVM/Rust tool chain is OK from the kernel point of view:

Code: Select all

~ # cd /usr/src/linux
/usr/src/linux # export LLVM=1
/usr/src/linux # make rustavailable
Rust is available!

My script then does the equivalent of this:

Code: Select all

~ # cd /usr/src/linux
/usr/src/linux # export LLVM=1
/usr/src/linux # make distclean
/usr/src/linux # zcat /proc/config.gz >.config
/usr/src/linux # make olddefconfig
/usr/src/linux # make menuconfig
/usr/src/linux # make -j64
/usr/src/linux # make modules_install

That works - and the script can either declare the LLVM=1 environment variable or not - depending on which tool chain I wish to use.
Since modules are involved - I know I can't mix two instances of the same kernel version this way - a single version has to be one or the other.

My question then relates to the next part of the kernel build, where this command is run:

Code: Select all

emerge --oneshot --jobs @module-rebuild

I have set up a Clang profile in /etc/portage/env:

Code: Select all

~ # cat /etc/portage/env/compiler-clang 
# Normal settings here
COMMON_FLAGS="-O2 -march=native"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"

CC="clang"
CPP="clang-cpp" # necessary for xorg-server and possibly other packages
CXX="clang++"
AR="llvm-ar"
NM="llvm-nm"
RANLIB="llvm-ranlib"

From the documentation I can find, if I want modules in @module-rebuild to match the kernel tool chain, I have to place an entry in /etc/portage/env/package.env for each package, specifying use of compiler-clang.

There are two problems with this from my point of view:
- I have to know what packages are in @module-rebuild and manually change the settings for each build
- I have to manually change the entries depending on whether I am using LLVM=1 or not.

So, the question is - can I temporarily override /etc/portage/env/package.env settings for an emerge invocation - so that all packages in that invocation use the same settings, without having to know which packages are being emerged?

[logrusx]

Perhaps do something in /etc/portage/bashrc. Or change thing manually. To my knowledge there's no space for scripting in package.env and limited scripting in env/category/package. Entries in package.env are processed by python and in env/category/package you can have variable expansion. Don't take my words to face value, I'm not an expert in that area. This is what I remember I once read in a bug that I can't find anymore.

Best Regards,
Georgi

[pingtoo]

You can try

Code: Select all

COMMON_FLAGS="-O2 -march=native" \
CFLAGS="-O2 -march=native" \
CXXFLAGS="-O2 -march=native" \
CC="clang" \
CPP="clang-cpp" \
CXX="clang++" \
AR="llvm-ar" \
NM="llvm-nm" \
RANLIB="llvm-ranlib" \
emerge --oneshot --jobs @module-rebuild

[ipic]

Perhaps do something in /etc/portage/bashrc. Or change thing manually. To my knowledge there's no space for scripting in package.env and limited scripting in env/category/package. Entries in package.env are processed by python and in env/category/package you can have variable expansion. Don't take my words to face value, I'm not an expert in that area. This is what I remember I once read in a bug that I can't find anymore.

Best Regards,
Georgi

I'd forgotten about /etc/portage/bashrc - thanks for pointing it out.

Interestingly, I have an entry in there that says this:

Code: Select all

# This hook is neccesary for automatic updating of the cfg-update index, please do not modify it!
pre_pkg_setup() {
	[[ $ROOT = / ]] && cfg-update --index
}

I guess cfg-update placed that there, although the file is not owned by cfg-update.

I see this statement in the Gentoo WiKi page for /etc/portage/bashrc (at https://wiki.gentoo.org/wiki//etc/portage/bashrc)

In order to execute manually-specified code during an ebuild, there are two distinct ways to hook into the ebuild process: via hook functions, and via environment variables.

In the context of /etc/portage/bashrc, hook functions should be used, as this file will be sourced multiple times during the build - at least once per phase, and several other times as well...

That seems to be saying don't set environment variables in this file - which seems to rule out doing

Code: Select all

export CC="clang"
...etc

in pre_pkg_setup

[ipic]

You can try

Code: Select all

COMMON_FLAGS="-O2 -march=native" \
CFLAGS="-O2 -march=native" \
CXXFLAGS="-O2 -march=native" \
CC="clang" \
CPP="clang-cpp" \
CXX="clang++" \
AR="llvm-ar" \
NM="llvm-nm" \
RANLIB="llvm-ranlib" \
emerge --oneshot --jobs @module-rebuild

Thanks for the suggestion.
Not being a portage/bash expert, does the pre-existence of environment variables stop emerge from going off and applying it's own values from make.conf?

If the pre-set values are used, would this work?

Code: Select all

source /etc/portage/env/compiler-clang
emerge --oneshot --jobs @module-rebuild

That would be neat (if it works), using the same setup as for other Clang builds.

[pingtoo]

You can try

Code: Select all

COMMON_FLAGS="-O2 -march=native" \
CFLAGS="-O2 -march=native" \
CXXFLAGS="-O2 -march=native" \
CC="clang" \
CPP="clang-cpp" \
CXX="clang++" \
AR="llvm-ar" \
NM="llvm-nm" \
RANLIB="llvm-ranlib" \
emerge --oneshot --jobs @module-rebuild

Thanks for the suggestion.
Not being a portage/bash expert, does the pre-existence of environment variables stop emerge from going off and applying it's own values from make.conf?

If the pre-set values are used, would this work?

Code: Select all

source /etc/portage/env/compiler-clang
emerge --oneshot --jobs @module-rebuild

That would be neat (if it works), using the same setup as for other Clang builds.

I know using single line will work (the example I post) I have not try source a file. But it should work the same as the simple line. You just need to be careful when using source command, from that point onward those variables are set, so if you have follow up emerge for something else you may need to unset those variable prior to run emerge.

apparently emerge favour run time environment over configuration file, i.e. env/package.env.

[Genone]

That seems to be saying don't set environment variables in this file - which seems to rule out doing

Code: Select all

export CC="clang"
...etc

in pre_pkg_setup

You can set env variables there. But as the file is sourced multiple times during the build (and maybe even for each phase) you should keep the code in global scope to a minimum. Also you should not set variables there that have any special meaning to portage itself as that will lead to undefined/misleading behavior. For example setting CFLAGS is technically ok, but the value you set in bashrc will not be picked up by emerge --info (one reason why package.env exists) and may bypass sanity checks/filters performed by the ebuild or eclasses. That is why this is generally discouraged in favor of package.env.

Thanks for the suggestion.
Not being a portage/bash expert, does the pre-existence of environment variables stop emerge from going off and applying it's own values from make.conf?

No, settings from the calling environment will usually override settings from config files.

If the pre-set values are used, would this work?
Code:

source /etc/portage/env/compiler-clang
emerge --oneshot --jobs @module-rebuild

That would be neat (if it works), using the same setup as for other Clang builds.

Yes, but keep in mind that the values set by the source command will persist in that bash session after the emerge job has finished. So I would not recommend doing it that way, better just do a full script for it.

[ipic]

I ran a few test builds with the following settings:
- just run emerge @module-rebuild
- run emerge @module-rebuild with entry for the module in /etc/portage/env/package.env tagging compiler-clang
- run `source /etc/portage/env/compiler-clang ; energe @module-rebuild
- run the command as @pingtoo suggested

In ALL of these cases, the build output is unchanged, and shows this:

Code: Select all

>>> Emerging (1 of 1) app-admin/ryzen_smu-0.1.5::gentoo
 * ryzen_smu-v0.1.5.tar.bz2 BLAKE2B SHA512 size ;-) ... 
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found sources for kernel version:
 *     6.13.3-gentoo-r1
 * Checking for suitable kernel configuration options ...
 * Preparing x86_64-pc-linux-gnu toolchain for kernel modules (override with KERNEL_CHOST) ...
 * Toolchain picked for kernel modules (override with KERNEL_CC, _LD, ...): '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang-19' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang++-19' '/usr/lib/llvm/19/bin/ld.lld' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-ar' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-nm' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objcopy' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objdump' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-readelf' '/usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-strip'
>>> Unpacking source...

Also, in ALL these cases the module builds, installs, and then loads correctly against the LLVM/Clang built kernel.

Looking at the selected tools on my system I see this:

Code: Select all

~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang-19
lrwxrwxrwx 1 root root 8 Feb  8 09:13 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang-19 -> clang-19
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang++-19
lrwxrwxrwx 1 root root 10 Feb  8 09:13 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-clang++-19 -> clang++-19
~ # ls -lh /usr/lib/llvm/19/bin/ld.lld
-rwxr-xr-x 1 root root 23K Feb 11 10:45 /usr/lib/llvm/19/bin/ld.lld
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-ar
lrwxrwxrwx 1 root root 7 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-ar -> llvm-ar
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-nm
lrwxrwxrwx 1 root root 7 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-nm -> llvm-nm
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objcopy
lrwxrwxrwx 1 root root 12 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objcopy -> llvm-objcopy
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objdump
lrwxrwxrwx 1 root root 12 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-objdump -> llvm-objdump
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-readelf
lrwxrwxrwx 1 root root 12 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-readelf -> llvm-readelf
~ # ls -lh /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-strip
lrwxrwxrwx 1 root root 10 Dec  7 12:55 /usr/lib/llvm/19/bin/x86_64-pc-linux-gnu-strip -> llvm-strip
~ # 

It thus appears that, without any additional external 'help', a portage kernel module package will select the tool chain that matches the kernel source it is compiling/linking against.
A quick search for 'Gentoo KERNEL_CHOST' doesn't turn up anything useful, so not sure what format it would need to be to replicate the env settings I have.

I'm not sure how robust the automatic selection done by portage will be, given I'm not sure what is causing those symlinks to be placed in usr/lib/llvm/19/bin

However, given it appears to 'Just Work' (trademark) - I'm going to go with it. Makes my kernel script much simpler

[ipic]

Yes, but keep in mind that the values set by the source command will persist in that bash session after the emerge job has finished. So I would not recommend doing it that way, better just do a full script for it.

Good point.

In my case, the commands are being issued from a python scrip, so each command gets its own environment setup as inherited from the calling task.

However, as above, turns out portage is way cleverer than I gave it credit for.