Can dev-build/steve map token to memory usage?

[fpemud]

From <<2025 in retrospect>>, this is really a good functionality:

We have created steve, an implementation of a token-accounting system-wide jobserver, and introduced experimental global jobserver support in Portage.

The tokens in dev-build/steve is for allocating CPU.
However, sometimes memory is the bottleneck during the compilation process.

I think it would be good that memory usage can also be tokenized.

For example, here are the memory consumption figures I've recorded:
# each job consumes 1.6G memory on average when compiling net-libs/webkit-gtk
# each job consumes 2G memory on average (in fact 2G for GCC and 1.5G for Clang according to ebuild) when compiling dev-qt/qtwebengine
# each job consumes 3G memory on average when compiling www-client/chromium
# each job consumes 2G memory (we guess) on average when compiling sci-libs/composable-kernel

If I have 32 CPU core and 64G memory, job number should be limited as only 64G/3G=21 when compiling www-client/chromium.

[Navar]

Interesting, wasn't aware of this package.

I think pre-estimating memory constraints are too problematic to attempt and an additional headache for the devs. You could maybe arguably have optional variables in the particular related ebuilds making these assumptions, and possible user overrides with portage evaluating how much available free, inclusive of swap, is available before attempting to risk forking off another job to emerge based upon what the user said was "fine" vs reality of what is currently seen now in the working environment.

There's a lot of things to consider and inconsistency in build systems. Chromium being a great example. It used to average a solid 2G per job in prior years and perhaps more when they allowed jumbo build option. However, my more recent experience for years has been:

An AMD 5950x system that I don't think touches anywhere near 64GB of the available system ram being compiled at -j32, but still uses a significant amount due to massive C++ templating utilized in that huge code base. It takes 4x as long to build here as it did a few years ago on same, but I'm still usually <4hrs. I suspect that's because the memory utilization, outside of end user control, has been adjusted by upstream in many ways, especially the elimination of jumbo builds.

The same build on an older i9 Sandy Bridge -j8 CPU with 32GB laptop won't get anywhere near using even half the available RAM. It used to in the past, sure. It also used to build much faster. Now, while I can get 100% CPU coverage, I can't even begin to make use of that RAM for assisting the build, including as a /var/tmp/portage tmpfs ramdisk for building, because 16G+ in size won't cover the build needs anymore. The build takes a rediculous amount of time and I end up stop and restarting via ebuild compile, etc. segmentation to resume existing work on actual SSD filesystem space due to the aforementioned issues. It's a multi-day affair that takes longer to build than the entire rest of the OS.

Why the two behave differently that way? I cannot say, perhaps ninja related, or clang/llvm/rust related, who knows.

That's not a Gentoo can really fix problem, that's a Google AI bloat boat caused issue. Usually what I do is --exclude the problematic big packages if I want to let portage try to go a bit crazy in speeding up the other 100+ smaller packages in the way, since the majority of time is more or less spent idle in the process:

Code: Select all

# aggressive
MAKEOPTS="-j32 -l64"
EMERGE_DEFAULT_OPTS="--jobs=32 --load-average=64 --quiet-build=y"

Then another update run either utilizing sane no additional jobs options, or carefully remove a package at a time on the prior exclude entry list until all done. Add in swap so you don't OOM in guestimations. Load average has never been my problem, memory resource is.

[sam_]

steve lets you control the amount of free memory before handing out another job, which should let you approximate this:

Code: Select all

$ stevie --help
usage: stevie [options] <argv>...

options:
    --help, -h             print this help message
    --version, -V          print version
    --jobserver PATH, -s PATH
                           jobserver FIFO path (default: /dev/steve)

other actions (executed before the command):
    --get-tokens, -t       print available token count
    --get-jobs, -j         print total job number
    --set-jobs JOBS, -J JOBS
                           set total job number
    --get-load-average, -l
                           print max load-average
    --set-load-average LOAD_AVG, -L LOAD_AVG
                           set max load-average
    --get-load-recheck-timeout, -r
                           print load-recheck-timeout
    --set-load-recheck-timeout TIMEOUT, -R TIMEOUT
                           set load-recheck-timeout
    --get-min-jobs, -m     print min-job number
    --set-min-jobs JOBS, -M JOBS
                           set min-job number
    --get-min-memory-avail, -a
                           print min. required available memory (in MIB)
    --set-min-memory-avail MIN_MEM_AVAIL, -A MIN_MEM_AVAIL
                           set min. required available memory (in MiB)
    --get-per-process-limit, -p
                           print per-process limit
    --set-per-process-limit LIMIT, -P LIMIT
                           set per-process limit

You can also change this at runtime with stevie (a client for the jobserver):

Code: Select all

$ stevie --help
usage: stevie [options] <argv>...

options:
    --help, -h             print this help message
    --version, -V          print version
    --jobserver PATH, -s PATH
                           jobserver FIFO path (default: /dev/steve)

other actions (executed before the command):
    --get-tokens, -t       print available token count
    --get-jobs, -j         print total job number
    --set-jobs JOBS, -J JOBS
                           set total job number
    --get-load-average, -l
                           print max load-average
    --set-load-average LOAD_AVG, -L LOAD_AVG
                           set max load-average
    --get-load-recheck-timeout, -r
                           print load-recheck-timeout
    --set-load-recheck-timeout TIMEOUT, -R TIMEOUT
                           set load-recheck-timeout
    --get-min-jobs, -m     print min-job number
    --set-min-jobs JOBS, -M JOBS
                           set min-job number
    --get-min-memory-avail, -a
                           print min. required available memory (in MIB)
    --set-min-memory-avail MIN_MEM_AVAIL, -A MIN_MEM_AVAIL
                           set min. required available memory (in MiB)
    --get-per-process-limit, -p
                           print per-process limit
    --set-per-process-limit LIMIT, -P LIMIT
                           set per-process limit

.. what's nice is, you could do that in /etc/portage/bashrc for certain packages, or in /etc/portage/env/www-client/chromium etc in pre_src_configure or similar.

Enhancements to the wiki page at https://wiki.gentoo.org/wiki/Steve are also welcome.

[Navar]

Cool, it literally implements what I mentioned , that's great! Kudos to the author(s)

[Ionen]

There's limits to how well something can handle this though, compiler jobs tend to slowly grow in memory usage and job tokens may be handed out before the usage shoots up all at once by some bad luck. So still need to estimate how much memory a job can use and not allow a "too big" total.

But still better than handing jobs when you're already running out because that one job decided to use 4GB rather than 200MB-2GB like others

[fpemud]

steve lets you control the amount of free memory before handing out another job

Wow, that's great. I'll try this.

[jordanp]

I've been trying out steve recently and it works well. However, I have the same issue with memory usage compiling certain packages. I tried a fairly large min memory setting, and as pointed out, a bunch of jobs will start and memory usage can still grow far past the limit. It would be nice if there was a way to limit jobs for certain ebuilds. I do see the setting for a per process limit, but I'd rather not limit all other packages. Currently, I have to make webkit-gtk not use the jobserver, so I can limit the number of jobs for it.

[sam_]

I've been trying out steve recently and it works well. However, I have the same issue with memory usage compiling certain packages. I tried a fairly large min memory setting, and as pointed out, a bunch of jobs will start and memory usage can still grow far past the limit. It would be nice if there was a way to limit jobs for certain ebuilds. I do see the setting for a per process limit, but I'd rather not limit all other packages. Currently, I have to make webkit-gtk not use the jobserver, so I can limit the number of jobs for it.

But you can! That's what I was suggesting above.

You can call stevie in pre_src_prepare or pre_src_configure or pre_src_compile in /etc/portage/bashrc or /etc/portage/env/www-client/chromium (or whatever), and then reset in post_src_compile to the old values.

That is, something like:

pre_src_compile() {
_JORDANP_BACKUP_JOBS=$(stevie --get-jobs)
stevie --set-jobs 5
}

post_src_compile() {
stevie --set-jobs ${_JORDANP_BACKUP_JOBS}
}

[jordanp]

Oh, cool. Sorry, I missed that part.