[SOLVED] job parallelism reduced because of low free space

[logrusx]

which sometimes pushed the active builds into the swapfile when it also tried to build multiple behemoth packages at once.

First, this is not how swap works. It's not a simple FIFO queue. Second, you can setup zswap in the kernel. Third, you can configure per-package tmp directory.
Best Regards,
Georgi

[axl]

I ran into this because I use ramfs instead of zram or tmpfs. It has a bit lower overhead. Slightly faster. But at the same time, can't report size. But I am pretty sure I have enough of it.

Aren't most options in portage opt-in vs opt-out?

I get the logic. But I am just noticing a change in spirit. I thought, failing is a normal outcome when you push limits. I mean a program failing, is something normal imho in gentoo. However portage holding your hand by default, that is something new. At least for me. EMERGE_DEFAULT_OPTS="--jobs-tmpdir-require-free-gb=0" will have to do.

I am not complaining that I had to spend a few minutes to look up what's going on here. And I found a fix. Its fine.

What bothers me, is that portage is making decisions for me, based on faulty data. It can't read how much free space I have where I compile stuff, and sure, my fault. But then it also "can't know" how big a package is. So it doesn't know how much space there is, how much was used, or how much it should be used. But it's making decisions by default. And maybe that is the direction portage should go into. I just never thought of it that way.

[sam_]

I ran into this because I use ramfs instead of zram or tmpfs. It has a bit lower overhead. Slightly faster. But at the same time, can't report size. But I am pretty sure I have enough of it.

Aren't most options in portage opt-in vs opt-out?

I get the logic. But I am just noticing a change in spirit. I thought, failing is a normal outcome when you push limits. I mean a program failing, is something normal imho in gentoo. However portage holding your hand by default, that is something new. At least for me. EMERGE_DEFAULT_OPTS="--jobs-tmpdir-require-free-gb=0" will have to do.

I am not complaining that I had to spend a few minutes to look up what's going on here. And I found a fix. Its fine.

What bothers me, is that portage is making decisions for me, based on faulty data. It can't read how much free space I have where I compile stuff, and sure, my fault. But then it also "can't know" how big a package is. So it doesn't know how much space there is, how much was used, or how much it should be used. But it's making decisions by default. And maybe that is the direction portage should go into. I just never thought of it that way.

This is a fair point and it's something I've been thinking about. I would generally like to make things a little bit friendlier where it is free (or very cheap) to do so, but I agree that this may be too sensitive with the current heuristics to be enabled by default. I'm still not sure yet.

The rationale here was to make it less likely for users to have parallel build failures (which can be not just a inconvenience but a big problem if it's in the middle of a large upgrade).

[logrusx]

Sam, I don't know how easy or hard would that be, but the temporary contents of the packages waiting to be merged can be deleted while the package is waiting. Only the image is necessary, right? Or is it not that simple?

I'm testing this feature now with a tmpfs temp partition and it works fine when big packages are not involved. Those are not that many, they can easily be configured to not use the tmpfs.

Also one more thing: the binary packages do not need that much space, can that be circumvented somehow?

Best Regards,
Georgi

[szatox]

The most funny thing is that the estimated required space does not even depend on the work queue.
I initially thought it did in a clunky way, but no, even very small packages want 27G out of 19 available. Just like the big packages. Go figure!

What bothers me, is that portage is making decisions for me, based on faulty data. It can't read how much free space I have where I compile stuff, and sure, my fault. But then it also "can't know" how big a package is.

What bothers you is portage _telling_ you it made a decision. And one that does not even change the end result it produces.
Do you know how resolving a dependency on alternatives works? E.g. anything virtual/* ? This one does actually affect what software you have installed, and yet we expect it to just work rather than keep asking about every single little thing.

[ewildgoose]

Thanks for noting the workaround:
--jobs-tmpdir-require-free-gb=0

I build images for Arm under QEMU and ...it's slow...

Getting portage to parallelise as much as possible is helpful, even then it's tough to get a load higher than 1-2...

If you haven't tried this, the issue is that portage is very very slow at installing things under Qemu. Like installing a large bunch of acct-user/xxx packages (which basically are empty packages), takes many 10s of seconds per package. So it's helpful to try and get parallelism as high as I can

The workaround is acceptable to me. Builds are reproducible so we can tweak flags for specific packages.

[dalek]

I have found in most cases, you can just ignore the message. It will stop for large packages where you don't have enough memory to compile, like emerge has always done. I just ignored this message in my case. The emerge continued on just fine and I couldn't tell any difference in speed of compiles.