[SOLVED] job parallelism reduced because of low free space

[devsk]

The latest update to portage as of today spits out this:

Code: Select all

--- /var/tmp/portage has not enough free space, emerge job parallelism reduced. free: 25.25 GiB, required 27.00 GiB

I thought, ok, maybe the requirements have changed, let me increase this. I increased it to 35GB but then it spits out this:

Code: Select all

--- /var/tmp/portage has not enough free space, emerge job parallelism reduced. free: 34.77 GiB, required 37.50 GiB

What is going on? Why is portage not happy and content? When is enough not enough?

[devsk]

On a serious note, how is it calculating the required number? Can I have it bypass this check?

[Hu]

What is the output of emerge --info? Knowing when you synced and what version of Portage is under discussion may guide the answers you receive.

[devsk]

Code: Select all

# emerge --version
Portage 3.0.74 (python 3.12.12-final-0, default/linux/amd64/23.0/split-usr/desktop/plasma, gcc-14, glibc-2.42-r3, 6.17.13-gentoo x86_64)

Its mostly an up to date system. 3.0.74 was pulled in this morning. I think this is a new feature in 3.0.74 to avoid killing the system with OOM kicking in or running out of disk space on /var/tmp/portage while emerging in parallel. But looks like its calculating the required space wrongly as is evident from the output I attached above.

[Banana]

Do you run it on a virtual machine? Maybe there is something not right...

[sam_]

portage-3.0.74 (2025-01-01)
--------------
[...]

* Monitor free space in PORTAGE_TMPDIR (bug #934382).

Introduce a new option --jobs-tmpdir-require-free-gb= to control the
free space required in PORTAGE_TMPDIR.

This is needed to better support FEATURES="merge-wait" which results in
more packages in a pending state in PORTAGE_TMPDIR.

If the limit is reached, parallelism is reduced and the merge-wait
queue is flushed.

[devsk]

@sam_: do you know how it calculates the "required" number? It looks like it has a bug. It came up with a number and when I increased that number with 10GB, it just set the "required" above that.

There is no combination of packages that have needed more than 25GB in last 2 decades of my gentoo use. And suddenly, even 35GB is not enough! Very sus!

[Hu]

There is no combination of packages that have needed more than 25GB in last 2 decades of my gentoo use. And suddenly, even 35GB is not enough! Very sus!

Some packages can exceed 25GiB. Building clang with -ggdb can do this.

[devsk]

But I am not building clang with -ggdb...

I haven't changed anything in my system. And like I said, I have never run out of space in /var/tmp/portage for last 2 decades.

I think you failed to notice that when the /var/tmp/portage size is 25GB, it said it needs 27GB and when the FS size is 35GB, it said it needs 37GB. If it needed 27GB, why is it complaining when it has 35GB? Without building any big package, its not happy that the FS got past its initial request of 27GB by 8GB.

Clearly, the newly added code in portage-3.0.74 has a bug or behavior which is not understood.

[logrusx]

Reduce number of parallel emerge jobs. Or set the parameter Sam mentioned.

And post that emerge --info at least. Don't give us bits and pieces, that's annoying.

Best Regards,
Georgi

[devsk]

anyways...

Code: Select all

def scale_to_jobs(num):
                        # The newly started job is fully taken into account.
                        res = num 
                        # All currently running jobs are taken into account with less weight,
                        # since it is likely that they are already using space in PORTAGE_TMPDIR.
                        for i in range(2, running_job_count + 2):
                            res += (1 / i) * num
                        return res

Basically, by default, it starts at 18GB required (worst case because of chromium) and then keeps adding to it at 9+6+4.5+3.6+3+2.6+2.2+2+1.8....as running jobs increase. That explains the behavior above! Sure!

I think that's aggressive but at least I now know what its doing to get that number and how to avoid this calculation.

[devsk]

Why I think its overtly aggressive is because if chromium is emerging at 18GB, the load average is going to be such that no other package requiring 9GB of space is going to be emerge'ing at the same time because its going to be a heavy weight if its 9GB.

In my run of the world update, it used less than 2 GB of space but kept complaining, implying that the heuristic based on worst case is punishing regular, low space users by reducing their parallelism.

That's a sign of badly written algorithm!

I think the answer maybe to scale the space required with the size of download for each package in the emerge set. The set of packages and their sizes are known, we can multiply by a factor of 10 to account for tar.xz to expanded disk usage. That way if it is chromium being emerged, you know you need to start at 20GB but if its just some small 50 odd python packages, you don't need to reduce the parallelism!

[sam_]

You're free to file a bug report describing your circumstances and a suggested improvement. I'll note that once the space is there again, when a job is next requested, it'll allocate it. We generally don't expect a huge number of packages to be in the merge-wait queue nor do we want that queue to be large, as that itself inhibits more parallelism.

[Hu]

But I am not building clang with -ggdb...

Currently, you are not. My point was that there exist some extremely large packages, and it's not even that hard to drive them into serious failure modes. Thus, I appreciate why Portage gained this feature.

I think you failed to notice that when the /var/tmp/portage size is 25GB, it said it needs 27GB and when the FS size is 35GB, it said it needs 37GB. If it needed 27GB, why is it complaining when it has 35GB? Without building any big package, its not happy that the FS got past its initial request of 27GB by 8GB.

No, I saw that, but I have not looked at the Portage code involved, and had nothing helpful to say about the calculation.

Why I think its overtly aggressive is because if chromium is emerging at 18GB, the load average is going to be such that no other package requiring 9GB of space is going to be emerge'ing at the same time because its going to be a heavy weight if its 9GB.

Load average is inherently a lagging warning though. Suppose you had emerge set at --jobs=2 --load-average=4, and there are exactly 2 packages available to start: chromium and clang, both of which are known to be huge. Since neither has started yet, load average is still sitting near system idle, so your --load-average parameter will not prevent starting both at once. Absent this feature, Portage would then start both, and yes, once they get past their configure stage and start creating compile jobs, load average will go up, but by then, it's too late for Portage to realize that starting both at once was a bad idea. With this feature, assuming the calculation is decent[1], Portage would recognize the upcoming problem and avoid starting both.

I think the answer maybe to scale the space required with the size of download for each package in the emerge set. The set of packages and their sizes are known, we can multiply by a factor of 10 to account for tar.xz to expanded disk usage. That way if it is chromium being emerged, you know you need to start at 20GB but if its just some small 50 odd python packages, you don't need to reduce the parallelism!

This is an interesting idea, but a flat multiply by 10 seems overly optimistic to me. Going back again to my example of clang, the main source archive for clang-20.1.8 is only 141M, which your multiplier would predict to need ~1.41G. However, when built with -ggdb (which, again, uses up a lot of extra space due to debug symbols), clang easily exceeds 25G before it finishes building. Therefore, we need either to stipulate that there are user configurations for which this feature will be too optimistic[2] (and therefore it allows builds that an ideal algorithm would prohibit) or we need the feature to be more clever, whether by having a much more pessimistic multiplier, by examining more user state to try to pick a situation-appropriate multiplier, or by using some other algorithm.

[1]: I read the code snippet pasted here, but have not tried to evaluate whether this is a good choice of algorithm.
[2]: It's fine if this feature doesn't cover everyone. Building clang with full debug symbols is probably pretty unusual, and I have no complaints if the Gentoo developers want to take the position that this feature only covers people using the recommended simple flags of -O2 -march=something. I know to be careful when I'm building clang with debug symbols.

[sirlark]

Just throwing in my 2c here ...

(a) I'm also seeing this problem. What's particularly confusing is that every time I up the size of the tmpfs, the requirements go up to just exceed it?

(b) some way to override the reduction in parallelism either in make.conf or at emerge time.

[Hu]

As I read the help text for the new option, adding --jobs-tmpdir-require-free-gb 0 to the command line should inhibit the new feature:

Code: Select all

        "--jobs-tmpdir-require-free-gb": {
            "help": "Specifies the required remaining capacity (in GiB) of PORTAGE_TMPDIR before a new emerge job is started. Set to 0 to disable this check",

[devsk]

There is no way to upvote or mark the post as the solution in gentoo forums (*). But the solution was presented above by @sam_ with this option that @Hu mentioned, and I posted the code snippet around this option to further elaborate what's missing.

So, @sirlark, I hope the above post from @Hu helps. This is what I ended up doing and is the reason why this thread is marked as SOLVED.

(*) Is there an effort to move to more advanced forums, like the rest of the world, any time soon?

[Hu]

Yes, there are several threads over the last few years discussing changing the forum software. Forums Upgrade Status 2024 and Mobile support for forums? discuss this, at least in part.

[sirlark]

Yeah, --jobs-tmpdir-require-free-gb=0 is good enough for me for now, thanks @Hu and @devsk

[eeckwrk99]

Just got

Code: Select all

--- /var/tmp/portage has insufficient free space, emerge job parallelism reduced. free: 16.00 GiB, required 27.00 GiB

message today as well due to portage 3.0.77-r1 stabilization.

My first idea was to increase tmpfs size but then I found this thread and apparently the size requirements keep growing so I guess it's not worth it?

Is

Code: Select all

--jobs-tmpdir-require-free-gb=0

advisable or should we just ignore the warning?

[logrusx]

What's particularly confusing is that every time I up the size of the tmpfs, the requirements go up to just exceed it?

It's inspecting the first batch of packages and tells you nope, I can't handle all those packages in parallel. One of two things happen:

• [1]the space portage perceives necessary to build the first batch of packages exceeds the size of the partition => reduced parallelism.
  [2]the number of the packages reaches the limit of parallel emerge jobs without the required space exceeding the size of the partition => full throttle.

It just informs you that your tmp partition sill doesn't have enough space and prints the value at which the space required exceeds the size. Your mistake is you increased it just above the printed value, while in reality there are more packages to be evaluated which contribute to space requirements. Once you increase the size above the threshold the last package in the batch scheduled for merge it won't increase requirements.

Here comes Szatox' suggestion above that portage should re-evaluate the space requirement at every job finishing so that it knows it if it needs to wait for more jobs to be finished before it can safely spawn the next.

Best Regards,
Georgi

[sirlark]

It just informs you that your tmp partition sill doesn't have enough space and prints the value at which the space required exceeds the size. Your mistake is you increased it just above the printed value, while in reality there are more packages to be evaluated which contribute to space requirements. Once you increase the size above the threshold the last package in the batch scheduled for merge it won't increase requirements.

Ah! THANK YOU. This was the really confusing part to me.

[radio_flyer]

I get this message occasionally now, and I appreciate that the devs added it to portage. Smart or not, I have 64G of memory with /var/tmp/portage mounted tmpfs at the default 32G, and a large backing swapfile just in case. Portage has been getting aggressive recently with pushing a lot of finished builds into the merge-wait queue, which sometimes pushed the active builds into the swapfile when it also tried to build multiple behemoth packages at once. It has now stopped doing that, and my cores are back to maxxing out doing compiles, not I/O. Maybe the calculations could be improved, or not, but I certainly don't intend to disable this new feature.

[Sargastic]

Hello,

First, yes, I know there is another thread with this very subject.

The idea to reduce parallel jobs when the temporary space is limited is a good thing, but it currently looks very heavy-handed and somehow unrelated to the real needs of the current emerge :

Code: Select all

$ emerge -u1 portage                                                              
                                                                                                                                                                                           
Calculating dependencies... done!                                                             
Dependency resolution took 11.73 s (backtrack: 0/20).                   
                                                                                              
>>> Verifying ebuild manifests                                                                
>>> Running pre-merge checks for sys-apps/portage-3.0.77-r3             
 * Determining the location of the kernel source code                   
 * Found kernel source directory:                                                             
 *     /usr/src/linux                                                                         
 * Found sources for kernel version:                                                          
 *     5.10.246-gentoo-LTS                                                                    
 * Checking for suitable kernel configuration options ...                                                                                                                             [ ok ]
>>> Emerging (1 of 2) dev-util/debugedit-5.2-r1::gentoo                 
>>> Jobs: 0 of 2 complete, 1 running                                    Load avg: 0.68, 0.36, 0.50
--- /var/tmp/portage has insufficient free space, emerge job parallelism reduced. free: 25.40 GiB, required 27.00 GiB
>>> Installing (1 of 2) dev-util/debugedit-5.2-r1::gentoo               
>>> Completed (1 of 2) dev-util/debugedit-5.2-r1::gentoo                
>>> Emerging (2 of 2) sys-apps/portage-3.0.77-r3::gentoo                
>>> Installing (2 of 2) sys-apps/portage-3.0.77-r3::gentoo              
>>> Completed (2 of 2) sys-apps/portage-3.0.77-r3::gentoo               
>>> Jobs: 2 of 2 complete                                               Load avg: 0.93, 0.47, 0.53

I'm only updating two packages, with a quite small space footprint, and still get parallel jobs reduced. This is IMHO wrong (even if, here, the time lost because of this reduction is nil).

The parallelism-reduction should somehow (i.e. no need to be perfect) take into account which packages are to be emerged, and this does not seem to be the case.

This has been merged from Job parallelism reduced because of low free space(1176893) into this topic job parallelism reduced because of low free space(1176493). -- Banana

[sam_]

You can turn it off with the option discussed above, and also discussed was the issue with taking into account packages (information isn't available, nor is it consistent across machines).

EDIT: Tweaked given thread was merged.