Can multiple emerge commands be run at the same time?

CanadianOtaku · n00b Joined: 13 Apr 2021 Posts: 20 Location: Canada

I remember when I was using arch pacman didn't allow that.

fpemud · Guru Joined: 15 Feb 2012 Posts: 349

Yes. There's a lock in the install stage so that parallel emerge operation won't break the system.

Tony0945 · Posted: Sat May 15, 2021 6:44 am Post subject:

Yes, I have run emerge in different VT's simultaneously. Caution: If the emerge's want to install the same thing or the same dependencies, each will calculate what needs to be done only once depending on the predicate of your command.
Example if you are building @world in one VT then start @system in the other, you may well be emerging the same rging twice, say if calculate gcc needs updating in the first and the second, both will emerge gcc independently. Portage only checks once for each invocation.

But this can be useful. Say you are building an update to @world with --keep-going. Package "packageA" fails. You fix this by changing accept-keywords or masking the a stable version or unmasking an unstable version. You can then emerge the fixed package in a second window. But if packageA wants dependencies that are already scheduled in the first window, then both windows will emerge the same dependencies. If the duplicates are something like gcc, rust or webkit you are doing quite a bit of heavy consumption of resources that will impact the time that the first update takes. This is best done if your second (or third or fourth) emerge updates/fixes something that will impact the first emerge.

Normally, you wouldn't do this. Just adjust --jobs or something else.

NeddySeagoon · Posted: Sat May 15, 2021 11:36 am Post subject:

CanadianOtaku,

As others have said, it can be done but depending on what you want to do, it may be wasteful.
Portage can do it itself too. See the --jobs parameter to emerge.

A word of warning. Your MAKEOPTS setting applies to each build that is runing concurrently.
With small packages its not a problem. With large C++ projects, allow 2G RAM per concurrent make thread.

With MAKEOPTS="-j8" portage will use about 16G RAM on a big C++ project.
With emerge --jobs=4 ... and MAKEOPTS="-j8" it may need 64G RAM.

There are other factors to consider. Once you get to using swap. things get very slow. So stay out of swapping. zswap can delay the onset of swapping but consumes CPU time.
Then there is CPU cache thrashing. As builds progress with more parallel make threads, so more and more cache lines that could remain resident are evicted. The must be fetched from the much slower main RAM, when the CPU needs them again.

I tend to run emerge with one job in one window and fix things that break, including building, one packages at a tine in another.
I
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Tony0945 · Posted: Sat May 15, 2021 12:15 pm Post subject:

DeadlySyns · n00b Joined: 14 May 2021 Posts: 8

eccerr0r · Posted: Sat May 15, 2021 10:02 pm Post subject:

yes
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Sat May 15, 2021 10:26 pm Post subject:

DeadlySyns,

Yes, you should still have swap space.

Swap space is used only for dynamically allocated RAM. Not having swap does not prevent swapping, it only robs the kernel of one of its swap options.

The kernel can flush dirty buffers and drop them to free RAM.
The kernel can drop clean buffers at any time and reload them from permanent storage as required.
Clean buffers includes code that you may want to execute is a few microseconds.

Due to the way Linux manages Virtual Memory, pages can be mapped but not resident in physical RAM.
Any attempt to access such a page generates a page fault and a context switch so another process can run, while the non resident page is loaded.
When the original process runs again all is well.
This mechanism allows Linux to execute programs that are too large to fit in available RAM all at once.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21649

Also, although not relevant to the problem at hand, swap is also necessary if you want your system to hibernate. Suspend-to-RAM has gotten much better, but there are some cases where suspend-to-disk is still a better choice, such as a laptop that you will unplug overnight, or any system where you expect wall power to fail. A swap partition is necessary for storing that suspended state.