Can multiple emerge commands be run at the same time?
Message
- CanadianOtaku
- n00b
- Posts: 20
- Joined: Tue Apr 13, 2021 2:49 am
- Location: Canada
Can multiple emerge commands be run at the same time?
I remember when I was using arch pacman didn't allow that.
Yes, I have run emerge in different VT's simultaneously. Caution: If the emerge's want to install the same thing or the same dependencies, each will calculate what needs to be done only once depending on the predicate of your command.
Example if you are building @world in one VT then start @system in the other, you may well be emerging the same rging twice, say if calculate gcc needs updating in the first and the second, both will emerge gcc independently. Portage only checks once for each invocation.
But this can be useful. Say you are building an update to @world with --keep-going. Package "packageA" fails. You fix this by changing accept-keywords or masking the a stable version or unmasking an unstable version. You can then emerge the fixed package in a second window. But if packageA wants dependencies that are already scheduled in the first window, then both windows will emerge the same dependencies. If the duplicates are something like gcc, rust or webkit you are doing quite a bit of heavy consumption of resources that will impact the time that the first update takes. This is best done if your second (or third or fourth) emerge updates/fixes something that will impact the first emerge.
Normally, you wouldn't do this. Just adjust --jobs or something else.
Example if you are building @world in one VT then start @system in the other, you may well be emerging the same rging twice, say if calculate gcc needs updating in the first and the second, both will emerge gcc independently. Portage only checks once for each invocation.
But this can be useful. Say you are building an update to @world with --keep-going. Package "packageA" fails. You fix this by changing accept-keywords or masking the a stable version or unmasking an unstable version. You can then emerge the fixed package in a second window. But if packageA wants dependencies that are already scheduled in the first window, then both windows will emerge the same dependencies. If the duplicates are something like gcc, rust or webkit you are doing quite a bit of heavy consumption of resources that will impact the time that the first update takes. This is best done if your second (or third or fourth) emerge updates/fixes something that will impact the first emerge.
Normally, you wouldn't do this. Just adjust --jobs or something else.
- NeddySeagoon
- Administrator
- Posts: 56077
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
CanadianOtaku,
As others have said, it can be done but depending on what you want to do, it may be wasteful.
Portage can do it itself too. See the --jobs parameter to emerge.
A word of warning. Your MAKEOPTS setting applies to each build that is runing concurrently.
With small packages its not a problem. With large C++ projects, allow 2G RAM per concurrent make thread.
With MAKEOPTS="-j8" portage will use about 16G RAM on a big C++ project.
With emerge --jobs=4 ... and MAKEOPTS="-j8" it may need 64G RAM.
There are other factors to consider. Once you get to using swap. things get very slow. So stay out of swapping. zswap can delay the onset of swapping but consumes CPU time.
Then there is CPU cache thrashing. As builds progress with more parallel make threads, so more and more cache lines that could remain resident are evicted. The must be fetched from the much slower main RAM, when the CPU needs them again.
I tend to run emerge with one job in one window and fix things that break, including building, one packages at a tine in another.
I
As others have said, it can be done but depending on what you want to do, it may be wasteful.
Portage can do it itself too. See the --jobs parameter to emerge.
A word of warning. Your MAKEOPTS setting applies to each build that is runing concurrently.
With small packages its not a problem. With large C++ projects, allow 2G RAM per concurrent make thread.
With MAKEOPTS="-j8" portage will use about 16G RAM on a big C++ project.
With emerge --jobs=4 ... and MAKEOPTS="-j8" it may need 64G RAM.
There are other factors to consider. Once you get to using swap. things get very slow. So stay out of swapping. zswap can delay the onset of swapping but consumes CPU time.
Then there is CPU cache thrashing. As builds progress with more parallel make threads, so more and more cache lines that could remain resident are evicted. The must be fetched from the much slower main RAM, when the CPU needs them again.
I tend to run emerge with one job in one window and fix things that break, including building, one packages at a tine in another.
I
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
THIS.NeddySeagoon wrote:I tend to run emerge with one job in one window and fix things that break, including building, one packages at a tine in another.
When I referred to having three or four VT's running emerge together those were @world updates with like a hundred packages being updated (sequentially) in the first window and the others fixing packages that failed. I usually run with "--keep-going" so that little packages can complete while I'm figuring out how to fix a failure. Those small libraries and perl/python packages can keep chugging merrily on. You could do N independent packages simultaneously but betteer to put them on the same line (invocation) so portage can schedule them in order and parallel them with --jobs.
- DeadlySyns
- n00b
- Posts: 8
- Joined: Fri May 14, 2021 1:31 pm
Do people still use swap? I have not had a swap partition since ~2010 now that RAM is so cheap per GB. Even for scientific computing (where speed is more critical), I have found not many set up a swap partition.NeddySeagoon wrote:CanadianOtaku,
Once you get to using swap. things get very slow. So stay out of swapping. zswap can delay the onset of swapping but consumes CPU time.
Then there is CPU cache thrashing. As builds progress with more parallel make threads, so more and more cache lines that could remain resident are evicted. The must be fetched from the much slower main RAM, when the CPU needs them again.
I tend to run emerge with one job in one window and fix things that break, including building, one packages at a tine in another.
I
Even still, swapping on a NVME is going to be lightyears faster than swapping onto a spinning SATA or SCSI/FC disk.
- NeddySeagoon
- Administrator
- Posts: 56077
- Joined: Sat Jul 05, 2003 9:37 am
- Location: 56N 3W
DeadlySyns,
Yes, you should still have swap space.
Swap space is used only for dynamically allocated RAM. Not having swap does not prevent swapping, it only robs the kernel of one of its swap options.
The kernel can flush dirty buffers and drop them to free RAM.
The kernel can drop clean buffers at any time and reload them from permanent storage as required.
Clean buffers includes code that you may want to execute is a few microseconds.
Due to the way Linux manages Virtual Memory, pages can be mapped but not resident in physical RAM.
Any attempt to access such a page generates a page fault and a context switch so another process can run, while the non resident page is loaded.
When the original process runs again all is well.
This mechanism allows Linux to execute programs that are too large to fit in available RAM all at once.
Yes, you should still have swap space.
Swap space is used only for dynamically allocated RAM. Not having swap does not prevent swapping, it only robs the kernel of one of its swap options.
The kernel can flush dirty buffers and drop them to free RAM.
The kernel can drop clean buffers at any time and reload them from permanent storage as required.
Clean buffers includes code that you may want to execute is a few microseconds.
Due to the way Linux manages Virtual Memory, pages can be mapped but not resident in physical RAM.
Any attempt to access such a page generates a page fault and a context switch so another process can run, while the non resident page is loaded.
When the original process runs again all is well.
This mechanism allows Linux to execute programs that are too large to fit in available RAM all at once.
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Also, although not relevant to the problem at hand, swap is also necessary if you want your system to hibernate. Suspend-to-RAM has gotten much better, but there are some cases where suspend-to-disk is still a better choice, such as a laptop that you will unplug overnight, or any system where you expect wall power to fail. A swap partition is necessary for storing that suspended state.