| View previous topic :: View next topic |
| Author |
Message |
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
 Posted: Mon Jan 06, 2020 9:38 pm Post subject: [SOLVED] emerge running multiple jobs at once? |
 |
|
So guys i am having a little bit of issues running jobs in parallel
I read the Gentoo wiki and its a bit confusing compared to most of their guides
https://wiki.gentoo.org/wiki/EMERGE_DEFAULT_OPTS
What i really need is some code examples to understand.
So when i try to set the EMERGE_DEFAULT_OPTS it does not seem to be having any effect. When i add --jobs to command line it seems to work , but not how i want.
What i want to do is be using all 32 threads at 90% load and fetching multiple packages at once.
These are examples of what i have tried and the results. These changes are in the make.conf file and terminal cli.
| Code: | | EMERGE_DEFAULT_OPTS="--jobs 32" |
no effect on emerge
| Code: | | emerge --ask --update --deep --with-bdeps=y --newuse @world --jobs=32 |
starts merging around 10 jobs at once, but than slowly drops down to 1 running job.
Can anyone give me some make.conf examples and explain to me what i am doing wrong?
Last edited by finalturismo on Thu Jan 09, 2020 7:40 am; edited 1 time in total |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Mon Jan 06, 2020 10:01 pm Post subject: |
|
|
finalturismo,
| Code: | | EMERGE_DEFAULT_OPTS="--jobs 32" |
is not the same as
| Code: | | emerge --ask --update --deep --with-bdeps=y --newuse @world --jobs=32 |
Do you see the difference?
--jobs passed to emerge means to run that many package emerges in parallel if the dependency graph allows it.
e.g. if B depends on A, then A and B cannot be built in parallel.
There is a twist. Each of the --jobs above is permitted to start MAKEOPTS instances of gcc.
Values of MAKEOPTS above about 30 don't do much. Make can't start new jobs fast enough.
On a 96 core system with 128G RAM, I use MAKEOPTS="-j100" and --jobs=6.
It rarely gets to a load average of 100.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Mon Jan 06, 2020 10:16 pm Post subject: |
|
|
| NeddySeagoon wrote: | finalturismo,
| Code: | | EMERGE_DEFAULT_OPTS="--jobs 32" |
is not the same as
| Code: | | emerge --ask --update --deep --with-bdeps=y --newuse @world --jobs=32 |
Do you see the difference?
--jobs passed to emerge means to run that many package emerges in parallel if the dependency graph allows it.
e.g. if B depends on A, then A and B cannot be built in parallel.
There is a twist. Each of the --jobs above is permitted to start MAKEOPTS instances of gcc.
Values of MAKEOPTS above about 30 don't do much. Make can't start new jobs fast enough.
On a 96 core system with 128G RAM, I use MAKEOPTS="-j100" and --jobs=6.
It rarely gets to a load average of 100. |
I am slightly confused, still honestly.
When you speak of dependency graph what is this?
I know gcc is the compiler instances....
I know is the core count of 32
Enables multiple fetching?
| Code: | | FEATURES="${FEATURES} parallel-fetch" |
Have no earthly idea of how to put in the load average code is it .9 or 0.9 or 9?
Also i don't fully understand the DEFAULT_OPS argument.... 
| Code: | | EMERGE_DEFAULT_OPTS="--jobs 20 --load-average 9" |
Here is my make.conf file
| Code: | GNU nano 4.2 /etc/portage/make.conf Modified
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
COMMON_FLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CFLAGS="-march=sandybridge -O2 -pipe"
CXXFLAGS="${CFLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/var/db/repos/gentoo"
DISTDIR="/var/cache/distfiles"
PKGDIR="/var/cache/binpkgs"
# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
MAKEOPTS="-j32"
CPU_FLAGS_X86="aes avx mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3"
LC_MESSAGES=C
#GRUB_PLATFORMS="efi-64"
ACCEPT_LICENSE="*"
USE="elogind X udev fpm"
GENTOO_MIRRORS="http://www.gtlib.gatech.edu/pub/gentoo rsync://rsync.gtlib.gatech.edu/gentoo ftp://ftp.ussg.iu.edu/pub/linux/gentoo https://gentoo.ussg.indiana.edu/ http://gentoo.ussg.indiana.edu/ https://gentoo.osuosl.org/ http://gentoo.osuosl.org/ http://gentoo.mirrors.pair.com/ https://mirrors.rit.edu/gentoo/ >
EMERGE_DEFAULT_OPTS="--jobs 20 --load-average 9"
FEATURES="${FEATURES} parallel-fetch" |
Can i see an example if your make.conf file to help me better understand? |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Tue Jan 07, 2020 12:00 am Post subject: |
|
|
finalturismo,
I'll explain your make.conf. Its easier than explaining mine as mine has 18 year of accumulated rubbish. :)
Lets start with | Code: | | EMERGE_DEFAULT_OPTS="--jobs 20 --load-average 9" |
EMERGE_DEFAULT_OPTS= is a list of options you want to pass to emerge just as if they were typed on the command line. It save you some typing, that's all.
--jobs 20 is not correct. White space is a delimiter. --jobs requires a value.
--jobs=20 is what you should write.
is correct accoridng to the man page, which also goes on to say | Code: | See make(1) for in‐
formation about analogous options that should be configured via
MAKEOPTS in make.conf(5). |
That sets limits on what portage will do.
When portage calculates what needs to be done, it also calculates the order in which packages must be installed. This order is called the dependency graph, or dep graph for short.
It places limits on what packages can be built in parallel. EMERGE_DEFAULT_OPTS="--jobs=20" places an upper limit on that, the build order may restrict it to one job at a time.
If package A is needed to link package B, then A must be installed before B can be built.
MAKEOPTS="-j32" says that for each of the 20 jobs in EMERGE_DEFAULT_OPTS="--jobs=20", that make can start 32 parallel instances of gcc.
On the face of it, That's 640 parallel instances of gcc.
As you also give --load-average 9 to emerge, lets say its started 9 parallel packages. Each of these 9 packages can start instances of gcc.
That's because you only told emerge about --load-average 9. You did not tell make. MAKEOPTS= takes the -l (Lowercase letter L, dot digit 1) to pass a load average value to make.
You choose the numbers. | Code: | | MAKEOPTS="-j32 -l30" |
FEATURES="${FEATURES} parallel-fetch" FEATURES= is additive, like USE flags, so ${FEATURES} is not required.
A long time ago, portage used download a package build it, download the next package, build that ...
parallel-fetch was added so that poratge would download the second and later packages while building was in progress. That has been the default for some years.
Portage only fetches one file at a time.
Don't do That accepts any licence. Even Microsoft Licences.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Tue Jan 07, 2020 3:24 am Post subject: |
|
|
Thanks that explains ALOT. Infact that explanation is the best ive found on the net and is finally what ive been looking for to finally understand this.
I have been seeing so much conflicting information on this, but you totally look like you know whats your talking about.
MAKEOPTS="-j32 -l32"
are those good numbers you think for 4x 8core Xeons? i am guessing its best to keep HT off when you have 32 cores for portage?
I knew about the license acceptance why is it bad to accept all licenses?
Can i see your make file? iam interested to see what 18 years of experience looks like. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Tue Jan 07, 2020 3:03 pm Post subject: |
|
|
2 different programs take the same argument.
one is emerge. when you pass -j 32 to emerge, it will _try_ to emerge up to 32 programs at the same time. i say try, because dependencies make this impossible at times. but it tries.
the second program is make, or ninja, or meson. and in that case, the build will try to use up to 32 threads to build your program.
so when you put 32 on one and 32 on the other you will need up to ~900 cores to do it all. multiplication.
EMERGE_DEFAULT_OPTS="--jobs 32"
This is equivalent to emerge --jobs 32. I suggest you set this lower to something more sane like 10.
MAKEOPTS="-j32"
This is the equivalent of make -j32. if you have 32 cores and set the other one 10... I guess it is ok.
I hope you have enough memory for this. Coz if you hit like 3-4 packages that use c++ you can run out of memory very quickly. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Tue Jan 07, 2020 3:17 pm Post subject: |
|
|
finalturismo,
My Phenom II make.conf. Do not copy it. It won't work for most users.
To expand on axls point about RAM, each C++ make thread can require 2G RAM.
You would be unlucky to build several packages like that in parallel but it happens.
If you have 32 cores and you use them all for building, you may find that interactive performance is poor.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Tue Jan 07, 2020 6:22 pm Post subject: |
|
|
jobs=x is how many packages you will allow emerge to run at the same time
makeopts=x is how many threads you will allow to run at the same time PER package
so a jobs=3 makeopts=4 means you will have a max of 3 applications emerge at the same time with a max of 12 threads (however the minimum will remain 1 application and 1 thread) |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Tue Jan 07, 2020 6:29 pm Post subject: |
|
|
I'm ashamed to say I completely forgot for like ... I don't even know how long... that emerge supports --jobs. I know I knew it at some point, but maybe didn't have the hardware or smth... i just forgot, then... I just never put 2 and 2 together. Last year in 2019, in fact while bitching here in a thread about portage in C... I realized... and finally 2 and 2 became 4. by way of multiplication.
And I think, if other gentooers simply forgot about this detail too, they should pay attention. coz it's a cool feature. |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Tue Jan 07, 2020 6:32 pm Post subject: |
|
|
would be interesting if someone knew when --jobs was added to emerge. I tend to think it was in the early 2000. I kinda think I forgot and blocked it out because when I first learned about it we were still in the unicore era. thus it was -j1 all across the board.
it wasn't that I didn't knew it was there. I just blocked it out. and in 2019 i was like... why didn't I know this was there... for like 20 years. it was a major DOH!!! moment. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Tue Jan 07, 2020 7:34 pm Post subject: |
|
|
| axl wrote: | 2 different programs take the same argument.
one is emerge. when you pass -j 32 to emerge, it will _try_ to emerge up to 32 programs at the same time. i say try, because dependencies make this impossible at times. but it tries.
the second program is make, or ninja, or meson. and in that case, the build will try to use up to 32 threads to build your program.
so when you put 32 on one and 32 on the other you will need up to ~900 cores to do it all. multiplication.
EMERGE_DEFAULT_OPTS="--jobs 32"
This is equivalent to emerge --jobs 32. I suggest you set this lower to something more sane like 10.
MAKEOPTS="-j32"
This is the equivalent of make -j32. if you have 32 cores and set the other one 10... I guess it is ok.
I hope you have enough memory for this. Coz if you hit like 3-4 packages that use c++ you can run out of memory very quickly. |
I have 256GB of ram, i am in recycling business so i get access to lots of equipment for next to nothing.
So lower -j32 to -j10 what about -l?
how does -j10 and -l10 with jobs at 8 sound? |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Tue Jan 07, 2020 7:39 pm Post subject: |
|
|
I thought my 128 was exaggerated. yeah... you are prolly fine with whatever jobs u set.
i just... can you imagine what the apollo nasa team would have done with soooooooooooo much memory. damn. soooo much memory. people dont have ssd's as large as your memory. damn. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Tue Jan 07, 2020 7:49 pm Post subject: |
|
|
| axl wrote: | I thought my 128 was exaggerated. yeah... you are prolly fine with whatever jobs u set.
i just... can you imagine what the apollo nasa team would have done with soooooooooooo much memory. damn. soooo much memory. people dont have ssd's as large as your memory. damn. |
It used to have 512GB ram, but i had to sell at least half of it so i didn't get yelled at around here. 
Its a HP DL560 Gen 8 with 4 8 core Xeons in it. In fact a DL560 gen 8 is one of the best servers you can get when it comes to $$$ to CPU crunching ratio.
That's why i crept it out of the server pallet and stuck it in our rack
My websites been doing for the past few days getting this thing up on nginx with gentoo on the newest kernel. Having to transfer all my old server stuff over to this one. I also got it set up as a dhcp server with connections being monitored on a monitor in the rack. Internal network response time is insane. Just need to find a rock solid 24-48 port enterprise un-managed switch to boost ip association time. Managed switches tend to take long to associated so getting an ip takes longer. Trying to get network and kernel latency as low as possible for websites to be extremely snappy. Might switch over to btrfs soon, considering it seems to be stomping other file systems in recent benchmarks. Especially with sql databases.
Last edited by finalturismo on Tue Jan 07, 2020 7:55 pm; edited 1 time in total |
|
| Back to top |
|
|
axl
Veteran
Joined: 11 Oct 2002
Posts: 1144
Location: Romania
|
| Posted: Tue Jan 07, 2020 7:54 pm Post subject: |
|
|
| yeah. lost me at HP and DL. having to wait 3-4 minutes for bios to ... do whatever it did. nightmares. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Tue Jan 07, 2020 8:37 pm Post subject: |
|
|
finalturismo.
With 32 cores and 256G RAM. You should set the load average to 32 in both places. Your limiting factor will be CPU cores
You may as well put /var/tmp/portage into tmpfs if you haven't already.
It won't make emerge any faster, it will save writes to HDD that will never be read. That will free up some IO bandwidth too.
What you want for jobs varies with workload.
On a 96 core system I use emerge with --jobs=6 because there are few situations where that is actually limiting factor.
Very rarely are 6 packages emerging together.
I use MAKEOPTS="-j100" again, that's not a practical limit on most packages.
I don't set load average at all and the system almost never gets to a load average of 100.
Play with some numbers. See what works for your workload.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Tue Jan 07, 2020 8:46 pm Post subject: |
|
|
| NeddySeagoon wrote: | finalturismo.
With 32 cores and 256G RAM. You should set the load average to 32 in both places. Your limiting factor will be CPU cores
You may as well put /var/tmp/portage into tmpfs if you haven't already.
It won't make emerge any faster, it will save writes to HDD that will never be read. That will free up some IO bandwidth too.
What you want for jobs varies with workload.
On a 96 core system I use emerge with --jobs=6 because there are few situations where that is actually limiting factor.
Very rarely are 6 packages emerging together.
I use MAKEOPTS="-j100" again, that's not a practical limit on most packages.
I don't set load average at all and the system almost never gets to a load average of 100.
Play with some numbers. See what works for your workload. |
| Code: | MAKEOPTS="-j10 -l10"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=32" |
What do you mean set it in both places?
How does that look, i am starting to get a clear understanding of this. Still confused on how the --load-average fits into this math problem. |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54237
Location: 56N 3W
|
| Posted: Tue Jan 07, 2020 9:08 pm Post subject: |
|
|
finalturismo,
MAKEOPTS="-j10 -l32"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=32"
There is only a single load average for the system. If you set it at 10 in MAKEOPTS=, make will not start any new threads once the system load average reaches 10 and 22 cores will be idle.
The MAKEOPTS="-j10" is per job in EMERGE_DEFAULT_OPTS="--jobs=8" but the load average is a system wide thing, its not per emerge job.
Suppose you were using the system for something else and the load average was already 11.
| Code: | | EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=32" |
would allow portage to start, because 32 is bigger than 11.
Make would stall because | Code: | | MAKEOPTS="-j10 -l10" |
so make is not permitted to start any jobs, 10 is smaller than 11.
The load average is displayed in the first line of top
| Code: | | top - 21:03:05 up 12:24, 1 user, load average: 0.41, 0.31, 0.28 |
The three numbers are the load average over different periods of time.
The -l and --load-average= both look at the same system wide operating parameter.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
finalturismo
Guru
Joined: 06 Jan 2020
Posts: 410
|
| Posted: Tue Jan 07, 2020 10:17 pm Post subject: |
|
|
| NeddySeagoon wrote: | finalturismo,
MAKEOPTS="-j10 -l32"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=32"
There is only a single load average for the system. If you set it at 10 in MAKEOPTS=, make will not start any new threads once the system load average reaches 10 and 22 cores will be idle.
The MAKEOPTS="-j10" is per job in EMERGE_DEFAULT_OPTS="--jobs=8" but the load average is a system wide thing, its not per emerge job.
Suppose you were using the system for something else and the load average was already 11.
| Code: | | EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=32" |
would allow portage to start, because 32 is bigger than 11.
Make would stall because | Code: | | MAKEOPTS="-j10 -l10" |
so make is not permitted to start any jobs, 10 is smaller than 11.
The load average is displayed in the first line of top
| Code: | | top - 21:03:05 up 12:24, 1 user, load average: 0.41, 0.31, 0.28 |
The three numbers are the load average over different periods of time.
The -l and --load-average= both look at the same system wide operating parameter. |
got ya, so is a load average 50 mean 50% cpu usage on average?
Anyway i got working great. Emerge is finally destroying that code much faster than before cut my time down by multiple times at least. |
|
| Back to top |
|
|
jmq
n00b
Joined: 28 Apr 2021
Posts: 2
Location: Seattle, WA
|
| Posted: Sun May 09, 2021 2:34 am Post subject: |
|
|
| finalturismo wrote: |
got ya, so is a load average 50 mean 50% cpu usage on average?
|
Load average is not a simple number anymore. It used to be that you
could compute percentage of total CPU "load" with
$LOAD_AVERAGE / $NUM_CORES, in other words a load of 3.5 on a
quad-core system would be ~87.5% total load, but that is no longer the
case.
It is now a mostly meaningless number that people think is important. On
any given system, it only really has meaning with respect to itself, measured
at different times. (In other words, to approximate the total system load
over time, as compared to other measurements on the same system.)
I would recommend not setting a load value. Just forget the parameter exists.
_________________
Jordan McQueen
Senior Software Engineer |
|
| Back to top |
|
|
|
Portage & Programming
