View previous topic :: View next topic |
Author |
Message |
fpemud Guru
Joined: 15 Feb 2012 Posts: 349
|
Posted: Sat Sep 22, 2012 4:05 pm Post subject: help: auto set "-j" in MAKEOPTS by /etc/portage/ba |
|
|
I want to set "-j" according to my CPU core number. But It doesn't work as I expected.
File "/etc/portage/bashrc.d/adv-compiler" does the main job, please help me check it.
Code: | fpemud-workstation portage # cat bashrc
#!/bin/sh
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
for f in /etc/portage/bashrc.d/* ; do
[ -x "${f}" ] && "${f}"
done
|
Code: | fpemud-workstation portage # tree bashrc.d/
bashrc.d/
└── adv-compiler
0 directories, 1 files
|
Code: | fpemud-workstation portage # cat bashrc.d/adv-compiler
#!/bin/sh
CFLAGS="-O2 -pipe"
CXXFLAGS="${CFLAGS}"
CORE_NUM=$(cat /proc/cpuinfo | grep 'processor' | wc -l)
MAKEOPTS="${MAKEOPTS} -j$(expr ${CORE_NUM} + 1)"
unset CORE_NUM
|
Code: | fpemud-workstation portage # cat make.conf
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# Portage features
FEATURES=""
PORTAGE_NICENESS="15"
# These are the Keywords that were used
ACCEPT_KEYWORDS=""
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
U_HARDWARE="acpi bluetooth lm_sensors opengl usb"
U_HARDWARE_NOT_GLOBAL=""
U_SOFTWARE="a52 aac accessibility apng bluray cairo caps cdda cjk consolekit dbus djvu doc dts dv dvb \
dvd dvdr encode exif ffmpeg flac fontconfig fuse gd gif gnutls gstreamer icu jadetex jpeg jpeg2k jpg kvm lame \
lzma mad mms mng mp3 mpeg musepack networkmanager nsplugin ogg png \
policykit pulseaudio python speex startup-notification svg theora tiff truetype udev vala \
vcd vorbis wavpack wmf"
U_SOFTWARE_NOT_GLOBAL="aio airplay alac device-mapper egl g3dvl gallium grilo gtk3 openvg nautilus pic xps vdpau vpx winbind"
U_FORBIDDEN=""
USE="${U_HARDWARE} ${U_HARDWARE_NOT_GLOBAL} ${U_SOFTWARE} ${U_SOFTWARE_NOT_GLOBAL} ${U_FORBIDDEN}"
# Language options
LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA \
eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN \
he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my \
nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq \
sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu"
# Other options
VIDEO_CARDS="dummy vesa intel nouveau redeon"
INPUT_DEVICES="keyboard mouse synaptics evdev"
PHP_TARGETS="php5-3"
DRACUT_MODULES="lvm"
# Mirrors
GENTOO_MIRRORS="http://mirrors.163.com/gentoo/ http://gentoo.localhost.net.ar/"
# Overlays
source /var/lib/layman/make.conf
PORTDIR_OVERLAY="$PORTDIR_OVERLAY /usr/local/portage /usr/portage"
|
MAKEOPTS should be in "emerge --info", but it's not:
Code: | fpemud-workstation portage # emerge --info
Portage 2.1.11.9 (default/linux/amd64/10.0, gcc-4.5.4, glibc-2.15-r2, 3.2.12-gentoo x86_64)
=================================================================
System uname: Linux-3.2.12-gentoo-x86_64-Intel-R-_Core-TM-_i5-2500K_CPU_@_3.30GHz-with-gentoo-2.1
Timestamp of tree: Sat, 22 Sep 2012 13:00:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
app-shells/bash: 4.2_p37
dev-java/java-config: 2.1.11-r3
dev-lang/python: 2.7.3-r2, 3.2.3
dev-util/cmake: 2.8.8-r3
dev-util/pkgconfig: 0.27.1
sys-apps/baselayout: 2.1-r1
sys-apps/openrc: 0.10.5
sys-apps/sandbox: 2.5
sys-devel/autoconf: 2.13, 2.68
sys-devel/automake: 1.9.6-r3, 1.11.6
sys-devel/binutils: 2.22-r1
sys-devel/gcc: 4.5.4
sys-devel/gcc-config: 1.7.3
sys-devel/libtool: 2.4-r1
sys-devel/make: 3.82-r3
sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers)
sys-libs/glibc: 2.15-r2
Repositories: x11 sunrise gentoo-zh java-overlay x-portage gentoo
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirrors.163.com/gentoo/ http://gentoo.localhost.net.ar/"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/x11 /var/lib/layman/sunrise /var/lib/layman/gentoo-zh /var/lib/layman/java-overlay /usr/local/portage /usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="a52 aac accessibility acl acpi aio airplay alac amd64 apng berkdb bluetooth bluray bzip2 cairo caps cdda cjk cli consolekit cracklib crypt cups cxx dbus device-mapper djvu doc dri dts dv dvb dvd dvdr egl encode exif ffmpeg flac fontconfig fortran fuse g3dvl gallium gd gdbm gif gnutls gpm grilo gstreamer gtk3 iconv icu ipv6 jadetex jpeg jpeg2k jpg kvm lame lm_sensors lzma mad mms mmx mng modules mp3 mpeg mudflap multilib musepack nautilus ncurses networkmanager nls nptl nsplugin ogg opengl openmp openvg pam pcre pic png policykit pppd pulseaudio python readline session speex sse sse2 ssl startup-notification svg tcpd theora tiff truetype udev unicode usb vala vcd vdpau vorbis vpx wavpack winbind wmf xps zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DRACUT_MODULES="lvm" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="dummy vesa intel nouveau redeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
|
|
|
Back to top |
|
|
toralf Developer
Joined: 01 Feb 2004 Posts: 3922 Location: Hamburg
|
Posted: Sat Sep 22, 2012 4:26 pm Post subject: |
|
|
Might be I miss the point here , but why didn't you add line into /etc/make.conf like ? |
|
Back to top |
|
|
fpemud Guru
Joined: 15 Feb 2012 Posts: 349
|
Posted: Sat Sep 22, 2012 5:56 pm Post subject: |
|
|
According to Gentoo Handbook and my experience, this value is purely calculated by CPU core number.
Then why specify it manually? I wonder if I can fix out a way to let the computer decide this value for itself and hide it from user.
I have made a ebuild to install this /etc/portage/bashrc and /etc/portage/bashrc.d.
So if it is feasible, it will be very simple to use. |
|
Back to top |
|
|
Gusar Advocate
Joined: 09 Apr 2005 Posts: 2665 Location: Slovenia
|
Posted: Sat Sep 22, 2012 6:09 pm Post subject: |
|
|
Isn't make.conf basically a bash script that is being sourced? So simply putting this directly in make.conf should work: Code: | CORE_NUM=$(grep -c ^processor /proc/cpuinfo)
MAKEOPTS="-j$((CORE_NUM+1))" |
I don't see why a package would be needed for something this simple. Oh, and as you can see I also improved your cringe-worthy code
And if you use BFS you can drop the +1 stuff, BFS is good enough that it'll fully utilize the CPU with just -j$CORE_NUM |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6747
|
Posted: Sat Sep 22, 2012 7:09 pm Post subject: |
|
|
Gusar wrote: | Isn't make.conf basically a bash script that is being sourced? |
No, it isn't. It is parsed by a python module and does not recognize any "commands" except for the most primitive variable substitution.
fpemud wrote: | Code: | cat /proc/cpuinfo | grep 'processor' |
|
Just an unrelated remark: This is useless use of cat You want Code: | grep processor /proc/cpuinfo |
Time is here somewhat important, since /etc/portage/bashrc is sourced in every phase of emerging of every package. Since /etc/portage/bashrc is guaranteed to be bash code, I would also recommend to avoid the call to the external `expr' and instead use the bash/posix $((...)). But even better is to execute this code only in the setup phase. (See e.g. portage-bashrc-mv from the mv overlay how to write code in /etc/portage/bashrc.d which is executed only in a particular phase; with that package, if you name you code /etc/portage/bashrc.d/30-... you should also see automatically which value of MAKEOPTS is actually used).
fpemud, your conjecture is false: emerge --info should not list such things, because you cannot "push up" a variable from bash to the python code. However, during emerge of a package the variable should be set correctly.
Another (less intrusive) possibility is to export MAKEOPTS in the environment (e.g. in /etc/profile).
However, the value "number of CPUs + 1" is just an empirical value, not necessarily the best choice for all systems. What is the best choice depends on so many things (and also on the package) that it is not possible to give a universal recommendation. The recommendation in the manual was written when the number of processors on machines was usually 1 or at most 2... |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Sep 22, 2012 7:39 pm Post subject: |
|
|
Code: | MAKEOPTS="-j$(lscpu -p | grep -cv '#')" |
|
|
Back to top |
|
|
fpemud Guru
Joined: 15 Feb 2012 Posts: 349
|
Posted: Sun Sep 23, 2012 11:36 am Post subject: |
|
|
ok, i know.
Since "-j" is not calculated by some fixed formula, then i agree maually set it in make.conf should be the best way. |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6747
|
Posted: Sun Sep 23, 2012 7:06 pm Post subject: |
|
|
Ant P. wrote: | Code: | MAKEOPTS="-j$(lscpu -p | grep -cv '#')" |
|
As mentioned, this does not work in /etc/portage/make.conf. You must use it in some other startup script (e.g. in /etc/profile.d) and export MAKEOPTS to set it in the global environment. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9531 Location: beyond the rim
|
Posted: Wed Sep 26, 2012 10:23 am Post subject: |
|
|
To answer the original question: bashrc isn't executed for the --info action. |
|
Back to top |
|
|
Suicidal l33t
Joined: 30 Jul 2003 Posts: 959 Location: /dev/null
|
Posted: Mon Apr 15, 2013 12:48 pm Post subject: |
|
|
I use a script in profile.d. I use gentoo mostly as vmware guests so the number of cpu's can change drastically depending on the deployment.
I prefer to only use physical cores and omit hyperthreading from the results.
/etc/profile.d/makeopts.sh: | #!/bin/bash
DEBUG="FALSE"
# get physical cpus from lscpu
SOCKETS="$(lscpu | grep Socket\(s\): | awk '{print $2}' )"
# get cores per socket from lscpu
CORES="$(lscpu | grep Core\(s\)\ \per\ \socket: | awk '{print $4}')"
# multiply sockets * cores +1
REAL_CORES="$((${SOCKETS}*${CORES}+1))"
export MAKEOPTS="--jobs="${REAL_CORES}" --silent --load-average="${REAL_CORES}" "
if [[ ${DEBUG} = "TRUE" ]]; then
echo SOCKETS=${SOCKETS}
echo CORES=${CORES}
echo REAL_CORES=${REAL_CORES}
echo MAKEOPTS=${MAKEOPTS}
fi
|
|
|
Back to top |
|
|
nouboh n00b
Joined: 04 Apr 2003 Posts: 14
|
Posted: Mon Oct 05, 2015 1:29 pm Post subject: |
|
|
Suicidal wrote: | I use a script in profile.d. I use gentoo mostly as vmware guests so the number of cpu's can change drastically depending on the deployment.
I prefer to only use physical cores and omit hyperthreading from the results.
/etc/profile.d/makeopts.sh: | #!/bin/bash
DEBUG="FALSE"
# get physical cpus from lscpu
SOCKETS="$(lscpu | grep Socket\(s\): | awk '{print $2}' )"
# get cores per socket from lscpu
CORES="$(lscpu | grep Core\(s\)\ \per\ \socket: | awk '{print $4}')"
# multiply sockets * cores +1
REAL_CORES="$((${SOCKETS}*${CORES}+1))"
export MAKEOPTS="--jobs="${REAL_CORES}" --silent --load-average="${REAL_CORES}" "
if [[ ${DEBUG} = "TRUE" ]]; then
echo SOCKETS=${SOCKETS}
echo CORES=${CORES}
echo REAL_CORES=${REAL_CORES}
echo MAKEOPTS=${MAKEOPTS}
fi
|
|
Nice, just a question. Why do you prefer to just use physical cores? |
|
Back to top |
|
|
khayyam Watchman
Joined: 07 Jun 2012 Posts: 6227 Location: Room 101
|
Posted: Mon Oct 05, 2015 6:43 pm Post subject: |
|
|
Suicidal wrote: | Code: | # get cores per socket from lscpu
CORES="$(lscpu | grep Core\(s\)\ \per\ \socket: | awk '{print $4}')" |
|
seems silly to use both grep and awk ...
Code: | CORES="$(lscpu | awk '/^Core\(s\)/{print $NF}')" |
... though, really, grep would probably suffice ...
Code: | CORES="$(grep -c 'processor' /proc/cpuinfo)" |
best ... khay |
|
Back to top |
|
|
freke l33t
Joined: 23 Jan 2003 Posts: 977 Location: Somewhere in Denmark
|
Posted: Tue Oct 06, 2015 3:47 pm Post subject: |
|
|
I like the scripted approach and trying it out - I only have one type of hw to test it on atm though... Code: | CORES="$(grep -c 'processor' /proc/cpuinfo)" | is the number I want if using both physical cores + HT? (+1 for the actual jobs number?)
ie. discarding the SOCKETS and REAL_CORES in the script and just use CORES when exporting the MAKEOPTS-setting. |
|
Back to top |
|
|
khayyam Watchman
Joined: 07 Jun 2012 Posts: 6227 Location: Room 101
|
Posted: Tue Oct 06, 2015 7:41 pm Post subject: |
|
|
freke wrote: | I like the scripted approach and trying it out - I only have one type of hw to test it on atm though... |
freke ... honestly, I really don't see the point, the number of processors, and so MAKEOPTS, isn't something that will change. I'd be more inclined to set it and be done.
freke wrote: | Code: | CORES="$(grep -c 'processor' /proc/cpuinfo)" |
is the number I want if using both physical cores + HT? (+1 for the actual jobs number?) ie. discarding the SOCKETS and REAL_CORES in the script and just use CORES when exporting the MAKEOPTS-setting. |
Someone can correct me but I've never, ever, set +1 ... I'm not sure why people have this. Anyhow, I'd suggest simply the number of 'processor(s)' reported by /proc/cpuinfo, eg:
Code: | # MAKEOPTS="-j$(grep -c 'processor' /proc/cpuinfo)" ; emerge foo |
... or, if you must +1
Code: | # MAKEOPTS="-j$(($(grep -c 'processor' /proc/cpuinfo) +1))" ; emerge foo |
HTH & best ... khay |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3136
|
Posted: Tue Oct 06, 2015 9:25 pm Post subject: |
|
|
Quote: | Someone can correct me but I've never, ever, set +1 ... I'm not sure why people have this | Hiding latency caused by starting new jobs, IO wait, cache misses etc. Whenever one of your tasks is blocked your CPU can switch to another task not wasting any cycles waiting for other components.
At least it looks good on paper. I've never tested whether it actually speeds up compilation, or maybe hinders it e.g. forcing pipeline flush every time it changes tasks. |
|
Back to top |
|
|
freke l33t
Joined: 23 Jan 2003 Posts: 977 Location: Somewhere in Denmark
|
Posted: Tue Oct 06, 2015 10:40 pm Post subject: |
|
|
khayyam wrote: | freke wrote: | I like the scripted approach and trying it out - I only have one type of hw to test it on atm though... |
freke ... honestly, I really don't see the point, the number of processors, and so MAKEOPTS, isn't something that will change. I'd be more inclined to set it and be done. |
I agree - it belongs back on the minimal image - so we won't even have to worry about it
I've migrated installations from one pc to another on a few occasions - so it could be one thing less to remember to check But yeah - normally it would be set and forget.
khayyam wrote: | freke wrote: | Code: | CORES="$(grep -c 'processor' /proc/cpuinfo)" |
is the number I want if using both physical cores + HT? (+1 for the actual jobs number?) ie. discarding the SOCKETS and REAL_CORES in the script and just use CORES when exporting the MAKEOPTS-setting. |
Someone can correct me but I've never, ever, set +1 ... I'm not sure why people have this. Anyhow, I'd suggest simply the number of 'processor(s)' reported by /proc/cpuinfo, eg:
Code: | # MAKEOPTS="-j$(grep -c 'processor' /proc/cpuinfo)" ; emerge foo |
... or, if you must +1
Code: | # MAKEOPTS="-j$(($(grep -c 'processor' /proc/cpuinfo) +1))" ; emerge foo |
HTH & best ... khay |
Dunno if the +1 matters - never tested, just seems to be the standard (or at least the Handbook says to use +1). |
|
Back to top |
|
|
|