Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
help: auto set "-j" in MAKEOPTS by /etc/portage/bashrc
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
fpemud
Tux's lil' helper
Tux's lil' helper


Joined: 15 Feb 2012
Posts: 123

PostPosted: Sat Sep 22, 2012 4:05 pm    Post subject: help: auto set "-j" in MAKEOPTS by /etc/portage/ba Reply with quote

I want to set "-j" according to my CPU core number. But It doesn't work as I expected.
File "/etc/portage/bashrc.d/adv-compiler" does the main job, please help me check it.

Code:
fpemud-workstation portage # cat bashrc
#!/bin/sh
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

for f in /etc/portage/bashrc.d/* ; do
   [ -x "${f}" ] && "${f}"
done


Code:
fpemud-workstation portage # tree bashrc.d/
bashrc.d/
└── adv-compiler

0 directories, 1 files


Code:
fpemud-workstation portage # cat bashrc.d/adv-compiler
#!/bin/sh

CFLAGS="-O2 -pipe"
CXXFLAGS="${CFLAGS}"

CORE_NUM=$(cat /proc/cpuinfo | grep 'processor' | wc -l)
MAKEOPTS="${MAKEOPTS} -j$(expr ${CORE_NUM} + 1)"
unset CORE_NUM


Code:
fpemud-workstation portage # cat make.conf
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"

# Portage features
FEATURES=""
PORTAGE_NICENESS="15"

# These are the Keywords that were used
ACCEPT_KEYWORDS=""

# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
U_HARDWARE="acpi bluetooth lm_sensors opengl usb"
U_HARDWARE_NOT_GLOBAL=""
U_SOFTWARE="a52 aac accessibility apng bluray cairo caps cdda cjk consolekit dbus djvu doc dts dv dvb \
            dvd dvdr encode exif ffmpeg flac fontconfig fuse gd gif gnutls gstreamer icu jadetex jpeg jpeg2k jpg kvm lame \
            lzma mad mms mng mp3 mpeg musepack networkmanager nsplugin ogg png \
            policykit pulseaudio python speex startup-notification svg theora tiff truetype udev vala \
            vcd vorbis wavpack wmf"
U_SOFTWARE_NOT_GLOBAL="aio airplay alac device-mapper egl g3dvl gallium grilo gtk3 openvg nautilus pic xps vdpau vpx winbind"
U_FORBIDDEN=""
USE="${U_HARDWARE} ${U_HARDWARE_NOT_GLOBAL} ${U_SOFTWARE} ${U_SOFTWARE_NOT_GLOBAL} ${U_FORBIDDEN}"

# Language options
LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA  \
         eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN                                       \
         he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my      \
         nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq     \
         sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu"

# Other options
VIDEO_CARDS="dummy vesa intel nouveau redeon"
INPUT_DEVICES="keyboard mouse synaptics evdev"
PHP_TARGETS="php5-3"
DRACUT_MODULES="lvm"

# Mirrors
GENTOO_MIRRORS="http://mirrors.163.com/gentoo/ http://gentoo.localhost.net.ar/"

# Overlays
source /var/lib/layman/make.conf
PORTDIR_OVERLAY="$PORTDIR_OVERLAY /usr/local/portage /usr/portage"


MAKEOPTS should be in "emerge --info", but it's not:

Code:
fpemud-workstation portage # emerge --info
Portage 2.1.11.9 (default/linux/amd64/10.0, gcc-4.5.4, glibc-2.15-r2, 3.2.12-gentoo x86_64)
=================================================================
System uname: Linux-3.2.12-gentoo-x86_64-Intel-R-_Core-TM-_i5-2500K_CPU_@_3.30GHz-with-gentoo-2.1
Timestamp of tree: Sat, 22 Sep 2012 13:00:01 +0000
distcc 3.1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          4.2_p37
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.3-r2, 3.2.3
dev-util/cmake:           2.8.8-r3
dev-util/pkgconfig:       0.27.1
sys-apps/baselayout:      2.1-r1
sys-apps/openrc:          0.10.5
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.9.6-r3, 1.11.6
sys-devel/binutils:       2.22-r1
sys-devel/gcc:            4.5.4
sys-devel/gcc-config:     1.7.3
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r3
sys-kernel/linux-headers: 3.4-r2 (virtual/os-headers)
sys-libs/glibc:           2.15-r2
Repositories: x11 sunrise gentoo-zh java-overlay x-portage gentoo
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /usr/share/openvpn/easy-rsa"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles news parallel-fetch parse-eapi-ebuild-head protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirrors.163.com/gentoo/ http://gentoo.localhost.net.ar/"
LC_ALL="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/x11 /var/lib/layman/sunrise /var/lib/layman/gentoo-zh /var/lib/layman/java-overlay /usr/local/portage /usr/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="a52 aac accessibility acl acpi aio airplay alac amd64 apng berkdb bluetooth bluray bzip2 cairo caps cdda cjk cli consolekit cracklib crypt cups cxx dbus device-mapper djvu doc dri dts dv dvb dvd dvdr egl encode exif ffmpeg flac fontconfig fortran fuse g3dvl gallium gd gdbm gif gnutls gpm grilo gstreamer gtk3 iconv icu ipv6 jadetex jpeg jpeg2k jpg kvm lame lm_sensors lzma mad mms mmx mng modules mp3 mpeg mudflap multilib musepack nautilus ncurses networkmanager nls nptl nsplugin ogg opengl openmp openvg pam pcre pic png policykit pppd pulseaudio python readline session speex sse sse2 ssl startup-notification svg tcpd theora tiff truetype udev unicode usb vala vcd vdpau vorbis vpx wavpack winbind wmf xps zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" DRACUT_MODULES="lvm" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="af ak am ar as ast az be bg bn bn_BD bn_IN bo br brx bs ca ca_XV cs csb cy da de dgo dz el en en_CA en_GB en_US en_ZA eo es es_AR es_CL es_ES es_MX et eu fa fi fo fr fy fy_NL ga ga_IE gd gl gu gu_IN he hi hi_IN hr hu hy hy_AM ia id is it ja ka kk km kn ko kok ks ku ky lb lg lo lt lv mai mk ml mn mni mr ms mt my nb nb_NO nds ne nl nn nn_NO nr nso oc om or pa pa_IN pl pt pt_BR pt_PT rm ro ru rw sa_IN sat sd sh si sk sl son sq sr ss st sv sv_SE sw sw_TZ ta ta_LK te th tg tn tr ts tt ug uk uz ve vi wa xh yi zh zh_CN zh_HK zh_TW zu" PHP_TARGETS="php5-3" PYTHON_TARGETS="python3_2 python2_7" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="dummy vesa intel nouveau redeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Back to top
View user's profile Send private message
toralf
Advocate
Advocate


Joined: 01 Feb 2004
Posts: 2614
Location: Hamburg/Germany

PostPosted: Sat Sep 22, 2012 4:26 pm    Post subject: Reply with quote

Might be I miss the point here , but why didn't you add line into /etc/make.conf like
Code:
MAKEOPTS="-j5"
?
Back to top
View user's profile Send private message
fpemud
Tux's lil' helper
Tux's lil' helper


Joined: 15 Feb 2012
Posts: 123

PostPosted: Sat Sep 22, 2012 5:56 pm    Post subject: Reply with quote

According to Gentoo Handbook and my experience, this value is purely calculated by CPU core number.

Then why specify it manually? I wonder if I can fix out a way to let the computer decide this value for itself and hide it from user.

I have made a ebuild to install this /etc/portage/bashrc and /etc/portage/bashrc.d.
So if it is feasible, it will be very simple to use.
Back to top
View user's profile Send private message
Gusar
Advocate
Advocate


Joined: 09 Apr 2005
Posts: 2633
Location: Slovenia

PostPosted: Sat Sep 22, 2012 6:09 pm    Post subject: Reply with quote

Isn't make.conf basically a bash script that is being sourced? So simply putting this directly in make.conf should work:
Code:
CORE_NUM=$(grep -c ^processor /proc/cpuinfo)
MAKEOPTS="-j$((CORE_NUM+1))"

I don't see why a package would be needed for something this simple. Oh, and as you can see I also improved your cringe-worthy code :)

And if you use BFS you can drop the +1 stuff, BFS is good enough that it'll fully utilize the CPU with just -j$CORE_NUM
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 3784

PostPosted: Sat Sep 22, 2012 7:09 pm    Post subject: Reply with quote

Gusar wrote:
Isn't make.conf basically a bash script that is being sourced?

No, it isn't. It is parsed by a python module and does not recognize any "commands" except for the most primitive variable substitution.
fpemud wrote:
Code:
cat /proc/cpuinfo | grep 'processor'

Just an unrelated remark: This is useless use of cat :wink: You want
Code:
grep processor /proc/cpuinfo

Time is here somewhat important, since /etc/portage/bashrc is sourced in every phase of emerging of every package. Since /etc/portage/bashrc is guaranteed to be bash code, I would also recommend to avoid the call to the external `expr' and instead use the bash/posix $((...)). But even better is to execute this code only in the setup phase. (See e.g. portage-bashrc-mv from the mv overlay how to write code in /etc/portage/bashrc.d which is executed only in a particular phase; with that package, if you name you code /etc/portage/bashrc.d/30-... you should also see automatically which value of MAKEOPTS is actually used).

fpemud, your conjecture is false: emerge --info should not list such things, because you cannot "push up" a variable from bash to the python code. However, during emerge of a package the variable should be set correctly.

Another (less intrusive) possibility is to export MAKEOPTS in the environment (e.g. in /etc/profile).

However, the value "number of CPUs + 1" is just an empirical value, not necessarily the best choice for all systems. What is the best choice depends on so many things (and also on the package) that it is not possible to give a universal recommendation. The recommendation in the manual was written when the number of processors on machines was usually 1 or at most 2...
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 2207
Location: UK

PostPosted: Sat Sep 22, 2012 7:39 pm    Post subject: Reply with quote

Code:
MAKEOPTS="-j$(lscpu -p | grep -cv '#')"
Back to top
View user's profile Send private message
fpemud
Tux's lil' helper
Tux's lil' helper


Joined: 15 Feb 2012
Posts: 123

PostPosted: Sun Sep 23, 2012 11:36 am    Post subject: Reply with quote

ok, i know.
Since "-j" is not calculated by some fixed formula, then i agree maually set it in make.conf should be the best way.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 3784

PostPosted: Sun Sep 23, 2012 7:06 pm    Post subject: Reply with quote

Ant P. wrote:
Code:
MAKEOPTS="-j$(lscpu -p | grep -cv '#')"

As mentioned, this does not work in /etc/portage/make.conf. You must use it in some other startup script (e.g. in /etc/profile.d) and export MAKEOPTS to set it in the global environment.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 8993
Location: beyond the rim

PostPosted: Wed Sep 26, 2012 10:23 am    Post subject: Reply with quote

To answer the original question: bashrc isn't executed for the --info action.
Back to top
View user's profile Send private message
Suicidal
l33t
l33t


Joined: 30 Jul 2003
Posts: 940
Location: /dev/null

PostPosted: Mon Apr 15, 2013 12:48 pm    Post subject: Reply with quote

I use a script in profile.d. I use gentoo mostly as vmware guests so the number of cpu's can change drastically depending on the deployment.
I prefer to only use physical cores and omit hyperthreading from the results.

/etc/profile.d/makeopts.sh:
#!/bin/bash

        DEBUG="FALSE"

        # get physical cpus from lscpu
        SOCKETS="$(lscpu | grep Socket\(s\): | awk '{print $2}' )"

        # get cores per socket from lscpu
        CORES="$(lscpu | grep Core\(s\)\ \per\ \socket: | awk '{print $4}')"

        # multiply sockets * cores +1
        REAL_CORES="$((${SOCKETS}*${CORES}+1))"

        export MAKEOPTS="--jobs="${REAL_CORES}" --silent --load-average="${REAL_CORES}" "

        if [[ ${DEBUG} = "TRUE" ]]; then
                echo SOCKETS=${SOCKETS}
                echo CORES=${CORES}
                echo REAL_CORES=${REAL_CORES}
                echo MAKEOPTS=${MAKEOPTS}
        fi

_________________
Who controls the past now, controls the future
Who controls the present now, controls the past.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum