Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
thoughts about resources while compiling [answered]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Thu Jul 11, 2019 5:59 pm    Post subject: thoughts about resources while compiling [answered] Reply with quote

Greetings,

a few months ago I got a new machine, 16GB of ram, ryzen x2700, the mb is ga B450 aorus m.
I have 4 hdds within a sw raid. my /tmp/portage is tmpfs of 10 gb.
from the start, I got some peaks where the machine got sluggish or froze for a few seconds and continued while compiling heavy pkgs.
after upgrade to kernel 5.2, I got complete freeze of minutes even.
my previous machine was an i7-2600 with slower ram (16GB), the mb was GA-Z68P-DS3 with the same hdds (/tmp/portage is tmpfs of 10 gb. too) and I don't remember getting sluggish or complete freeze. moreover, I had boinc running in the background.

I read that there is a rule of 2GB per compiling thread but the 10 gb isn't enough for the previous setup.
so I wonder, what can be the issue that causes the system to freeze on heavy loads?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein


Last edited by DaggyStyle on Fri Jul 12, 2019 11:58 am; edited 1 time in total
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1306
Location: Frankfurt, Germany

PostPosted: Thu Jul 11, 2019 6:13 pm    Post subject: Reply with quote

  1. Please show us the output of
    Code:
    emerge --info

  2. Is multithreading enabled? In this case, you would need up to 42 GB Ram (16 x 2GB + 10 GB /var/tmp/portage) if you want to avoid swapping. Well, 32GB will probably be sufficient, but 16GB is not enough.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43222
Location: 56N 3W

PostPosted: Thu Jul 11, 2019 6:41 pm    Post subject: Reply with quote

DaggyStyle,

The 2G per gcc thread is good for large C++ packages.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
ff11
Tux's lil' helper
Tux's lil' helper


Joined: 10 Mar 2014
Posts: 84

PostPosted: Thu Jul 11, 2019 7:35 pm    Post subject: Reply with quote

I don't like it too, but is the way that the source codes are evolving.

THEN I will evolve too:
* I have created my /etc/portage/env/makeopts-j2:
Code:
MAKEOPTS='-j2'

* My /etc/portage/package.env:
Code:
dev-qt/qtwebengine makeopts-j2
dev-lang/spidermonkey makeopts-j2

* On my user, I create the AppImage folder, and have:
- Audacity-2.1.2.glibc2.15-x86_64.AppImage: because I don't wanna compile x11-libs/wxGTK for just 1 or 2 packages
- wxHexEditor-0.23-x86_64.AppImage: because I don't wanna compile x11-libs/wxGTK for just 1 or 2 packages
- MKVToolNix_GUI-35.0.0-x86_64.AppImage: because Appimage is small (40MB), and the 7MB source take a lot to compile (for it size)
- wine-i386_x86_64-archlinux.AppImage: because I don't wanna compile all x86 packages just for wine alone
- sabaki-v0.43.3-linux-x64.AppImage: because Electron release apps should be simple like that.

* Using no-multilib profile, because I don't wanna compile 32bits. If I need, one day, of a chroot 32bits, I will do using Debootstrap and install a debian 32bits chroot. It should be secure enough for just some few apps.

And now, my notebook is resting of this "madness compiler". Meanwhile, I keep on cleaning.
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Thu Jul 11, 2019 7:38 pm    Post subject: Reply with quote

mike155 wrote:
  1. Please show us the output of
    Code:
    emerge --info

  2. Is multithreading enabled? In this case, you would need up to 42 GB Ram (16 x 2GB + 10 GB /var/tmp/portage) if you want to avoid swapping. Well, 32GB will probably be sufficient, but 16GB is not enough.

Code:

Portage 2.3.66 (python 3.6.5-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-9.1.0, glibc-2.29-r2, 5.2.0-gentoo x86_64)
=================================================================
System uname: Linux-5.2.0-gentoo-x86_64-AMD_Ryzen_7_2700X_Eight-Core_Processor-with-gentoo-2.6
KiB Mem:    16414044 total,    171992 free
KiB Swap:   16437244 total,  16420812 free
Head commit of repository gentoo: 388c1f507acf4203196c3ebdd1650d25435006ae

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          4.4_p23-r1::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.28.2-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.14.3::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.41.2::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.13.4-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo
sys-devel/gcc:            9.1.0-r1::gentoo
sys-devel/gcc-config:     2.0::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r2::gentoo
Repositories:

gentoo
    location: /mnt/portage/tree/official
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/repo/gentoo.git
    priority: -1000

Utopia_Platinia
    location: /mnt/portage/tree/overlay
    masters: gentoo kde
    priority: 0

kde
    location: /var/lib/layman/kde
    sync-type: laymansync
    sync-uri: https://anongit.gentoo.org/git/proj/kde.git
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE @FREE AdobeFlash-11.x MPEG-4 Skype-TOS linux-fw-redistributable no-source-code hplip-plugin OPERA-2014 unRAR"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -mtune=native"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -mtune=native"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going --quiet-build=n"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe -march=native -mtune=native"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs clean-logs collision-protect config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userpriv usersandbox usersync"
FFLAGS="-O2 -pipe -march=native -mtune=native"
GENTOO_MIRRORS="https://mirror.isoc.org.il/pub/gentoo/ http://mirror.isoc.org.il/pub/gentoo/ ftp://ftp.gtlib.gatech.edu/pub/gentoo ftp://ftp.ussg.iu.edu/pub/linux/gentoo"
LANG="en_us.US-ASCII"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en es"
MAKEOPTS="-j17 -l8"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--timeout=180"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi activities alsa amd64 berkdb branding bzip2 cairo cdda cdr cli crypt cryptsetup cups cxx dbus declarative dri drm dts dvd dvdr elogind emboss encode exif fam ffmpeg flac fortran gdbm gif glamor gpm gtk iconv icu ipv6 java joystick jpeg kde kipi kwallet lcms ldap libnotify libtirpc lm_sensors lto lvm mad matroska mng mp3 mp4 mpeg multilib ncurses nls nptl nsplugin ogg opencl opengl openmp pam pango pcre pdf pdfimport pgo phonon physics plasma png policykit postproc ppds python qml qt5 rar readline samba sdl seccomp sound spell sqlite ssl startup-notification svg system-harfbuzz system-heimdal system-icu system-jpeg system-jsoncpp system-libevent system-libvpx system-llvm system-sqlite system-webp tcpd tiff truetype udev udisks unicode upower usb vaapi vdpau vlc vorbis vulkan webkit widgets wxwidgets x264 x265 xattr xcb xcomposite xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="evdev" KERNEL="linux" L10N="he es en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="AArch64 ARM AMDGPU BPF" NETBEANS_MODULES="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python3_6" QEMU_SOFTMMU_TARGETS="x86_64 arm aarch64" QEMU_USER_TARGETS="x86_64 arm aarch64" RUBY_TARGETS="ruby24" SANE_BACKENDS="hp" USERLAND="GNU" VIDEO_CARDS="v4l amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS


as said, according to this calc, 8*2+10=26 > 16 on the intel setup and no issue there
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein


Last edited by DaggyStyle on Thu Jul 11, 2019 7:41 pm; edited 1 time in total
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Thu Jul 11, 2019 7:40 pm    Post subject: Reply with quote

ff11 wrote:
I don't like it too, but is the way that the source codes are evolving.

THEN I will evolve too:
* I have created my /etc/portage/env/makeopts-j2:
Code:
MAKEOPTS='-j2'

* My /etc/portage/package.env:
Code:
dev-qt/qtwebengine makeopts-j2
dev-lang/spidermonkey makeopts-j2

* On my user, I create the AppImage folder, and have:
- Audacity-2.1.2.glibc2.15-x86_64.AppImage: because I don't wanna compile x11-libs/wxGTK for just 1 or 2 packages
- wxHexEditor-0.23-x86_64.AppImage: because I don't wanna compile x11-libs/wxGTK for just 1 or 2 packages
- MKVToolNix_GUI-35.0.0-x86_64.AppImage: because Appimage is small (40MB), and the 7MB source take a lot to compile (for it size)
- wine-i386_x86_64-archlinux.AppImage: because I don't wanna compile all x86 packages just for wine alone
- sabaki-v0.43.3-linux-x64.AppImage: because Electron release apps should be simple like that.

* Using no-multilib profile, because I don't wanna compile 32bits. If I need, one day, of a chroot 32bits, I will do using Debootstrap and install a debian 32bits chroot. It should be secure enough for just some few apps.

And now, my notebook is resting of this "madness compiler". Meanwhile, I keep on cleaning.

so reduce the count for some pkgs, may I'll do that, thanks
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Thu Jul 11, 2019 7:42 pm    Post subject: Reply with quote

NeddySeagoon wrote:
DaggyStyle,

The 2G per gcc thread is good for large C++ packages.


I still don't understand why I didn't saw this behavior with the previous system
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3964
Location: Dallas area

PostPosted: Thu Jul 11, 2019 7:47 pm    Post subject: Reply with quote

DaggyStyle wrote:
as said, according to this calc, 8*2+10=26 > 16 on the intel setup and no issue there


you've set it up for -j17, 16 gig is kind of pushing it in your case, at least for heavy compiles.

Have you tried with something like -j10 or -j12 to see if the problem goes away?
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3964
Location: Dallas area

PostPosted: Thu Jul 11, 2019 7:49 pm    Post subject: Reply with quote

DaggyStyle wrote:
NeddySeagoon wrote:
DaggyStyle,

The 2G per gcc thread is good for large C++ packages.


I still don't understand why I didn't saw this behavior with the previous system


i7-2600 4 core - 8 thread

2700x 8 core - 16 thread

That's not exactly comparing apples with apples :wink:
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
ff11
Tux's lil' helper
Tux's lil' helper


Joined: 10 Mar 2014
Posts: 84

PostPosted: Thu Jul 11, 2019 7:50 pm    Post subject: Reply with quote

DaggyStyle wrote:
NeddySeagoon wrote:
DaggyStyle,

The 2G per gcc thread is good for large C++ packages.


I still don't understand why I didn't saw this behavior with the previous system

This "2G per gcc thread" isn't for all packages. The optimizations of heavy frameworks do this. For some hardware the optimizations use more RAM, because the code optimizations for AMD isn't the same code of the optimizations for Intel.
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Fri Jul 12, 2019 11:57 am    Post subject: Reply with quote

I always think wcs, e.g. 2GB but even for firefox, it doesn't means that all cores are using 2GB per core, for example. 0.5GB per core means 8*0.5+10 = 14GB where 16*0.5+10 = 18GB.

I'll limit the core count for firefox.

Thanks.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1306
Location: Frankfurt, Germany

PostPosted: Fri Jul 12, 2019 3:23 pm    Post subject: Reply with quote

Try the following:
  1. Set
    Code:
    MAKEOPTS="-j16"
    and run
    Code:
    time emerge --oneshot gcc

  2. Disable hyperthreading in your BIOS and make sure that it's disabled after you booted Linux (look at the output of 'lscpu': it should print: 'CPU(s): 8' and 'Thread(s) per core: 1').

    Set
    Code:
    MAKEOPTS="-j8"
    and run
    Code:
    time emerge --oneshot gcc

  3. Compare the two results. Most probably, you will see that the hyperthreading test is NOT faster than the non-hyperthreading test. That's the reason why I disable hyperthreading on all of my machines. Disabling hyperthreading also reduces the amount of RAM needed and makes administration and monitoring easier.
Back to top
View user's profile Send private message
firephoto
Veteran
Veteran


Joined: 29 Oct 2003
Posts: 1604
Location: +48° 5' 23.40", -119° 48' 30.00"

PostPosted: Fri Jul 12, 2019 4:29 pm    Post subject: Reply with quote

On my week old 2700x system I've had my makeopts at -j13 and have experienced no noticeable slowdowns even during parallel builds of big things like qtwebengine while various other things finish. With this setup I've noticed the ram use does hit 16G (I have 32G). I purposely have kept it backed off because of the parallelism.

Relevant make.conf bits.
Code:

MAKEOPTS="-j13 -l13"
EMERGE_DEFAULT_OPTS="--jobs 8 --load-average 8.2 --quiet-build=n --with-bdeps=y ${EMERGE_DEFAULT_OPTS}"
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"


I just fired off a 'MAKEOPTS="-j17 -l16" emerge qtwebengine' so I'll update this with the results from the other day. I don't notice any responsiveness slowdowns but it does look like the cpus are all loaded up pretty good. This isn't a direct comparison to your system because I can obviously flirt above 16G of ram without touching swap but might give some ideas as far as loads go. I suspect the -j17 isn't the best setting overall but then you don't have a --jobs set (no idea what the default is or if it derives something when not set). Also qtwebengine might not be the best example since it uses tons of ram to build...

I think I see your problem. You have your tmpfs using 10G of ram. Some builds are going to eat this up plus the unused ram, but you're not set for parallel builds (that I can tell) so maybe not an issue but if you have failed builds stuck in the tmpfs they can eat up the ram unless you have failed-build set. With 16G of ram you're best to setup a notmpfs env for webkit, webengine, maybe firefox, anything that needs the max ram available when building. I was forced into this since about January or so on my old system with 16G ram due to either gcc changes or something and combined with a later discovered failing ram module. Some builds would just spike on ram use combined with my parallelism and failed builds not cleaning up I'd hit swap and then the whole system freezes for minutes.

10 minutes in the Load avg: 15.6, 13.4, 7.8.
After 15 minutes the Load avg: 15.9, 15.1, 10.5.
Finished Load avg: 9.3, 14.1, 14.2

MAKEOPTS="-j17 -l16" emerge qtwebengine -va1 34042.54s user 1404.11s system 1383% cpu 42:42.64 total

Code:

dev-qt/qtwebengine-5.12.4:5/5.12::gentoo [5.12.4:5/5.12::qt] USE="alsa jumbo-build pulseaudio system-ffmpeg system-icu widgets -bindist -debug -designer -pax_kernel -test

     Thu Jul  4 11:35:56 2019 >>> dev-qt/qtwebengine-5.12.4
       merge time: 42 minutes and 44 seconds.

     Fri Jul 12 09:35:14 2019 >>> dev-qt/qtwebengine-5.12.4
       merge time: 42 minutes and 7 seconds.

No improvement, this was while browsing the forum here and similar to the last build time.
_________________
#gentoo-kde on freenode
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Fri Jul 12, 2019 4:45 pm    Post subject: Reply with quote

mike155 wrote:
Try the following:
  1. Set
    Code:
    MAKEOPTS="-j16"
    and run
    Code:
    time emerge --oneshot gcc

  2. Disable hyperthreading in your BIOS and make sure that it's disabled after you booted Linux (look at the output of 'lscpu': it should print: 'CPU(s): 8' and 'Thread(s) per core: 1').

    Set
    Code:
    MAKEOPTS="-j8"
    and run
    Code:
    time emerge --oneshot gcc

  3. Compare the two results. Most probably, you will see that the hyperthreading test is NOT faster than the non-hyperthreading test. That's the reason why I disable hyperthreading on all of my machines. Disabling hyperthreading also reduces the amount of RAM needed and makes administration and monitoring easier.

it is impossible to disable ht on my mb, I think it is something with the cpu
gcc never caused slowdowns on freeze, I think the issue is with programs that requires space to compile such as firefox, libreoffice, dev-qt/qtwebkit, webkit-gtk or icedtea which I've installed.

I wonder, I see that dev-qt/qtwebkit or webkit-gtk requires more than 20GB, my /var and 20GB and the tempfs is 10GB, looks like I should increase my /var folder and remove the tmpfs mount or I should get more ram.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Fri Jul 12, 2019 5:10 pm    Post subject: Reply with quote

qtwebengine is a pkg I don't intend to install on my system.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
firephoto
Veteran
Veteran


Joined: 29 Oct 2003
Posts: 1604
Location: +48° 5' 23.40", -119° 48' 30.00"

PostPosted: Fri Jul 12, 2019 5:16 pm    Post subject: Reply with quote

DaggyStyle wrote:
qtwebengine is a pkg I don't intend to install on my system.


Which is irrelevant to the problem. It was an example of -j13 vs -j17 on a resource intensive package.

You've realized your problem. Setup notmpfs for those packages you've listed.

https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs#Per-package_choices_at_compile_time

Don't mess up the mount point and land it inside an existing tmp that is tmpfs. (I did that once).

Firefox before and after. -j13 vs -j17.

Code:

     Wed Jul 10 18:14:28 2019 >>> www-client/firefox-68.0
       merge time: 21 minutes and 29 seconds.

     Fri Jul 12 10:22:07 2019 >>> www-client/firefox-68.0
       merge time: 22 minutes and 14 seconds.

_________________
#gentoo-kde on freenode


Last edited by firephoto on Fri Jul 12, 2019 5:39 pm; edited 2 times in total
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Fri Jul 12, 2019 5:36 pm    Post subject: Reply with quote

I've ended up binding /var/tmp/portage to an old hdd I have, might loose performance but better than the alternative
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
firephoto
Veteran
Veteran


Joined: 29 Oct 2003
Posts: 1604
Location: +48° 5' 23.40", -119° 48' 30.00"

PostPosted: Fri Jul 12, 2019 5:40 pm    Post subject: Reply with quote

DaggyStyle wrote:
I've ended up binding /var/tmp/portage to an old hdd I have, might loose performance but better than the alternative



Trying with SMT off in the bios now. -j8 -l8 . I figure since it's using actual cores now I'd stick with actual numbers instead of bumping it +1. I suspect it's going to be a slower build but hopefully I'm wrong.

Do you have any compile times for firefox or the others you listed? Curious what those might be for reference.

SMT off time was longer but not much and I was watching youtube so probably about the same.

Code:

     Fri Jul 12 10:22:07 2019 >>> www-client/firefox-68.0
       merge time: 22 minutes and 14 seconds.

     Fri Jul 12 10:57:22 2019 >>> www-client/firefox-68.0
       merge time: 25 minutes and 4 seconds.

_________________
#gentoo-kde on freenode
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5369

PostPosted: Fri Jul 12, 2019 6:43 pm    Post subject: Reply with quote

Code:

     Wed Jun 12 19:16:17 2019 >>> www-client/firefox-67.0.2
       merge time: 58 minutes and 55 seconds.

     Thu Jun 20 12:46:46 2019 >>> www-client/firefox-67.0.3
       merge time: 56 minutes and 48 seconds.

     Fri Jun 21 08:07:37 2019 >>> www-client/firefox-67.0.4
       merge time: 40 minutes and 13 seconds.

     Thu Jul 11 19:31:30 2019 >>> www-client/firefox-68.0
       merge time: 1 hour, 6 minutes and 25 seconds.

     Sat May  4 22:36:16 2019 >>> app-office/libreoffice-6.1.5.2
       merge time: 1 hour, 19 minutes and 28 seconds.

     Thu May 23 03:21:25 2019 >>> app-office/libreoffice-6.1.5.2
       merge time: 2 hours, 41 minutes and 18 seconds.

     Sat Jun  8 07:52:36 2019 >>> app-office/libreoffice-6.1.5.2
       merge time: 45 minutes and 52 seconds.

     Mon Jun 10 20:03:50 2019 >>> app-office/libreoffice-6.2.4.2
       merge time: 44 minutes and 26 seconds.

     Sat Jan 26 12:46:46 2019 >>> dev-qt/qtwebkit-5.212.0_pre20180120
       merge time: 20 minutes and 22 seconds.

     Sat Feb 16 23:26:27 2019 >>> dev-qt/qtwebkit-5.212.0_pre20180120
       merge time: 51 minutes and 41 seconds.

     Sat May  4 20:30:33 2019 >>> dev-qt/qtwebkit-5.212.0_pre20180120
       merge time: 37 minutes and 48 seconds.

     Thu May 23 02:35:26 2019 >>> dev-qt/qtwebkit-5.212.0_pre20180120
       merge time: 2 hours, 21 minutes and 39 seconds.

     Sat May  4 21:58:33 2019 >>> net-libs/webkit-gtk-2.24.1
       merge time: 41 minutes and 57 seconds.

     Mon May 20 19:49:16 2019 >>> net-libs/webkit-gtk-2.24.2
       merge time: 22 minutes and 17 seconds.

     Thu May 23 02:36:57 2019 >>> net-libs/webkit-gtk-2.24.2
       merge time: 1 hour, 56 minutes and 15 seconds.

     Fri Jun 14 13:00:44 2019 >>> net-libs/webkit-gtk-2.24.2
       merge time: 21 minutes and 2 seconds.

     Sat Mar 23 21:27:08 2019 >>> dev-java/icedtea-3.11.0
       merge time: 9 minutes and 44 seconds.

     Sat Mar 30 19:22:50 2019 >>> dev-java/icedtea-3.11.0
       merge time: 9 minutes and 57 seconds.

     Sat May  4 15:55:09 2019 >>> dev-java/icedtea-3.11.0
       merge time: 9 minutes and 21 seconds.

     Fri May 10 08:51:36 2019 >>> dev-java/icedtea-3.12.0
       merge time: 9 minutes and 25 seconds.


I assume the spikes are due to the memory consumption.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum