Seeking to Optimize dev-qt/qtwebengine compile

Message

cctsurf · Post by **cctsurf** » Wed Jan 18, 2023 4:00 am

I have a pretty powerful computer (ryzen 3700x, 32gb of ram, compiling in zram file system), and I keep having to make concessions to build dev-qt/qtwebengine. I'm down to MAKEOPTS=-j1 and it fails to compile as part of a larger emerge, but will compile on its own.
Last night it took 12 hours. This is insane for this system!
I'm looking for ways to optimize my compile so that I don't have to wait this long.
My ram usage never gets above 16gb during a compile, and usually stays just about 8gb. Yet I somehow seem to be running out of system resources.
Would I be better off running with all my cores or at least more cores without the "jumbo-build" use flag enabled?

Here's my emerge --info, but I really don't want to do this just for me. If possible, I would like to put together a sticky for how to compile this in a somewhat efficient manner.

Code: Select all

emerge --info dev-qt/qtwebengine
Portage 3.0.44 (python 3.10.9-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-12, glibc-2.36-r6, 6.0.11-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-6.0.11-gentoo-x86_64-AMD_Ryzen_7_3700X_8-Core_Processor-with-glibc2.36
KiB Mem:    32802952 total,  21987788 free
KiB Swap:   16777212 total,  16777212 free
Timestamp of repository gentoo: Tue, 17 Jan 2023 00:45:01 +0000
Head commit of repository gentoo: 08fd3aa913284cddc9f861e9b27b0087455d827d
Timestamp of repository brave-overlay: Mon, 16 Jan 2023 21:17:22 +0000
Head commit of repository brave-overlay: 0e61a2d526183621ffcb84972c0432709f00e4f0

sh bash 5.2_p15-r1
ld GNU ld (Gentoo 2.40 p1) 2.40
app-misc/pax-utils:        1.3.6-r1::gentoo
app-shells/bash:           5.2_p15-r1::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.36.0-r1::gentoo
dev-lang/python:           3.10.9::gentoo, 3.11.1::gentoo, 3.12.0_alpha4::gentoo
dev-lang/rust:             1.66.1::gentoo
dev-util/cmake:            3.25.1::gentoo
dev-util/meson:            1.0.0::gentoo
sys-apps/baselayout:       2.9::gentoo
sys-apps/openrc:           0.46::gentoo
sys-apps/sandbox:          2.30-r1::gentoo
sys-devel/autoconf:        2.71-r5::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.40::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/gcc:             12.2.1_p20221231::gentoo
sys-devel/gcc-config:      2.10::gentoo
sys-devel/libtool:         2.4.7-r1::gentoo
sys-devel/llvm:            15.0.7::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  6.1::gentoo (virtual/os-headers)
sys-libs/glibc:            2.36-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    volatile: True
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 1

Home
    location: /usr/local/portage
    masters: gentoo
    priority: 0
    volatile: True

brave-overlay
    location: /var/db/repos/brave-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/brave-overlay.git
    masters: gentoo
    volatile: True

crossdev
    location: /var/db/repos/portage-crossdev
    masters: gentoo
    priority: 10
    volatile: True

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* PUEL"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/mnt/nfs_portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs 9 --with-bdeps y --buildpkg-exclude "virtual/* sys-kernel/*-sources""
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.osuosl.org/     http://gentoo-mirror.flux.utah.edu/     http://www.gtlib.gatech.edu/pub/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
LINGUAS="en en_US"
MAKEOPTS="--jobs 9 --load-average 9"
PKGDIR="/mnt/nfs_portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac aacs acl acpi activities alsa amd64 audiofile bluetooth bluray branding bzip2 cairo cdda cddb cdio cdr cli crypt css cups dbus declarative dri dts dv dvb dvd dvdnav dvdr dvdread elogind enca encode exif extras ffmpeg flac fontconfig fortran gdbm gif gimp gpm gtk gui iconv icu imagemagick java jpeg json kde kwallet ladspa lame lcms libglvnd libnotify libtirpc live lm-sensors lv2 mad midi mjpeg mng mp2 mp3 mp4 mpeg mtp musepack ncurses nls nptl ogg openal opengl openmp pam pango pcre pda pdf plasma png policykit ppds pulseaudio qml qt5 readline redistributable scanner sdl seccomp semantic-desktop snmp sound speech spell split-usr srt ssl startup-notification svg test-rust theora threads tiff truetype udev udisks unicode upower ups_drivers_usbhid-ups usb v4l vcd vim-syntax vorbis webkit widgets wxwidgets x264 x265 xattr xcb xft xinerama xml xmp xv xvfb xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2021" ALSA_CARDS="hda_intel usb_audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en_US la" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="pdfimport presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby25 ruby27 ruby30" SANE_BACKENDS="hp" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

dev-qt/qtwebengine-5.15.8_p20230112::gentoo was built with the following:
USE="alsa jumbo-build pulseaudio system-ffmpeg system-icu widgets -bindist -debug -designer -geolocation -kerberos -screencast -test" ABI_X86="(64)"
FEATURES="protect-owned config-protect-if-modified usersync distlocks news sfperms qa-unresolved-soname-deps unknown-features-warn ebuild-locks pid-sandbox usersandbox binpkg-docompress buildpkg-live sandbox fixlafiles multilib-strict network-sandbox ipc-sandbox unmerge-logs userfetch userpriv assume-digests strict xattr merge-sync unmerge-orphans binpkg-multi-instance binpkg-dostrip preserve-libs binpkg-logs buildpkg parallel-fetch"

Post by **sam_** » Wed Jan 18, 2023 4:36 am

This is the classic issue of trying to balance emerge jobs (you have --jobs 9 in EMERGE_DEFAULT_OPTS) with make jobs. At worst, when things go badly, you have 9*9=49 jobs running. If you assume that each job may take up at most 2GB of RAM, you then need 98GB of RAM. You can try mitigate this a bit by using --load-average in EMERGE_DEFAULT_OPTS too. I don't see how you could never have it go above 16GB of RAM with these settings, unless you had extraordinary luck and terrible parallelisation.

It would help to share a build.log of a failed compile from qtwebengine as well.

I believe that if compiling qtwebengine with Clang, it may take slightly less RAM to compile. But that's really missing the main problem as I mentioned.

ajgringo619 · Post by **ajgringo619** » Wed Jan 18, 2023 4:58 am

On my Gentoo KDE VM, I got this built in under 90 minutes with this setup:

Code: Select all

8 GB RAM, 8 vCPUs (Ryzen 7 2700X)
MAKEOPTS="-j3"
EMERGE_DEFAULT_OPTS="--jobs 2 --load-average 6.8"

I never run out of RAM, barely ever use any swap. As @sam_ pointed out, you must be overloading you system RAM, zRAM/swap something terrible. Never have tried with clang, though.

logrusx · Post by **logrusx** » Wed Jan 18, 2023 9:53 am

I see odd numbers for make jobs and although in thins particular case it's not likely you're running with hyerthreading disabled and thus with more OS than platform threads(emerge jobs addressed separately), I feel I need to address that issue.

Setting the number of make jobs to the number of threads + 1 is a huge misconception. I don't know where it originated from, but it shows its inventor's lack of understanding of how threads work and how that pertains to multiple CPU's, including multi-core CPU's.

Having an additional thread for the IO or whatever makes no sense. As soon as a thread blocks, another one is spawned and as soon as a thread finishes or blocks in turn, another one is spawned or awaken, thus keeping the number of active threads the same as the number of make jobs. The "IO thread" or whatever the person who invented it calls it is blocked, so the result is one extra OS thread which must be scheduled on a time sharing basis with the others, as opposed to per core basis, resulting in expensive context switches and unnecessary memory usage.

Having more OS threads than platform threads is only suitable for low CPU, high IO load where context switches would be insignificant compared to the delays of the IO operations. Portage cannot make difference of what thread does what, so it's best to keep to the formula #thread <= #(logical, if HT is enabled)cores.

Furthermore --load-average won't aid the insufficient memory issue. This is not something that's directly correlated. What I used to do was to have (RAM in gigabytes / 2) number of threads, disable physical swap and setup a zram swap drive prior to compiling such memory heavy packages to prevent swap thrashing and OOM's for just a few megs above the limit, or even a byte above it can cause an OOM.

The issue mentioned by sam_ is sure to cause OOM's or heavy swap use, which if you have an SSD wears it out and if you have an HDD makes your system unusable and it'll eventually never finish.

Nowadays I don't suffer the issue of lack of RAM and threads, but I still use zswap to prevent the swap using the SSD too much. I'd disable the swap, but I'd like the option to be able to hibernate the computer, so the zswap is good since swap usage when compiling chromium (which qtwebengine is) rarely exceed a couple of hundred megabytes. I've never tested if it the compressed swap frontend consumes it all or a smal part is swapped, but this is acceptable.

Make note I'm talking about zram and zswap, which are different things, on the old computer I didn't use physical swap and thus used zram, on the new I need swap to hibernate and thus I'm using zswap, but I have sufficient amount of RAM.

Also have in mind that zram consumes about 10% of the size it's setup to have just to be active, so be careful not to give it too large size. I used to set it up for something like 1-2 GB.

In the same time, the more memory you have, that allows for more threads, which makes the conditions for which all of threads consume 2 or more GB of RAM less likely, compared to for example 4 threads and 8 GB RAM, and the worst case is 2 threads. It regularly gave me OOM's even with 6GB RAM back then. I guess for one thread it needs at like 4GB to run smoothly, but I hope nowadays nobody needs to run like that.

@ajgringo619, my experience with clang is it has higher memory usage peaks thus more likely to cause OOM's.

Best Regards,
Georgi

wjb · Post by **wjb** » Wed Jan 18, 2023 10:57 am

With "--jobs 9" set in EMERGE_DEFAULT_OPTIONS, you've got potentially 9 ebuilds running simultaneously. The quick ebuilds shuffle on through maybe, but consider what happens when the huge builds try to run at the same time - each trying to use 9 threads @~2G RAM (from MAKEOPTS). Try it without the --jobs in EMERGE_DEFAULT_OPTIONS and set the MAKEOPTS value to N, where 32G > N*2G - TMPFS

TMPFS is the amount of RAM allocated to tmpfs at /var/tmp/portage, if you've done that.

MAKEOPTS can be tweaked per package using package.env

Section_8 · Post by **Section_8** » Wed Jan 18, 2023 1:43 pm

I also have a Ryzen 3700x with 32g ram. My last few builds of qtwebengine have taken around 2-3 hours:

Code: Select all

john@officepc ~ $ qlop -mt qtwebengine
...
2021-09-24T18:13:36 >>> dev-qt/qtwebengine: 1 hour, 57 minutes, 27 seconds
2021-11-12T18:15:26 >>> dev-qt/qtwebengine: 2 hours, 21 minutes, 10 seconds
2021-12-03T18:57:28 >>> dev-qt/qtwebengine: 2 hours, 44 minutes, 35 seconds
2021-12-31T10:52:12 >>> dev-qt/qtwebengine: 2 hours, 9 minutes, 28 seconds
2022-02-04T17:23:55 >>> dev-qt/qtwebengine: 2 hours, 56 minutes, 15 seconds
2022-02-25T16:30:47 >>> dev-qt/qtwebengine: 2 hours, 40 minutes, 20 seconds
2022-04-16T10:08:39 >>> dev-qt/qtwebengine: 2 hours, 8 minutes, 33 seconds
2022-06-10T20:06:22 >>> dev-qt/qtwebengine: 3 hours, 6 minutes, 22 seconds
2022-07-08T17:44:55 >>> dev-qt/qtwebengine: 2 hours, 34 minutes, 47 seconds
2022-11-26T10:20:51 >>> dev-qt/qtwebengine: 2 hours, 8 minutes, 3 seconds
2022-12-09T11:32:49 >>> dev-qt/qtwebengine: 2 hours, 32 minutes, 45 seconds

My emerge --info:

Code: Select all

john@officepc ~ $ emerge --info qtwebengine
Portage 3.0.43 (python 3.10.9-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-11, glibc-2.36-r5, 5.15.80-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-5.15.80-gentoo-x86_64-AMD_Ryzen_7_3700X_8-Core_Processor-with-glibc2.36
KiB Mem:    32804488 total,  12673372 free
KiB Swap:   43285500 total,  43283696 free
Timestamp of repository gentoo: Fri, 13 Jan 2023 14:02:25 +0000
Head commit of repository gentoo: 781a09f8c121f747991160e79434082a99167f83

Timestamp of repository brother-overlay: Tue, 03 Jan 2023 09:03:44 +0000
Head commit of repository brother-overlay: 3228cf8c0fcaebd419b2570f3ec28095c4541057

Timestamp of repository src_prepare-overlay: Fri, 13 Jan 2023 09:32:04 +0000
Head commit of repository src_prepare-overlay: 5f12cc410d17c366390ab2c7700cc5d8609ea50f

Timestamp of repository steam-overlay: Tue, 10 Jan 2023 17:47:44 +0000
Head commit of repository steam-overlay: 03e78314c4c6343b9fe7aab35b4538b0a3aba524

sh bash 5.1_p16-r2
ld GNU ld (Gentoo 2.39 p5) 2.39.0
app-misc/pax-utils:        1.3.5::gentoo
app-shells/bash:           5.1_p16-r2::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.36.0-r1::gentoo
dev-lang/python:           3.10.9::gentoo, 3.11.1::gentoo
dev-lang/rust:             1.66.1::gentoo
dev-util/cmake:            3.24.3::gentoo
dev-util/meson:            0.64.1::gentoo
sys-apps/baselayout:       2.9::gentoo
sys-apps/openrc:           0.45.2-r2::gentoo
sys-apps/sandbox:          2.29::gentoo
sys-devel/autoconf:        2.13-r7::gentoo, 2.71-r5::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.39-r4::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/clang:           15.0.6-r1::gentoo
sys-devel/gcc:             11.3.1_p20221209::gentoo
sys-devel/gcc-config:      2.8::gentoo
sys-devel/libtool:         2.4.7::gentoo
sys-devel/lld:             15.0.6::gentoo
sys-devel/llvm:            15.0.6-r1::gentoo
sys-devel/make:            4.3::gentoo
sys-kernel/linux-headers:  5.15-r3::gentoo (virtual/os-headers)
sys-libs/glibc:            2.36-r5::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000
    volatile: True
    sync-git-verify-commit-signature: true

brother-overlay
    location: /var/db/repos/brother-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/brother-overlay.git
    masters: gentoo
    volatile: True

officepc
    location: /var/db/repos/localrepo
    masters: gentoo
    volatile: True

src_prepare-overlay
    location: /var/db/repos/src_prepare-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/src_prepare-overlay.git
    masters: gentoo
    volatile: True

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo
    volatile: True

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE PUEL dlj-1.1 googleearth AdobeFlash-11.x Oracle-BCLA-JavaSE google-chrome RAR RTCW unRAR"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/init.d /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /usr/share/x11"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/mnt/data/portage_distfiles"
EMERGE_DEFAULT_OPTS="--jobs 4 --load-average 17 --quiet-build --with-bdeps y --verbose-conflicts --autounmask-write=n --autounmask-keep-masks"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg buildpkg-live clean-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-march=native -O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US -/usr/share/locale/en_GB"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
LINGUAS="en-US"
MAKEOPTS="-j4 -l17"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X a52 aac acpi activities alsa amd64 audiofile bash-completion bluetooth branding bzip2 cdda cdparanoia cdr cleartype cli clucene cups dbus declarative dri dvd dvdr dvdread elogind encode exif flac gif gphoto2 gpm gui iconv icu imagemagick infinality joystick jpeg kde kipi kwallet lcms libglvnd libtirpc lm-sensors lock lvm mad midi mng mp3 mp4 mpeg mpris mtp multilib musicbrainz nas nptl nvidia ogg opengl openmp pam pango pcre pdf plasma png policykit ppds pulseaudio qml qt5 readline realtime scanner sdl seccomp semantic-desktop sound spell split-usr ssl startup-notification test-rust threads tiff truetype udev udisks urandom usb v4l v4l2 vorbis widgets wxwidgets x264 xattr xcb xcomposite xft xine xinerama xml xulrunner xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2021" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en-US" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27" SANE_BACKENDS="epson epson2" USERLAND="GNU" VIDEO_CARDS="nvidia vesa fbdev" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, LC_ALL, LD, LFLAGS, LIBTOOL, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

=================================================================
                        Package Settings
=================================================================

dev-qt/qtwebengine-5.15.7_p20221122::gentoo was built with the following:
USE="alsa jumbo-build pulseaudio system-ffmpeg system-icu widgets -bindist -debug -designer -geolocation -kerberos -screencast -test" ABI_X86="(64)"
FEATURES="binpkg-dostrip config-protect-if-modified qa-unresolved-soname-deps buildpkg binpkg-docompress distlocks strict clean-logs pid-sandbox assume-digests merge-sync news binpkg-logs userfetch protect-owned unmerge-logs userpriv preserve-libs sfperms usersandbox fixlafiles ebuild-locks buildpkg-live parallel-fetch unmerge-orphans network-sandbox usersync sandbox unknown-features-warn ipc-sandbox multilib-strict"

I have /var/tmp/portage as a 16g tmpfs, but qtwebengine has PORTAGE_TMPDIR set to a HDD in package.env.

steve_v · Post by **steve_v** » Wed Jan 18, 2023 1:57 pm

On the topic of qtwebengine...

Code: Select all

>>> Running pre-merge checks for dev-qt/qtwebengine-5.15.8_p20230112
 * Checking for at least 20 GiB RAM ...                                                                                                             [ ok ]
 * Checking for at least 7 GiB disk space at "/var/tmp/portage/dev-qt/qtwebengine-5.15.8_p20230112/temp" ...                                        [ ok ]
 * Checking for at least 150 MiB disk space at "/usr" ...

Seriously? 20GB of RAM? You have got to be kitten me.

Presumably this takes make threads into account, right? So:

Code: Select all

$ grep qtwebengine /etc/portage/package.env 
dev-qt/qtwebengine reduced-parallel-build.conf
$ cat /etc/portage/env/reduced-parallel-build.conf 
MAKEOPTS="-j10 -l10"

is 2GB * 10?

Also:

Code: Select all

     Mon Dec  5 09:17:50 2022 >>> dev-qt/qtwebengine-5.15.7_p20221122
       merge time: 1 hour, 9 minutes and 9 seconds.

If it's taking OP 12 hours there's something not right. This box is quick, but it's not that quick.

grknight · Post by **grknight** » Wed Jan 18, 2023 2:07 pm

steve_v wrote:On the topic of qtwebengine...

Code: Select all

>>> Running pre-merge checks for dev-qt/qtwebengine-5.15.8_p20230112
 * Checking for at least 20 GiB RAM ...                                                                                                             [ ok ]
 * Checking for at least 7 GiB disk space at "/var/tmp/portage/dev-qt/qtwebengine-5.15.8_p20230112/temp" ...                                        [ ok ]
 * Checking for at least 150 MiB disk space at "/usr" ...

Seriously? 20GB of RAM? You have got to be kitten me.

Presumably this takes make threads into account, right? So:

Code: Select all

$ grep qtwebengine /etc/portage/package.env 
dev-qt/qtwebengine reduced-parallel-build.conf
$ cat /etc/portage/env/reduced-parallel-build.conf 
MAKEOPTS="-j10 -l10"

is 2GB * 10?

Code is behind this:

Code: Select all

        # Estimate the amount of RAM required
        # Multiplier is *10 because Bash doesn't do floating point maths.
        # Let's crudely assume ~2GB per compiler job for GCC.
        local multiplier=20

        # And call it ~1.5GB for Clang.
        if tc-is-clang ; then
                multiplier=15
        fi

        local CHECKREQS_DISK_BUILD="7G"
        local CHECKREQS_DISK_USR="150M"
        if ! has "distcc" ${FEATURES} ; then
                # bug #830661
                # Not super realistic to come up with good estimates for distcc right now
                local CHECKREQS_MEMORY=$(($(makeopts_jobs)*multiplier/10))G
        fi

So on gcc and -j10 in MAKEOPTS, assume 20GiB as in your output

steve_v · Post by **steve_v** » Wed Jan 18, 2023 2:15 pm

Good to know. I was starting to wonder if this monstrosity had metastasised again.

cctsurf · Post by **cctsurf** » Wed Jan 18, 2023 2:27 pm

Appreciate all the help!
I should note that I am changing the MAKEOPTS in /etc/portage/package.env to "-j1" right now. BTW, would it be possible to change --jobs and --load-average in there, too?

As to Makeopts, I essentially followed what was said here a few months ago: https://wiki.gentoo.org/wiki/MAKEOPTS
when I ran into this problem with an icedtea build failure related to make 4.4 and an error about "use MAKE JOBS=n". viewtopic-t-1159103-highlight-.html It was unrelated to the Makeopts, but I kept the change because it moved all the little builds through quite nicely...
Previous to that I had been using MAKEOPTS="-j9" I also had trouble building a couple of packages that way, including qtwebengine.

As a side note, I have NO desire to move to CLANG for building to save memory. I think I should have enough memory to build anything on this system... If I don't, I'm doing something wrong.

As to the "#threads +1" comment. It was the suggestion of the Gentoo Handbook back in the day. Here's a reference to that fact in 2013, https://blogs.gentoo.org/ago/2013/01/14 ... imization/ It always worked, and I haven't had a lot of reasons to change it... Notice that I am not doing that right now, my CPU has 16 threads, and I'm running 9.

I guess my question still stands, and maybe it's treated somewhere and I'm missing it, but with these HUGE compiles, it would be helpful to have some discussion on what are the best settings to compile them in the least amount of time.
Thanks!

Post by **sam_** » Wed Jan 18, 2023 3:40 pm

Please see my request for a build.log. Also, you don't seem to have addressed the issue I raised about different types of jobs.

Post by **pietinger** » Wed Jan 18, 2023 3:56 pm

cctsurf,

maybe you want read this: https://wiki.gentoo.org/wiki/User:Pieti ... FAULT_OPTS

cctsurf · Post by **cctsurf** » Wed Jan 18, 2023 4:31 pm

pietinger wrote:cctsurf,

maybe you want read this: https://wiki.gentoo.org/wiki/User:Pieti ... FAULT_OPTS

That is EXACTLY the sort of thing I was looking for. I have a similar set of things in the package.env and env directories, but getting to the correct settings is what has escaped me, and is essentially my question. I especially appreciate your discussion on not using the tmpfs for portage's tmpdir and why not to use it with these massive builds. I thought perhaps I could push it through faster with the build tree in ram. I wish this were on the wiki.gentoo.org site / more accessible to the gentooing public.

@sam_, I see what you are saying about mathmatically needing 98gb of ram, and I am looking at changing that. In practice for most builds, it doesn't get anywhere near that and it seems really to move the little builds along. As I asked in my last post, would it be possible to change the --jobs and --load-average settings in the portage.env and env settings for just these massive builds? I'll see if I can get a build.log file sometime here, this is a difficulty with building in a tmpfs, they disappear when I shut down the computer.

Thanks again!!

Post by **sam_** » Wed Jan 18, 2023 4:45 pm

cctsurf wrote:
pietinger wrote:cctsurf,

maybe you want read this: https://wiki.gentoo.org/wiki/User:Pieti ... FAULT_OPTS

That is EXACTLY the sort of thing I was looking for. I have a similar set of things in the package.env and env directories, but getting to the correct settings is what has escaped me, and is essentially my question. I especially appreciate your discussion on not using the tmpfs for portage's tmpdir and why not to use it with these massive builds. I thought perhaps I could push it through faster with the build tree in ram. I wish this were on the wiki.gentoo.org site / more accessible to the gentooing public.

@sam_, I see what you are saying about mathmatically needing 98gb of ram, and I am looking at changing that. In practice for most builds, it doesn't get anywhere near that and it seems really to move the little builds along. As I asked in my last post, would it be possible to change the --jobs and --load-average settings in the portage.env and env settings for just these massive builds? I'll see if I can get a build.log file sometime here, this is a difficulty with building in a tmpfs, they disappear when I shut down the computer.

Thanks again!!

Unfortunately, you can't vary EMERGE_DEFAULT_OPTS per-package as far as I know (if it works - and it shouldn't, I'd consider it undefined behaviour, because emerge is a global thing, not a per-package thing -- how do you apply --load-average to emerge only for a single package?)

You can vary MAKEOPTS in package.env though, of course (which can include --jobs and --load-average for make/ninja/..., it covers all of them in one variable).

My guess is that a bunch of other packages get built in parallel when you're building qtwebengine and then the OOM.

Post by **pietinger** » Wed Jan 18, 2023 4:48 pm

cctsurf wrote:[...] I especially appreciate your discussion on not using the tmpfs for portage's tmpdir and why not to use it with these massive builds.[...]

Please dont misunderstand me. I am using a tmpfs also - because my SSD loves it - for all other packages. I am really using these examples (in my article) for my system.

cctsurf wrote:Thanks again!!

You are very welcome !

Have fun with Gentoo.