Unable to emerge gcc and many packages

Message

atrix · Post by **atrix** » Mon Jan 16, 2023 3:54 am

An old issue I was facing back in March has resurfaced. Several packages will "hang" during compilation (100% CPU usage on a single core and will not progress further). I made a post back then and did quite a bit of troubleshooting trying to narrow down the root cause, but was unable to - viewtopic-t-1148318-view-previous.html . Ultimately, I rebuilt my system in a stage 3 chroot and copied it to my actual system and things were working as expected up until December.

All of the symptoms described in my previous post are suddenly present again - no I/O, no strace output, no overutilization of memory/swapping, nothing in kernel logs. The same problematic packages will not build unless I remove -O optimization in gcc, and this only works for some packages (opus). Some packages will build with my normal flags, many will not.

GCC appears to be the culprit here, and I am unable to upgrade, downgrade, or rebuild the current version of both gcc 11 and gcc 12. When attempting to emerge any version of gcc, progress stops and f951 is using 100% of a single core. Another example is libtool, the build hangs and cc1 uses 100% of a single core.

I've tried restoring gcc from several different backups but see the same behavior. I've also pulled gcc from stage3, same result. My understanding is that gcc builds the tools needed to build itself in a "clean" environment after using some low level tools to build an environment for itself, presumably f951 (fortran?), so not seeing a difference between older versions that used to work and from stage3 was surprising to me. I don't understand gcc enough to know what I'm missing and am assuming it has to do with something that was upgraded in late November or early December, but I'm unable to downgrade a handful of packages to test due to the issue with gcc, so I'm stuck on this one and am unable to update my system. This is a list of packages emerged around the time I believe the problem started (https://dpaste.com/5VSHW4BMS).

I can restore from backup, but since this is the second instance of this occurring I'm really hoping to determine why it's happened again so I know how to deal with this going forward.

SessionID removed by NeddySeagoon

Post by **NeddySeagoon** » Mon Jan 16, 2023 11:05 am

atrix,

Post your

Code: Select all

emerge --info

output and pastebin the entire build log. for a package that fails. Preferably not gcc.

atrix · Post by **atrix** » Mon Jan 16, 2023 6:23 pm

Sure thing.

emerge --info:

Code: Select all

Portage 3.0.39 (python 3.10.8-final-0, default/linux/arm64/17.0/systemd, gcc-12, glibc-2.36-r6, 6.1.1-gentoo aarch64)
=================================================================
System uname: Linux-6.1.1-gentoo-aarch64-with-glibc2.36
KiB Mem:     3782148 total,   1442000 free
KiB Swap:   20970492 total,  20548604 free
Timestamp of repository gentoo: Mon, 16 Jan 2023 01:00:01 +0000
Head commit of repository gentoo: 3f76e934ed8247476725cf09d0b364959e180734
Timestamp of repository hamari: Sun, 15 Jan 2023 16:02:17 +0000
Head commit of repository hamari: 4166d05182c2759537620ea50faf5c3d130840a3

Timestamp of repository natinst: Sat, 14 Jan 2023 11:33:00 +0000
Head commit of repository natinst: 0437d174cf0b38fd155538b53af0022e3940d6fe

Timestamp of repository pentoo: Sat, 14 Jan 2023 11:32:40 +0000
Head commit of repository pentoo: c953fb562400a3ae6ebff292922672bec10495f9

Timestamp of repository quarks: Sat, 14 Jan 2023 11:32:40 +0000
Head commit of repository quarks: 240d410448d2dff776559cb74a0ea73234a2e49e

Timestamp of repository tatsh-overlay: Mon, 16 Jan 2023 00:47:02 +0000
Head commit of repository tatsh-overlay: 051b793d759ed8fa8c17888daa4309adc3c4344d

Timestamp of repository thegreatmcpain: Sun, 15 Jan 2023 03:47:03 +0000
Head commit of repository thegreatmcpain: 1cfa0f454d581aa8ad582cb79143dcb03169fef2

sh bash 5.2_p15
ld GNU ld (Gentoo 2.39 p5) 2.39.0
distcc 3.4 aarch64-unknown-linux-gnu [disabled]
ccache version 4.7.4 [disabled]
app-misc/pax-utils:        1.3.5::gentoo
app-shells/bash:           5.2_p15::gentoo
dev-java/java-config:      2.3.1::gentoo
dev-lang/perl:             5.36.0-r1::gentoo
dev-lang/python:           3.9.15_p3::gentoo, 3.10.8_p3::gentoo, 3.11.0_p2::gentoo, 3.12.0_alpha2::gentoo
dev-lang/rust:             1.65.0::gentoo
dev-lang/rust-bin:         1.66.0::gentoo
dev-util/ccache:           4.7.4::gentoo
dev-util/cmake:            3.25.0::gentoo
dev-util/meson:            0.64.1::gentoo
sys-apps/baselayout:       2.9::gentoo
sys-apps/sandbox:          2.29::gentoo
sys-apps/systemd:          252.2::gentoo
sys-devel/autoconf:        2.13-r7::gentoo, 2.71-r5::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.39-r4::gentoo
sys-devel/binutils-config: 5.4.1::gentoo
sys-devel/clang:           14.0.6-r1::gentoo, 15.0.5::gentoo
sys-devel/gcc:             11.3.1_p20221007::gentoo, 12.2.1_p20221008::gentoo
sys-devel/gcc-config:      2.8::gentoo
sys-devel/libtool:         2.4.7-r1::gentoo
sys-devel/lld:             14.0.6-r1::gentoo
sys-devel/llvm:            14.0.6-r2::gentoo, 15.0.5::gentoo
sys-devel/make:            4.4::gentoo
sys-kernel/linux-headers:  6.1::gentoo (virtual/os-headers)
sys-libs/glibc:            2.36-r6::gentoo
Repositories:

gentoo
    location: /mnt/ssd/gentoo/var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: yes

hamari
    location: /mnt/ssd/gentoo/var/db/repos/hamari
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/hamari.git
    masters: gentoo

natinst
    location: /mnt/ssd/gentoo/var/db/repos/natinst
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/natinst.git
    masters: gentoo

pentoo
    location: /mnt/ssd/gentoo/var/db/repos/pentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/pentoo.git
    masters: gentoo

quarks
    location: /mnt/ssd/gentoo/var/db/repos/quarks
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/quarks.git
    masters: gentoo

tatsh-overlay
    location: /mnt/ssd/gentoo/var/db/repos/tatsh-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/tatsh-overlay.git
    masters: gentoo

thegreatmcpain
    location: /mnt/ssd/gentoo/var/db/repos/thegreatmcpain
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/thegreatmcpain.git
    masters: gentoo

ACCEPT_KEYWORDS="arm64 ~arm64"
ACCEPT_LICENSE="@FREE"
CBUILD="aarch64-unknown-linux-gnu"
CFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -ftree-vectorize -fuse-linker-plugin -flto=6 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -floop-strip-mine -floop-interchange -floop-block -ftree-loop-distribute-patterns -mfix-cortex-a53-835769 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256"
CHOST="aarch64-unknown-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt /var/lib/pihole"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.1/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -ftree-vectorize -fuse-linker-plugin -flto=6 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -floop-strip-mine -floop-interchange -floop-block -ftree-loop-distribute-patterns -mfix-cortex-a53-835769 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS=" --exclude sys-kernel/gentoo-sources"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -ftree-vectorize -fuse-linker-plugin -flto=6 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -floop-strip-mine -floop-interchange -floop-block -ftree-loop-distribute-patterns -mfix-cortex-a53-835769 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -ftree-vectorize -fuse-linker-plugin -flto=6 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -floop-strip-mine -floop-interchange -floop-block -ftree-loop-distribute-patterns -mfix-cortex-a53-835769 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256"
GENTOO_MIRRORS="rsync://mirrors.rit.edu/gentoo rsync://rsync.gtlib.gatech.edu/gentoo https://gentoo.osuosl.org https://mirrors.rit.edu/gentoo"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
MAKEOPTS="-j15 -l2"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/mnt/ssd/gentoo"
SHELL="/bin/bash"
USE="X aac acl alsa arm64 bash-completion bluetooth bzip2 cairo calendar clamdtop cli crypt curl dbus directfb dri egl fbcon ffmpeg flac fortran ftp fuse gdbm geoip gif gimp gles1 gles2 gnutls gstreamer gtk iconv imagemagick imap impv4 inotify introspection ipv6 javascript jingle jit jpeg lame ldap libglvnd libtirpc lirc lzo mad maildir mmap mp3 mp4 mpeg mplayer ncurses neon nfs nls nptl nsplugin odbc offensive ogg oggvorbis openmp openssl pam pcre pdf png posix postgres pulseaudio qt3support qt5 rar raw readline sasl sdl seccomp snmp sound spamassassin speex spell split-usr ssl svg symlink syslog systemd test-rust theora threads tiff udev unicode upnp upnp-av urandom usb vda video vorbis wayland xattr xml xmpp xorg xpm xrandr xv xvmc zlib" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="edsp thumb vfp vfpv3 vfpv4 vfp-d32 aes sha1 sha2 crc32 v4 v5 v6 v7 v8 thumb2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_10" PYTHON_TARGETS="python3_10" RUBY_TARGETS="ruby27 ruby30" USERLAND="GNU" VIDEO_CARDS="fbdev panfrost" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

Build log for libtool: https://dpaste.com/6TVWWUKWS

This one gets stuck at

Code: Select all

checking whether the compiler supports GNU Fortran 77...

Post by **NeddySeagoon** » Mon Jan 16, 2023 7:11 pm

atrix,

With only 4G RAM,

Code: Select all

MAKEOPTS="-j15 -l2"

is a bit aggressive. -j2 will be better but even that will fail on a few things.
What system is this?

Code: Select all

CFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -ftree-vectorize -fuse-linker-plugin -flto=6 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -floop-strip-mine -floop-interchange -floop-block -ftree-loop-distribute-patterns -mfix-cortex-a53-835769 --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=256"

is rather aggressive.

Use

Code: Select all

CFLAGS="-O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -mfix-cortex-a53-835769

Code: Select all

-flto=6

breaks things but its getting better, and needs lots of RAM. Its something that the linker does and even with distcc, linking is done on the helpee not the helpers.
Swap is not RAM substitute in lots of cases. dmesg will show you the OOM in action.

I suspect that your toolchain is broken due to the overly aggressive CFLAGS.
Packages that come to mind are gcc, glibc, binutils and texinfo. Maybe more.
Restore them from a stage3. Do not use your backups,
You will need to untar a stage3 into a new directory, then quickpkg the packages of interest.

If your system will run Raspberry Pi 4 code, I have a /binhost which you are welcome to use. The Pi does not have the crypto extensions.

atrix · Post by **atrix** » Mon Jan 16, 2023 8:56 pm

Thanks Neddy,

Those MAKEOPTS are there because I typically use distcc, but I've changed this to '-j2' and am using your recommended CFLAGS to simplify things. This is an Odroid N2+

I don't see anything from OOM in dmesg, but will keep an eye on this.

I've restored gcc (11), glibc, binutils and texinfo from stage3 quickpkg binaries. I selected gcc 11 in eselect/gcc-config. Emerging opus failed with "C compiler cannot create executables." I emerged binutils, then opus again and did not encounter the error but am getting stuck on

Code: Select all

libtool: compile:  aarch64-unknown-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1 -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/include -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/celt -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/silk -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/silk/float -I/mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/silk/fixed -O2 -pipe -march=armv8-a+crc+fp+simd+crypto -mtune=cortex-a73.cortex-a53 -mcpu=cortex-a73.cortex-a53+crc+fp+simd+crypto -mfix-cortex-a53-835769 -fvisibility=hidden -W -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -c /mnt/ssd/gentoo/portage/media-libs/opus-1.3.1-r4/work/opus-1.3.1/silk/float/LPC_analysis_filter_FLP.c  -fPIC -DPIC -o silk/float/.libs/LPC_analysis_filter_FLP.o

Restoring libtool from stage3 did not change anything with opus, attempting to emerge libtool results in the same Fortran error as before

Code: Select all

checking whether the compiler supports GNU Fortran 77...

Emerging the same version of gcc as was pulled from stage3 with default USE flags gets stuck on the same 'f951' process.

I suppose I'm missing a key package in the toolchain but am not sure which one. I'll continue to pull things from stage3 and test with packages in your /binhost (thanks) until I get a different result.

Post by **NeddySeagoon** » Mon Jan 16, 2023 9:06 pm

atrix,

The configure.log file may be useful when it gets stuck.

Ahh ... check your system time. Some configure scripts check that the files they have just generated are newer than the input files.
If they are not, configure runs again ... and again .... until they are.

On a Pi, the system time starts at 01-Jan-70, so you have over a 53 year wait for that to resolve itself.
Its possible to provoke a similar effect if the system clock was fast, then gets stepped back.

atrix · Post by **atrix** » Mon Jan 16, 2023 9:32 pm

Would the configure.log be in /var/log/portage? Or is this the configure portion of the build log?

I was hoping my time was off after reading your comment, but sadly, no, all looks good there..

I don't know what other packages would come into play here after reviewing https://wiki.gentoo.org/wiki/Project:Toolchain. I've restored everything listed here from stage3. Maybe time to start restoring packages in @system?

Post by **NeddySeagoon** » Mon Jan 16, 2023 9:35 pm

atrix,

It's in the packages /work/ dir somewhere near the top. In /var/tmp/portage/....

atrix · Post by **atrix** » Wed Jan 18, 2023 2:13 am

Thank you. I haven't found anything telling yet, and some packages make it past the configure stage, but I'm continuing to test with different packages until I can make some more sense of what's going on. I'm shooting blindly with quickpkg from stage3 now, no luck yet...

atrix · Post by **atrix** » Wed Jan 18, 2023 3:15 am

Found the issue. You were right, Neddy. Rebuilding gmp with the safe/vanilla CFLAGS you suggested appears to have fixed this. I'm going to upgrade @world overnight and will report back if anything gets stuck, but I haven't been able to reproduce the issue on any of the previously problematic packages after rebuilding gmp. Thanks for your help with this.