Can't emerge binutils (OOM issue)

Message

hanj · Post by **hanj** » Tue Sep 22, 2020 2:16 pm

So, I've been trying to emerge sys-devel/binutils-2.34-r2 over the last couple of days, and running into problems. Basically out-of-memory. I added an extra 2G of swap. The server is a small instance with only 500M of RAM. But watching memory, it never gets to swap, just runs out of memory and dies. Here is the weird thing, I recently created this image from another server. That other server was able to compile binutils-2.34-r2 without a problem. Same RAM. Same swap. Same.. everything.

Here are USE flags for binutils:

Code: Select all

[ebuild  NS    ] sys-devel/binutils-2.34-r2:2.34::gentoo [2.32-r1:2.32::gentoo, 2.33.1-r1:2.33::gentoo] USE="gold plugins -default-gold -doc -multitarget -nls -static-libs -test" 0 KiB

Here is the emerge output:

Code: Select all

x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -I. -I/var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34/gold  -I/var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34/gold -I/var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34/gold/../include -I/var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34/gold/../elfcpp -DLOCALEDIR="\"/usr/share/binutils-data/x86_64-pc-linux-gnu/2.34/locale\"" -DBINDIR="\"/usr/x86_64-pc-linux-gnu/binutils-bin/2.34\"" -DTOOLBINDIR="\"/usr/x86_64-pc-linux-gnu/bin\"" -DTOOLLIBDIR="\"/usr/x86_64-pc-linux-gnu/lib\""   -W -Wall    -Wstack-usage=262144 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -frandom-seed=incremental.o  -pthread -O2 -MT incremental.o -MD -MP -MF .deps/incremental.Tpo -c -o incremental.o /var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34/gold/incremental.cc
x86_64-pc-linux-gnu-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[4]: *** [Makefile:1133: incremental.o] Error 1
make[4]: Leaving directory '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/build/gold'
make[3]: *** [Makefile:1156: all-recursive] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/build/gold'
make[2]: *** [Makefile:896: all] Error 2
make[2]: Leaving directory '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/build/gold'
make[1]: *** [Makefile:6101: all-gold] Error 2
make[1]: Leaving directory '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/build'
make: *** [Makefile:853: all] Error 2
 * ERROR: sys-devel/binutils-2.34-r2::gentoo failed (compile phase):
 *   emake failed
 *
 * If you need support, post the output of `emerge --info '=sys-devel/binutils-2.34-r2::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=sys-devel/binutils-2.34-r2::gentoo'`.
 * The complete build log is located at '/var/log/builds/sys-devel:binutils-2.34-r2:20200922-135345.log'.
 * For convenience, a symlink to the build log is located at '/var/tmp/portage/sys-devel/binutils-2.34-r2/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sys-devel/binutils-2.34-r2/temp/environment'.
 * Working directory: '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/build'
 * S: '/var/tmp/portage/sys-devel/binutils-2.34-r2/work/binutils-2.34'
alarm signal

>>> Failed to emerge sys-devel/binutils-2.34-r2, Log file:

>>>  '/var/log/builds/sys-devel:binutils-2.34-r2:20200922-135345.log'

Here is the OOM message in logs:

Code: Select all

Sep 22 08:01:16 server.comp.com kernel: [298912.203327] [  21125]    40 21125    50197      781   135168     4168             0 named
Sep 22 08:01:16 server.comp.com kernel: [298912.203329] [  31475]     0 31475    54691       89    65536       74             0 bash
Sep 22 08:01:16 server.comp.com kernel: [298912.203330] [   5064]     0  5064    54665      158    73728        0             0 bash
Sep 22 08:01:16 server.comp.com kernel: [298912.203332] [  10690]     0 10690    58089     2709   106496        0             0 sec
Sep 22 08:01:16 server.comp.com kernel: [298912.203334] [  19814]   207 19814    13008      235    86016        0             0 pickup
Sep 22 08:01:16 server.comp.com kernel: [298912.203335] [  20816]     0 20816    76743    20227   245760        0             0 emerge
Sep 22 08:01:16 server.comp.com kernel: [298912.203337] [  23302]     0 23302    56376     1239    77824        0             0 python3.6m
Sep 22 08:01:16 server.comp.com kernel: [298912.203339] [  23304]     0 23304    56376     1248    73728        0             0 python3.6m
Sep 22 08:01:16 server.comp.com kernel: [298912.203340] [  23307]   250 23307      577       40    40960        0             0 sandbox
Sep 22 08:01:16 server.comp.com kernel: [298912.203342] [  23308]   250 23308    55507      913    73728        0             0 ebuild.sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203343] [  23324]   250 23324    55539      924    61440        0             0 ebuild.sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203345] [  23327]   250 23327    54861      278    61440        0             0 emake
Sep 22 08:01:16 server.comp.com kernel: [298912.203346] [  23329]   250 23329    54743      449    69632        0             0 make
Sep 22 08:01:16 server.comp.com kernel: [298912.203348] [  23331]   250 23331    54763      171    65536        0             0 sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203349] [  23343]   250 23343    54779      472    73728        0             0 make
Sep 22 08:01:16 server.comp.com kernel: [298912.203351] [  17020]   250 17020    54798      191    69632        0             0 sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203352] [  17026]   250 17026    54798      203    57344        0             0 sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203354] [  17036]   250 17036    54504      238    69632        0             0 make
Sep 22 08:01:16 server.comp.com kernel: [298912.203355] [  17039]   250 17039    54534      234    65536        0             0 make
Sep 22 08:01:16 server.comp.com kernel: [298912.203357] [  17040]   250 17040    54765      177    61440        0             0 sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203359] [  17058]   250 17058    54616      349    73728        0             0 make
Sep 22 08:01:16 server.comp.com kernel: [298912.203360] [  17200]   250 17200     1348      110    49152        0             0 x86_64-pc-linux
Sep 22 08:01:16 server.comp.com kernel: [298912.203362] [  17201]   250 17201    89842    74188   761856        0             0 cc1plus
Sep 22 08:01:16 server.comp.com kernel: [298912.203364] [  17205]     0 17205    54446       37    49152       54             0 crond
Sep 22 08:01:16 server.comp.com kernel: [298912.203365] [  17206]     0 17206      566       16    40960        0             0 run-parts
Sep 22 08:01:16 server.comp.com kernel: [298912.203367] [  17209]     0 17209      994       79    45056        0             0 xe-update-guest
Sep 22 08:01:16 server.comp.com kernel: [298912.203368] [  17210]     0 17210    54585       66    65536        0             0 logcheck.cron
Sep 22 08:01:16 server.comp.com kernel: [298912.203370] [  17211]     0 17211    54585       61    61440        0             0 sh
Sep 22 08:01:16 server.comp.com kernel: [298912.203371] [  17215]     0 17215      994       79    45056        0             0 xe-update-guest
Sep 22 08:01:16 server.comp.com kernel: [298912.203373] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,task=cc1plus,pid=17201,uid=250
Sep 22 08:01:16 server.comp.com kernel: [298912.203382] Out of memory: Killed process 17201 (cc1plus) total-vm:359368kB, anon-rss:296752kB, file-rss:0kB, shmem-rss:0kB, UID:250 pgtables:744kB oom_score_adj:0

Here is meminfo

Code: Select all

cat /proc/meminfo
MemTotal:         490432 kB
MemFree:          346092 kB
MemAvailable:     416536 kB
Buffers:           13512 kB
Cached:            52308 kB
SwapCached:         1540 kB
Active:            38964 kB
Inactive:          53760 kB
Active(anon):       7212 kB
Inactive(anon):    19788 kB
Active(file):      31752 kB
Inactive(file):    33972 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1999992 kB
SwapFree:        1970808 kB
Dirty:                72 kB
Writeback:             0 kB
AnonPages:         26260 kB
Mapped:            17012 kB
Shmem:                92 kB
KReclaimable:      17592 kB
Slab:              36192 kB
SReclaimable:      17592 kB
SUnreclaim:        18600 kB
KernelStack:        1536 kB
PageTables:         1560 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2245208 kB
Committed_AS:     140212 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        3020 kB
VmallocChunk:          0 kB
Percpu:             4032 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
DirectMap4k:      251904 kB
DirectMap2M:      268288 kB

Here is my make.conf. I removed -pipes, I changed MAKEOPTS to -j1 and tried CFLAGS set to -O0.

Code: Select all

CFLAGS="-O0"
CXXFLAGS="${CFLAGS}"
CHOST="x86_64-pc-linux-gnu"
MAKEOPTS="-j1"
PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages"
PORTDIR_OVERLAY="/usr/local/portage"
PORTAGE_ELOG_MAILFROM="groupsec@comp.com"
PORTAGE_ELOG_CLASSES="warn error log"
PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_MAILURI="groupsec@comp.com localhost"
PORT_LOGDIR="/var/log/builds"
LINGUAS="en"
PYTHON_TARGETS="python2_7 python3_4 python3_6 python3_7"
PYTHON_SINGLE_TARGET="python3_7"
USE_PYTHON="2.7 3.4 3.6 3.7"
ACCEPT_LICENSE="*"

Here are available gcc on the system. I tried all 4

Code: Select all

gcc-config -l
 [1] x86_64-pc-linux-gnu-8.2.0
 [2] x86_64-pc-linux-gnu-8.3.0
 [3] x86_64-pc-linux-gnu-9.2.0
 [4] x86_64-pc-linux-gnu-9.3.0 *

Here is my emerge --info output

Code: Select all

 emerge --info
Portage 3.0.4 (python 3.6.11-final-0, default/linux/amd64/17.0, gcc-9.3.0, glibc-2.31-r6, 5.4.60-gentoo x86_64)
=================================================================
System uname: Linux-5.4.60-gentoo-x86_64-AMD_Opteron-tm-_Processor_4332_HE-with-gentoo-2.7
KiB Mem:      490432 total,    293692 free
KiB Swap:    1999992 total,   1970808 free
Timestamp of repository gentoo: Tue, 22 Sep 2020 10:00:01 +0000
Head commit of repository gentoo: 742562f7fac885f1f4a78e438def443e8f655350
sh bash 5.0_p18
ld GNU ld (Gentoo 2.32 p2) 2.32.0
app-shells/bash:          5.0_p18::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r2::gentoo, 3.6.11-r2::gentoo, 3.7.8-r2::gentoo, 3.8.5::gentoo
dev-util/cmake:           3.16.5::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.69-r5::gentoo
sys-devel/automake:       1.15.1-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo, 2.33.1-r1::gentoo
sys-devel/gcc:            8.2.0-r6::gentoo, 8.3.0-r3::gentoo, 9.2.0-r2::gentoo, 9.3.0-r1::gentoo
sys-devel/gcc-config:     2.3.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.31-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts:
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O0"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/stunnel/stunnel.conf /usr/share/gnupg/qualified.txt /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O0"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.rackspace.com/gentoo/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en"
MAKEOPTS="-j1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="amd64 berkdb bzip2 cli crypt dri fortran gdbm iconv innodb libglvnd libtirpc maildir multilib mysql mysqli ncurses nptl nptlonly openmp openssh pam pcre php pwdb readline sasl seccomp snortsam split-usr ssl tcpd unicode vhosts xattr zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_4 python3_6 python3_7" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

Any ideas?

hanji

eccerr0r · Post by **eccerr0r** » Tue Sep 22, 2020 3:25 pm

512MB for x86_64 is really not enough memory.

I don't know what else you're running but I've found that even 2GB is not really enough to do a self hosting GUI Gentoo install. You don't have a GUI so you can get by with less, but with compilers growing in bloat, I don't know how long it will last.

For x86 (32 bit) it may be sufficient.

mike155 · Post by **mike155** » Tue Sep 22, 2020 3:39 pm

eccerr0r wrote:512MB for x86_64 is really not enough memory.

Really? Below is what our handbook says:

Code: Select all

AMD64 hardware requirements    Minimal CD    LiveDVD
Memory                         256 MB        512 MB

hanj · Post by **hanj** » Tue Sep 22, 2020 4:05 pm

eccerr0r wrote:512MB for x86_64 is really not enough memory.

I don't know what else you're running but I've found that even 2GB is not really enough to do a self hosting GUI Gentoo install. You don't have a GUI so you can get by with less, but with compilers growing in bloat, I don't know how long it will last.

For x86 (32 bit) it may be sufficient.

Also.. the issue of my other server (which this was imaged from) has the same swap, same RAM and was able to compile binutils?

hanji

eccerr0r · Post by **eccerr0r** » Wed Sep 23, 2020 1:23 pm

mike155 wrote:
eccerr0r wrote:512MB for x86_64 is really not enough memory.
Really? Below is what our handbook says:
Code: Select all
AMD64 hardware requirements    Minimal CD    LiveDVD
Memory                         256 MB        512 MB

Did you actually try it with this little RAM?

Portage itself uses so much RAM that it needs to completely swap out when you have so little RAM

If you do not plan to run "emerge" then maybe you can get away with less.

eccerr0r · Post by **eccerr0r** » Wed Sep 23, 2020 1:25 pm

hanj wrote: Also.. the issue of my other server (which this was imaged from) has the same swap, same RAM and was able to compile binutils?

hanji

I don't know what you did with the images, without knowing that it's really tough to know what's going on. Something is locking memory and prohibiting swapping. Did you perhaps change your swappiness perhaps to something really low, and you really need it default (60) or even higher?

From your report, the emerge process is currently using 80MB RAM which is a good fraction of your available memory. Gcc is using 300MB at time of death and was the killed program... Neither swapped.

Phoenix591 · Post by **Phoenix591** » Wed Sep 23, 2020 11:34 pm

I have ran a VM with just 512mb of ram for a while.. one thing I did is set CFLAGS to -Os (and still used -march=native), really need to make everything as absolutely small as possible.

eccerr0r · Post by **eccerr0r** » Thu Sep 24, 2020 12:00 am

That's the thing, I am hoping nobody is referencing "back then I was running..." -- Need to compare x86_64 machines here.

Heck I still have a x86 (32-bit) machine running Gentoo Linux with 64MB RAM. The machine is completely outdated and can't run a GUI, but it can run 'emerge' - though probably not anymore with today's tools. Such is the price of "progress" or is it "regress"?

Phoenix591 · Post by **Phoenix591** » Thu Sep 24, 2020 9:56 pm

No I don't mean back years ago, I mean a few weeks ago I had a perfectly happy ~amd64 server with 512Mb of ram that (slowly) built its own packages. Ran nginx and bind on it as well as a vpn, used only around 256Mb of ram idle. Rebuilding gcc pushes it to the limit, but most of the time it eventually finished, I modified the toolchain eclass to force it to use -Os just to get things that much smaller.

ct85711 · Post by **ct85711** » Thu Sep 24, 2020 10:25 pm

Though the fun part is, it seems more packages are switching and using multiple different compilers which do not necessarily follow gcc flags. For example, clang/llvm use their own compile flags independent of gcc, then there's rust doing their own thing. Both seem to become more commons a dependency because of either laziness to for them being "new/better". How well, those may deal with limited memory conditions, can very well be a issue.

Post by **Ionen** » Thu Sep 24, 2020 11:01 pm

If you can should use another machine to compile, that's what I do for my low'ish ram server, and it only installs binpkgs without ever compiling anything (with binpkgs don't have to worry about packages playing well with distcc too.. like I do need rust, also don't need to install build deps).

Edit: This amd64 server running nginx, inspircd, few other minor services, and some tmux+ncurses applications (one of which was compiled by rust) is using 74MB ram right now

Although it is kinda idle, using ssh+git to access the ::gentoo mirror on it (that other machines sync with) already sometime adds ~177MB for an instant