Not all binary packages are created with FEATURES=buildpkg

[ville.aakko]

Hi,

I have a binary package host (in a fakeroot) to create packages for a slow machine. For some reason, binary packages are often not created after running emerge.

Some relevant information (from the host):

Code: Select all

# emerge --info
Portage 2.3.103 (python 3.7.8-final-0, default/linux/x86/17.0, gcc-9.3.0, glibc-2.31-r6, 5.7.12-zen1-1-zen i686)
=================================================================
System uname: Linux-5.7.12-zen1-1-zen-i686-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.6
KiB Mem:    32823784 total,   6314916 free
KiB Swap:   33554420 total,  33525748 free
Timestamp of repository gentoo: Thu, 13 Aug 2020 16:30:01 +0000
Head commit of repository gentoo: 31bc42a28f46a27c1d2eaa8fdc91cdb7d516e36c
sh bash 5.0_p17
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
distcc 3.3.3 i586-pc-linux-gnu [disabled]
app-shells/bash:          5.0_p17::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r1::gentoo, 3.7.8-r2::gentoo, 3.8.4-r1::gentoo
dev-util/cmake:           3.16.5::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            9.3.0-r1::gentoo
sys-devel/gcc-config:     2.3::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.31-r6::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://RetroMonster/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24

ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="@FREE"
CBUILD="i586-pc-linux-gnu"
CFLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
CHOST="i586-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=4"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
GENTOO_MIRRORS="rsync://ftp.fi.muni.cz/pub/linux/gentoo/ rsync://mirror.dkm.cz/gentoo/ rsync://mirror.eu.oneandone.net/gentoo/ rsync://mirror.netcologne.de/gentoo/ rsync://ftp.halifax.rwth-aachen.de/gentoo/ rsync://ftp.fau.de/gentoo rsync://ftp-stud.hs-esslingen.de/gentoo/ rsync://mirror.leaseweb.com/gentoo/ rsync://ftp.snt.utwente.nl/gentoo ftp://mirror.mdfnet.se/gentoo http://mirror.mdfnet.se/gentoo rsync://mirror.bytemark.co.uk/gentoo/"
LANG="fi_FI.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="fi en"
MAKEOPTS="-j7"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl alsa bash-completion berkdb bzip2 cddb cdio cli crypt cue curl discid distcc dri elogind flac fortran gdbm gpm iconv ipv6 libglvnd libtirpc mad mikmod modplug ncurses nls nptl openmp pam pcre pulseaudio readline seccomp sid split-usr ssl tcpd timidity unicode vorbis wavpack x86 xattr zlib" ABI_X86="32" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx 3dnow 3dnowext" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="fi en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby25" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

and:

Code: Select all

# cat /etc/portage/make.conf 
# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
# COMMON_FLAGS="-O2 -march=i586 -pipe"
COMMON_FLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult https://wiki.gentoo.org/wiki/Changing_the_CHOST_variable before changing.
CHOST="i586-pc-linux-gnu"
CHOST_x86="i586-pc-linux-gnu"

# NOTE: This stage was built with the bindist Use flag enabled
PORTDIR="/usr/portage"
DISTDIR="/usr/portage/distfiles"
PKGDIR="/usr/portage/packages"

CPU_FLAGS_X86="mmx 3dnow 3dnowext"
USE="bash-completion distcc ipv6 cddb flac mad mikmod modplug vorbis wavpack curl timidity cdio cue sid alsa gpm pulseaudio discid cue elogind -consolekit -systemd -pie"

VIDEO_CARDS="nouveau"

LINGUAS="fi en"
L10N="fi en"


GENTOO_MIRRORS="rsync://ftp.fi.muni.cz/pub/linux/gentoo/ rsync://mirror.dkm.cz/gentoo/ rsync://mirror.eu.oneandone.net/gentoo/ rsync://mirror.netcologne.de/gentoo/ rsync://ftp.halifax.rwth-aachen.de/gentoo/ rsync://ftp.fau.de/gentoo rsync://ftp-stud.hs-esslingen.de/gentoo/ rsync://mirror.leaseweb.com/gentoo/ rsync://ftp.snt.utwente.nl/gentoo ftp://mirror.mdfnet.se/gentoo http://mirror.mdfnet.se/gentoo rsync://mirror.bytemark.co.uk/gentoo/"

FEATURES="buildpkg"
# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C

MAKEOPTS="-j7"

For example after running:

Code: Select all

# emerge -uDNvat @world sshfs

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[nomerge       ] dev-util/debugedit-4.15.1::gentoo 
[ebuild     U  ]  dev-libs/nss-3.55::gentoo [3.52.1-r1::gentoo] USE="-cacert -utils" 79 844 KiB
[nomerge       ] net-fs/nfs-utils-2.4.3::gentoo  USE="ipv6 libmount nfsidmap nfsv4 tcpd uuid -caps -junction -kerberos -ldap -nfsdcld -nfsv41 (-selinux)" 
[ebuild     U  ]  net-libs/rpcsvc-proto-1.4.2::gentoo [1.4.1-r1::gentoo] 61 KiB
[ebuild  N     ] net-fs/sshfs-3.7.0::gentoo  52 KiB
[nomerge       ] media-video/mpv-0.32.0-r1::gentoo  USE="X alsa cli egl iconv libass lua opengl pulseaudio uchardet xv zlib (-aqua) -archive -bluray -cdda (-coreaudio) -cplugins (-cuda) -debug (-doc) -drm -dvb -dvd -gamepad -gbm -jack -javascript -jpeg -lcms -libcaca -libmpv -luajit -openal -oss (-raspberry-pi) -rubberband -samba -sdl (-selinux) -test -tools -vaapi -vdpau (-vulkan) -wayland -zimg" PYTHON_TARGETS="python3_7 -python3_6 -python3_8" 
[ebuild     U  ]  x11-libs/libX11-1.6.11::gentoo [1.6.9::gentoo] USE="ipv6 -doc -static-libs -test" 2 334 KiB
[nomerge       ] net-misc/openssh-8.1_p1-r3::gentoo  USE="pam ssl -X -X509 -audit -bindist -debug -hpn -kerberos -ldns -libedit -libressl -livecd -pie -sctp (-selinux) -static -test -xmss" 
[nomerge       ]  sys-auth/pambase-20200304::gentoo  USE="cracklib elogind nullok sha512 -caps (-consolekit) -debug -minimal -mktemp -pam_krb5 -pam_ssh -passwdqc -securetty (-selinux) -systemd" 
[nomerge       ]   sys-auth/elogind-243.7::gentoo  USE="acl pam policykit -debug -doc (-selinux)" 
[nomerge       ]    sys-auth/polkit-0.116-r1::gentoo  USE="elogind introspection nls pam (-consolekit) -examples -gtk -jit -kde (-selinux) -systemd -test" 
[nomerge       ]     dev-lang/spidermonkey-60.5.2_p0-r4:60::gentoo  USE="jit system-icu -custom-cflags -custom-optimization -debug -minimal -test" 
[ebuild     U  ]      dev-libs/nspr-4.26::gentoo [4.25::gentoo] USE="-debug" 1 055 KiB
[ebuild     U  ] sys-apps/man-pages-5.07::gentoo [5.06::gentoo] L10N="-de -fr -it -ja -nl -pl -ru -zh-CN" 1 677 KiB
[nomerge       ] net-fs/sshfs-3.7.0::gentoo 
[ebuild  NS    ]  sys-fs/fuse-3.9.1:3::gentoo [2.9.9-r1:0::gentoo] USE="suid%* -test% (-examples%) (-static-libs%)" 0 KiB
[nomerge       ] virtual/man-0-r3::gentoo 
[nomerge       ]  sys-apps/man-db-2.8.7::gentoo  USE="berkdb gdbm manpager nls seccomp zlib (-selinux) -static-libs" 
[nomerge       ]   app-text/po4a-0.57::gentoo  USE="-test" 
[nomerge       ]    app-text/opensp-1.5.2-r3::gentoo  USE="nls -doc -static-libs -test" 
[ebuild     U  ]     net-libs/libnsl-1.3.0-r1:0/2::gentoo [1.2.0:0/2::gentoo] 314 KiB
[ebuild     U  ]      net-libs/libtirpc-1.2.6:0/3::gentoo [1.2.5:0/3::gentoo] USE="ipv6 (split-usr) -kerberos -static-libs" 502 KiB
[nomerge       ] sys-devel/gdb-9.1::gentoo  USE="client nls python server -lzma -multitarget -source-highlight -test -vanilla -xml -xxhash" PYTHON_SINGLE_TARGET="python3_7 -python3_6 -python3_8" 
[ebuild     U  ]  dev-libs/mpfr-4.1.0:0/6::gentoo [4.0.2:0/6::gentoo] USE="-static-libs" 1 490 KiB
[nomerge       ] net-fs/sshfs-3.7.0::gentoo 
[nomerge       ]  dev-util/meson-0.54.2::gentoo  USE="(-test)" PYTHON_TARGETS="python3_7 -python3_6 -python3_8" 
[nomerge       ]   dev-python/setuptools-46.4.0-r1::gentoo  USE="-test" PYTHON_TARGETS="python2_7 python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)" 
[nomerge       ]    dev-lang/python-2.7.18-r1:2.7::gentoo  USE="gdbm ipv6 ncurses readline sqlite ssl (threads) (wide-unicode) xml (-berkdb) -bluetooth -build -examples -hardened -libressl -tk -wininst" 
[ebuild     U  ]     dev-libs/libffi-3.3-r2:0/7::gentoo [3.3-r1:0/7::gentoo] USE="-debug -pax_kernel -static-libs -test" 0 KiB
[nomerge       ] virtual/libc-1-r1::gentoo 
[ebuild     U  ]  sys-libs/glibc-2.31-r6:2.2::gentoo [2.30-r8:2.2::gentoo] USE="(crypt) multiarch ssp (static-libs) -audit -caps (-cet) -compile-locales -custom-cflags -doc -gd -headers-only (-multilib) -nscd -profile (-selinux) -static-pie% -suid -systemtap -test (-vanilla)" 16 353 KiB

Total: 12 packages (10 upgrades, 1 new, 1 in new slot), Size of downloads: 103 678 KiB

Would you like to merge these packages? [Yes/No] Yes
>>> Verifying ebuild manifests
>>> Running pre-merge checks for sys-libs/glibc-2.31-r6
 * Checking general environment sanity.
make -j7 glibc-test CC=i586-pc-linux-gnu-gcc 
i586-pc-linux-gnu-gcc -O2 -march=i586 -pipe   -Wl,-O1 -Wl,--as-needed  glibc-test.c   -o glibc-test
 * Checking gcc for __thread support ...                                                                                                                                                                    [ ok ]
 * Checking running kernel version (5.7.12-zen1-1-zen >= 3.2.0) ...                                                                                                                                         [ ok ]
 * Checking linux-headers version (5.4.0 >= 3.2.0) ...                                                                                                                                                      [ ok ]
>>> Emerging (1 of 12) sys-libs/glibc-2.31-r6::gentoo
>>> Installing (1 of 12) sys-libs/glibc-2.31-r6::gentoo
>>> Emerging (2 of 12) dev-libs/libffi-3.3-r2::gentoo
>>> Installing (2 of 12) dev-libs/libffi-3.3-r2::gentoo
>>> Emerging (3 of 12) dev-libs/mpfr-4.1.0::gentoo
>>> Installing (3 of 12) dev-libs/mpfr-4.1.0::gentoo
>>> Emerging (4 of 12) net-libs/libtirpc-1.2.6::gentoo
>>> Installing (4 of 12) net-libs/libtirpc-1.2.6::gentoo
>>> Emerging (5 of 12) sys-fs/fuse-3.9.1::gentoo
>>> Installing (5 of 12) sys-fs/fuse-3.9.1::gentoo
>>> Emerging (6 of 12) net-libs/libnsl-1.3.0-r1::gentoo
>>> Installing (6 of 12) net-libs/libnsl-1.3.0-r1::gentoo
>>> Emerging (7 of 12) dev-libs/nspr-4.26::gentoo
>>> Installing (7 of 12) dev-libs/nspr-4.26::gentoo
>>> Emerging (8 of 12) sys-apps/man-pages-5.07::gentoo
>>> Installing (8 of 12) sys-apps/man-pages-5.07::gentoo
>>> Emerging (9 of 12) x11-libs/libX11-1.6.11::gentoo
>>> Installing (9 of 12) x11-libs/libX11-1.6.11::gentoo
>>> Emerging (10 of 12) net-fs/sshfs-3.7.0::gentoo
>>> Installing (10 of 12) net-fs/sshfs-3.7.0::gentoo
>>> Emerging (11 of 12) net-libs/rpcsvc-proto-1.4.2::gentoo
>>> Recording net-fs/sshfs in "world" favorites file...
>>> Installing (11 of 12) net-libs/rpcsvc-proto-1.4.2::gentoo
>>> Emerging (12 of 12) dev-libs/nss-3.55::gentoo
>>> Installing (12 of 12) dev-libs/nss-3.55::gentoo
>>> Jobs: 12 of 12 complete                         Load avg: 4.26, 5.44, 3.74

 * Messages for package sys-libs/glibc-2.31-r6:

 * Applying Gentoo Glibc Patchset 2.31-8
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * Regenerating GNU info directory index...
 * Processed 109 info files.
 * After world updates, it is important to remove obsolete packages with
 * emerge --depclean. Refer to `man emerge` for more information.

I can see glibc, man-pages, mpfr, libffi have not been created. However, sshfs, nss and a few others have been creted, but less than 50% of the expected packages are now in /usr/portage/packages.

Code: Select all

# for P in glibc man-pages mpfr libffi sshfs nss- ; do find /usr/portage/packages/ -iname "*${P}*" ; done
/usr/portage/packages/sys-libs/glibc-2.30-r8.tbz2
/usr/portage/packages/sys-apps/man-pages-posix-2013a.tbz2
/usr/portage/packages/sys-apps/man-pages-5.06.tbz2
/usr/portage/packages/dev-libs/mpfr-4.0.2.tbz2
/usr/portage/packages/dev-libs/libffi-3.3-r1.tbz2
/usr/portage/packages/virtual/libffi-3.3_rc0.tbz2
/usr/portage/packages/net-fs/sshfs-3.7.0.tbz2
/usr/portage/packages/dev-libs/nss-3.52.1-r1.tbz2
/usr/portage/packages/dev-libs/nss-3.55.tbz2
/usr/portage/packages/sys-auth/nss-mdns-0.14.1.tbz2

Any idea why this happens? I though FEATURES=buildpkg should build all packages, and the Wiki doesn't shed any light on this issue, unless I've missed something there.

I can work around this by running emerge -1b again or with buildpkg (but then I need to deal with configuration files with --include-config or otherwise), however after a while that has become an unnecessary hassle I'd rather not do (and/or find out why portage behaves this way). Actually, that may be the only reason earlier version of said packages are found via the find loop above.

[Ant P.]

The other ones are build deps and aren't needed on a running system.

[ville.aakko]

Thanks for your suggestion but glibc certainly is not (only) a build dependency. Neither are any of the packages I've mentioned individually, they are all required also as run-time dependencies (or at least pulled in on the client).

There might be some pattern relating to what is / is not in worldfile, but I'm not certain what it is (perhaps @system set and it's dependencies is never build into packages, but according to documentation, if that is what is happening, that should be an error?).

[Hu]

You mentioned a fakeroot. Do you mean you are building in a chroot environment? If yes, then the Portage configuration in the chroot is what matters, but you showed us the host configuration.

[Ionen]

It certainly "should" work given I do a similar thing in a chroot and have a glibc, ffi, etc.. binpkgs just fine with FEATURES=buildpkg

Also even build deps would be made into a package, these aren't normally excluded. I do exclude installing them on my target system where I also depclean using --with-bdeps=n to remove everything I can.

[Ant P.]

I misread the output... didn't notice that the binpkgs listed were all old versions. I'm not sure what's happening there.

How old are the files in your pkgdir? There should be an index file at the top, it's not human readable but this should at least make the dates comprehendable:

Code: Select all

awk '{print};/^BUILD_TIME: [0-9]+$/{print strftime("%Y-%m-%d %H:%M:%S",$2)}' $(portageq envvar PKGDIR)/Packages

[Anon-E-moose]

It certainly "should" work given I do a similar thing in a chroot and have a glibc, ffi, etc.. binpkgs just fine with FEATURES=buildpkg

Works for me too (chroot for laptop on desptop) it would and does build everything, including virtuals (which I've blocked from buildpkg as creating a pkg is silly)

You might try eclean-pkg to clear out some of the old pkgs in the package directory.

[Ionen]

including virtuals (which I've blocked from buildpkg as creating a pkg is silly)

There's actually a point to it, like when the target system doesn't use /var/db/repos/gentoo and only relies on the binpkg directory for all dependencies (-K option), in which case it needs to know about virtuals too.

But yeah, if using portage the more common way then virtual packages won't be needed.

Edit: well, more precisely it mostly need the entry for the virtual in the Packages file since that's where the final dependency info is. Then again the package itself does include the ebuild to run postinst functions, and there is uh... one.. virtual that use that.

[ville.aakko]

You mentioned a fakeroot. Do you mean you are building in a chroot environment? If yes, then the Portage configuration in the chroot is what matters, but you showed us the host configuration.

To clarify, the host is the fakeroot machine (the client is not). I do not want to run Gentoo on the host as the main OS (for various reasons, and to use it for other stuff, but that is OT...). EDIT: Also all the information I've given is from the host. The client is not a factor in this problem, for all we know, there might not be one (I might set up one later) as the problem is all produced in the host

I was hoping someone spots something wrong in the "emerge --info" or make.conf... but as that is not the case, and this kind of setup is working for others, the mystery continues for now

Thanks for all suggestions! I will try to 1) clean the packages (eclean-pkg), however I believe it should not be a factor. For some packages, portage/emerge will always either build the package, or not - this seems to be package-specific (name only; i.e. whether a package already exist in pkgdir for an old version, or even the same version with same use flags and other settings, is not a factor to the pattern - AFAICT!). 2) check the date of the packages, in case I can find a pattern that way, as in Ant P.'s suggestion 3) I could also try to to delete the manifest file "Packages" and regenerate it with emaint, as I believe I haven't tried it yet. Perhaps, for some reason, it could have become corrupted somehow?

I will of course report if I find a fix and/or cause. in case there are other ideas, they are welcome!

[NeddySeagoon]

ville.aakko,

As well as setting FEATURES in make.conf, it can be set on a per package basis too.
If you managed to set it on a per package basis with wildcards, you can even make it global.

Does

Code: Select all

FEATURES=buildpkg emerge <list of stuff to emerge>

build packages.

The command line overrides everything but portage won't remember it.

Have you lost/gained a filesystem on /usr/portage/packages ?

[ville.aakko]

Hi NeddySeagoon, thanks for your suggestions!

As well as setting FEATURES in make.conf, it can be set on a per package basis too.

Where can I do this? I'm not sure why I would want to, though. I think you mean /etc/portage/env?

However after reading your comment this I did do this (again, in the host):

Code: Select all

# cd /etc && grep -Hirn buildpkg .
./portage/make.conf:32:FEATURES="buildpkg"
./portage/env/systemset:10:FEATURES="-buildpkg"

I'm not sure if that configuration is a default, or if I've made the changes myself (this is a long-time hobby project so I may have well forgotten some changes I might have made, but there should be little changes to a barebone Gentoo installation, as this is only a binpkg host). That setting might make sense so that merged @system packages are compatible in the host (Intel i7) if case the client is not compatible with it (amd k6-III). Whole contents of the file:

Code: Select all

# cat /etc/portage/env/systemset 
COMMON_FLAGS="-O2 -march=i586 -pipe"
# COMMON_FLAGS="-O2 -march=k6-3 -pipe -fomit-frame-pointer"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"

CPU_FLAGS_X86="mmx" # 3dnow"

FEATURES="-buildpkg"

So I believe for now I will leave it in place, but at upgrade, I will first do emerge -1BuDNvat @system and then @world (on the host). Actually, I think this might well be what I've planned on doing, but forgotten!

I made a quick emerge -eB (more specifically: 'CPU_FLAGS_X86="mmx 3dnow 3dnowext" emerge -e1B @system') and, all binary packages were created for the system set. This does not explain why non-systemset packages are not created by emerge, such as sshfs, nss (as in my OP).

I did grep for buildpkg, buildsyspkg in /etc and examined /usr/portage/profiles/default/linux/x86/17.0 (my currently selected profile) for any settings relating to buildpkg and buildsyspkg. Any ideas, where (besides /etc/portage/env) buildpkg settings could have been set?

If you managed to set it on a per package basis with wildcards, you can even make it global.

I'm a bit lost here. I haven't used any wildcards for buildpkg, nor would I want to (I can't think of any reason I would want to do that in my situation). I specifically want it to act globally, but it does not seem to do that. Well, granted, I had forgotten about /etc/portage/env/systemset, but that setting actually makes sense!

Does

Code: Select all

FEATURES=buildpkg emerge <list of stuff to emerge>

build packages.

I'm well aware you can pass environment this way to any command, however, the problem is exactly that portage does not seem to respect the equivalent setting in make.conf (well, I've now found out why this is done for @system, but still not for non-system).

But I did try this, and this is also a working workaround; also -B (or -b), also seem to work.

So, any ideas why FEATURES=buildpkg does work from command line, but does not work from make.conf?

Have you lost/gained a filesystem on /usr/portage/packages ?

Not sure I'm following what you mean here (maybe you just meant this as a random troubleshooting advice?). I would assume any errors in the filesystem would cause portage to fail when it tries to create the package (not just silently continue)?

As for details about the (file)system: The chroot Gentoo isntallation is resting on an directory in a ext4 filesystem (in Arch Linux). The "/" is the only filesystem the chrooted system sees; I've made no additional mount points there (or symlinks etc.) at any point, nor any changes outside chrooting (after initial installation via stage tarball as per Wiki).

In summary: I've found out why @system set packages were not generated (thanks to NeddySeagoon for getting me to right track!), however, not for some non-@system set packages. For now, at least I know a few workarounds!

As for other suggestions: I've now run eclean-pkg and deleted the manifest and and re-generated with emaint. These had no effect on the issue, it seems.

Thanks, all!

[ville.aakko]

Actually, I think I may have a theory why some packages don't create binary packages. It's specifically caused by the a bit special configuration of the host serving the packages I have.

Possibly those non-@system packages which depend directly on packages in @system will not create binary packages, as the packages they are (directly) depeding on don't have the same settings when doing a merge (because of /etc/portage/env/systemset). Can someone verify if this theory is correct? I could check (manually) if this is the case with the problematic packages (and not the case with those which do get binary packages created).

I think if this is correct, I think I also now know of a working workaround:

Code: Select all

CPU_FLAGS=WHATEVER_CLIENT_NEEDS COMMON_FLAGS=WHATEVER_CLIENT_NEEDS emerge -1BuDN @system
CPU_FLAGS=WHATEVER_CLIENT_NEEDS COMMON_FLAGS=WHATEVER_CLIENT_NEEDS emerge -1BuDN @world
emerge -uDN @world
emerge -uDN [new packages I(/client) may want]

The third command is to actually upgrade the @system at host. I believe this is the only way to get 1) correctly optimized packages for the slow machine (client) and 2) make sure the host (chroot) serving the packages can make binary packages of everything correctly. However, I'm not sure this is always possible, in case build-time dependencies change (for packages in @system or @world).

I believe that portage can not keep track of which binary packages have (not) been created, so in case I first (need to) upgrade the host, there is no direct way of checking if all required binary packages are in place for the client. In case one screws up (or is forced to do upgrades in the "wrong" order, which I suspect is often the case), one can either oneshot missing packages or oneshot @system (or @world).

EDIT: Actually, after further thinking, I think the may still not work. It may be simpler to just make make.conf reflect the host and manually create optimized packages for the client by passing environment. However, I'm not even sure problem is even created as per my theory...