Crossdev for Raspberry Pi 4B

[costel78]

I use a Pi as home server. It is used for local rsync (including portage), web and ksmb server.
For the last year, I left it to compile alone over night, no distcc, no external aid, but recently I have to replicate it for the work and time matters.
So, I fired up crossdev and qemu on main PC and compiled almost all packagers and kernel on x86_64 PC. It worked quite well, except few and dev-perl/* and perl-core/* packages all together. Some of them (perl) does not compiled at all and some generate x86_64 code instead of aarch64.
I will post bellow the setup. Questions are:
1. What I am doing wrong ?
2. What can be improved ?
3. Is there a more elegant way to solve @diff set ?

There is the setup:
I have two make.conf, host for crossdev and qemu for systemd-nspawn
make.conf.host

Code: Select all

CHOST=aarch64-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu
ROOT=/usr/${CHOST}/
ACCEPT_KEYWORDS="${ARCH} ~${ARCH}"
PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"

ACCEPT_LICENSE="*"
ALSA_CARDS="usb-audio"
AUTOCLEAN="yes"
BINPKG_COMPRESS="zstd"
BINPKG_COMPRESS_FLAGS="-T0 -19"
BINPKG_FORMAT="gpkg"
CLEAN_DELAY="3"
CMAKE_MAKEFILE_GENERATOR=ninja
COLLISION_IGNORE="/bin /lib /lib64 /sbin /usr /var"
CPU_FLAGS_ARM="edsp neon thumb vfp vfpv3 vfpv4 vfp-d32 crc32 v4 v5 v6 v7 v8 thumb2"
DISTDIR="/mnt/linux/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=30 --keep-going --with-bdeps=y --complete-graph --quiet-build=n --autounmask-write --exclude app-emulation/qemu"
FEATURES="assume-digests binpkg-multi-instance buildpkg candy cgroup -collision-protect -distcc distlocks -ebuild-locks fail-clean ipc-sandbox merge-sync network-sandbox news nodoc noinfo parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned sandbox sfperms strict unmerge-orphans unknown-features-filter unknown-features-warn -userpriv usersandbox"
FETCHCOMMAND="/usr/bin/wget --timeout=10 -t 2 --passive-ftp --no-check-certificate -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
GENTOO_MIRRORS="https://gentoo.cweb.ro/ http://mirrors.xservers.ro/gentoo/ http://distfiles.gentoo.org/"
I_KNOW_WHAT_I_AM_DOING=1
INPUT_DEVICES="libinput"
INSTALL_MASK="/usr/share/locale -/usr/share/locale/ro -/usr/share/locale/en -/usr/share/locale/en_US"
I_PROMISE_TO_SUPPLY_PATCHES_WITH_BUGS=1
L10N="ro en"
LANG="ro_RO.UTF-8"
LC_ALL="ro_RO.UTF-8"
LC_MESSAGES=C
MAKEOPTS="-j24 --load-average=30"
NGINX_MODULES_HTTP="access auth_basic autoindex browser brotli charset fastcgi geoip2 gunzip gzip limit_conn limit_req memcached realip referer rewrite split_clients ssi sub userid naxsi"
NGINX_MODULES_STREAM="access geoip2 limit_conn realip"
PKGDIR="/mnt/pi/binpkgs/pi/"
PORTAGE_COMPRESS="zstd"
PORTAGE_COMPRESS_FLAGS="-T0 -19"
PORTAGE_NICENESS="19"
PORTAGE_RSYNC_EXTRA_OPTS="--human-readable --delete-before --progress --new-compress"
PORTDIR="/var/db/repos/gentoo"
PYTHON_SINGLE_TARGET="python3_11"
PYTHON_TARGETS="python3_11"
RESUMECOMMAND="/usr/bin/wget -c --timeout=10 -t 2 --passive-ftp --no-check-certificate -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
USE="${ARCH} -acl acpi dbus fam -fortran gpm icu lz4 openmp pgo server systemd usb -xattr zstd ${CPU_FLAGS_ARM}"
VIDEO_CARDS="v3d vc4"
WANT_MP="true"
ax_cv_c_float_words_bigendian=no

FOPTIM="-fomit-frame-pointer -fipa-pta -fno-semantic-interposition -fno-plt"
FGRAPHITE="-fgraphite-identity -floop-nest-optimize"
FLTO="-flto=auto -flto-compression-level=9 -fdevirtualize-at-ltrans"
CFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -O2 -pipe -g0 ${FOPTIM} ${FGRAPHITE} ${FLTO}"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--enable-new-dtags,--as-needed -flto=auto"

QEMU_SOFTMMU_TARGETS="x86_64 aarch64"
QEMU_USER_TARGETS="aarch64"

make.conf.quemu

Code: Select all

CHOST=aarch64-unknown-linux-gnu
ACCEPT_KEYWORDS="${ARCH} ~${ARCH}"

ACCEPT_LICENSE="*"
ALSA_CARDS="usb-audio"
AUTOCLEAN="yes"
BINPKG_COMPRESS="zstd"
BINPKG_COMPRESS_FLAGS="-T0 -19"
BINPKG_FORMAT="gpkg"
CLEAN_DELAY="3"
CMAKE_MAKEFILE_GENERATOR=ninja
COLLISION_IGNORE="/bin /lib /lib64 /sbin /usr /var"
CPU_FLAGS_ARM="edsp neon thumb vfp vfpv3 vfpv4 vfp-d32 crc32 v4 v5 v6 v7 v8 thumb2"
DISTDIR="/mnt/linux/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=30 --keep-going --with-bdeps=y --complete-graph --quiet-build=n --autounmask-write --exclude app-emulation/qemu"
FEATURES="assume-digests binpkg-multi-instance buildpkg candy cgroup -collision-protect -distcc distlocks -ebuild-locks fail-clean -ipc-sandbox merge-sync -network-sandbox news nodoc noinfo parallel-fetch parallel-install -pid-sandbox preserve-libs protect-owned -sandbox sfperms strict unmerge-orphans unknown-features-filter unknown-features-warn -userpriv -usersandbox"
FETCHCOMMAND="/usr/bin/wget --timeout=10 -t 2 --passive-ftp --no-check-certificate -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
GENTOO_MIRRORS="https://gentoo.cweb.ro/ http://mirrors.xservers.ro/gentoo/ http://distfiles.gentoo.org/"
I_KNOW_WHAT_I_AM_DOING=1
INPUT_DEVICES="libinput"
INSTALL_MASK="/usr/share/locale -/usr/share/locale/ro -/usr/share/locale/en -/usr/share/locale/en_US"
I_PROMISE_TO_SUPPLY_PATCHES_WITH_BUGS=1
L10N="ro en"
LANG="ro_RO.UTF-8"
LC_ALL="ro_RO.UTF-8"
LC_MESSAGES=C
MAKEOPTS="-j24 --load-average=30"
NGINX_MODULES_HTTP="access auth_basic autoindex browser brotli charset fastcgi geoip2 gunzip gzip limit_conn limit_req memcached realip referer rewrite split_clients ssi sub userid naxsi"
NGINX_MODULES_STREAM="access geoip2 limit_conn realip"
PKGDIR="/mnt/pi/binpkgs/pi/"
PORTAGE_COMPRESS="zstd"
PORTAGE_COMPRESS_FLAGS="-T0 -19"
PORTAGE_NICENESS="19"
PORTAGE_RSYNC_EXTRA_OPTS="--human-readable --delete-before --progress --new-compress"
PORTDIR="/var/db/repos/gentoo"
PYTHON_SINGLE_TARGET="python3_11"
PYTHON_TARGETS="python3_11"
RESUMECOMMAND="/usr/bin/wget -c --timeout=10 -t 2 --passive-ftp --no-check-certificate -O \"\${DISTDIR}/\${FILE}\" \"\${URI}\""
USE="${ARCH} -acl acpi dbus fam -fortran gpm icu lz4 openmp pgo server systemd usb -xattr zstd ${CPU_FLAGS_ARM}"
VIDEO_CARDS="v3d vc4"
WANT_MP="true"

FOPTIM="-fomit-frame-pointer -fipa-pta -fno-semantic-interposition -fno-plt"
FGRAPHITE="-fgraphite-identity -floop-nest-optimize"
FLTO="-flto=auto -flto-compression-level=9 -fdevirtualize-at-ltrans"
CFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -O2 -pipe -g0 ${FOPTIM} ${FGRAPHITE} ${FLTO}"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--enable-new-dtags,--as-needed -flto=auto"

QEMU_SOFTMMU_TARGETS="x86_64 aarch64"
QEMU_USER_TARGETS="aarch64"

update procedure:

Code: Select all

# PI
dir=`pwd`
cd /usr/aarch64-unknown-linux-gnu/etc/portage ; rm make.conf ; ln -s make.conf.host make.conf
ROOT=/usr/aarch64-unknown-linux-gnu/ aarch64-unknown-linux-gnu-emerge -auvDN world
ROOT=/usr/aarch64-unknown-linux-gnu/ aarch64-unknown-linux-gnu-emerge --deep --depclean -a
cd /usr/aarch64-unknown-linux-gnu/etc/portage ; rm make.conf ; ln -s make.conf.qemu make.conf
systemd-nspawn -D /usr/aarch64-unknown-linux-gnu/ --capability all --bind /tmp/ --bind /var/tmp/ --bind /var/db/repos/ --bind /mnt/linux/distfiles/ --bind /mnt/pi/
cd /usr/aarch64-unknown-linux-gnu/etc/portage ; rm make.conf ; ln -s make.conf.host make.conf
cd "$dir"

Inside of chroot (systemd-nspawn) the commands are:

Code: Select all

env-update && source /etc/profile ; emerge -auvDN world ; emerge --deep --depclean -a ; eclean --deep packages

recompile script:

Code: Select all

#!/bin/bash
ROOT="/usr/aarch64-unknown-linux-gnu/" emaint --fix all
mkdir /tmp/costel &>/dev/null
cd /tmp/costel
rm system_list.txt world_list.txt &>/dev/null
excluded=''
for f in dev-perl/ perl-core/ dev-libs/cyrus-sasl mail-mta/postfix www-client/w3m www-servers/nginx sys-apps/i2c-tools dev-python/RPi-GPIO \
  app-text/po4a app-admin/needrestart \
  sys-apps/portage sys-kernel/linux-headers sys-devel/gcc-config dev-libs/mpfr dev-libs/mpc sys-devel/libtool sys-devel/binutils sys-libs/glibc sys-devel/gcc ; \
  do excluded="${excluded}${f}\|" ; done
export excluded=${excluded::-2}
cd /usr/aarch64-unknown-linux-gnu/var/db/pkg
for p in `for d in * ; do for f in $d/* ; do echo $f ; done ; done | grep -v $excluded` ; do echo -n " ="$p ; done > /tmp/costel/world_list.txt
cd /tmp/costel
ROOT="/usr/aarch64-unknown-linux-gnu/" aarch64-unknown-linux-gnu-emerge -1 -j1 --quiet-build sys-apps/portage sys-kernel/linux-headers sys-devel/gcc-config dev-libs/mpfr dev-libs/mpc sys-devel/libtool sys-devel/binutils sys-libs/glibc sys-devel/gcc
ROOT="/usr/aarch64-unknown-linux-gnu/" aarch64-unknown-linux-gnu-emerge -O1 `cat /tmp/costel/world_list.txt`

dev-perl/* perl-core/* dev-libs/cyrus-sasl mail-mta/postfix www-client/w3m www-servers/nginx app-text/po4a app-admin/needrestart are recompiled inside chroot qemu. It is not fast, but emerge -O1 -j20 helps and they are small packages, so time is 30% compared to pi.
sys-apps/i2c-tools and dev-python/RPi-GPIO are compiled on pi itself. Does not compiled correctly with none of crossdev or qemu.

The diff set:

Code: Select all

acct-group/crontab
# app-alternatives/bc
# app-arch/libarchive
# app-arch/unzip
# app-crypt/rhash
app-eselect/eselect-iptables
# app-eselect/eselect-lib-bin-symlink
app-eselect/eselect-pinentry
app-portage/elt-patches
# app-text/build-docbook-catalog
app-text/docbook-xml-dtd
app-text/docbook-xml-dtd:4.1.2
app-text/docbook-xml-dtd:4.4
app-text/docbook-xsl-ns-stylesheets
# app-text/docbook-xsl-stylesheets
# app-text/opensp
app-text/po4a
# app-text/sgml-common
app-text/xmlto
dev-lang/tcl
# dev-libs/jsoncpp
# dev-perl/Capture-Tiny
dev-perl/Config-AutoConf
dev-perl/ExtUtils-CChecker
dev-perl/File-ShareDir-Install
# dev-perl/File-Slurper
# dev-perl/Locale-gettext
# dev-perl/MIME-Charset
# dev-perl/PerlIO-utf8_strict
# dev-perl/Pod-Parser
# dev-perl/SGMLSpm
# dev-perl/Syntax-Keyword-Try
# dev-perl/Text-CharWidth
# dev-perl/Text-WrapI18N
# dev-perl/Unicode-LineBreak
# dev-perl/XS-Parse-Keyword
# dev-perl/YAML-Tiny
dev-python/cython
dev-python/docutils
dev-python/flit_core
dev-python/gpep517
# dev-python/installer
dev-python/jinja
dev-python/lxml
# dev-python/markupsafe
# dev-python/pygments
dev-python/wheel
dev-util/cmake
dev-util/gdbus-codegen
dev-util/gperf
dev-util/gtk-doc-am
dev-util/meson
dev-util/meson-format-array
dev-util/ninja
dev-util/re2c
# net-libs/libnsl
# net-libs/libtirpc
sys-apps/help2man
sys-apps/texinfo
sys-devel/autoconf-archive
sys-devel/bc
sys-libs/libomp
virtual/perl-CPAN-Meta-Requirements
virtual/perl-File-Path
virtual/perl-IPC-Cmd

Because crossdev is using host cmake, meson etc, those packages are not required by crossdev, only inside chroot, so, to have them, I made a set.

And, finally, emerge --info:

Code: Select all

Portage 3.0.45.3 (python 3.11.2-final-0, default/linux/arm64/23.0/systemd, gcc-13, glibc-2.37-r1, 6.2.9-gentoo-costel x86_64)
=================================================================
System uname: Linux-6.2.9-gentoo-costel-x86_64-AMD_Ryzen_9_7900X_12-Core_Processor-with-glibc2.37
KiB Mem:    31952276 total,  27355464 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Tue, 04 Apr 2023 06:45:01 +0000
Head commit of repository gentoo: 0045be4d9bbac95e1901154f4f26db5b6b9a120c
sh bash 5.2_p15-r2
ld GNU ld (Gentoo 2.40 p4) 2.40.0
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p15-r2::gentoo
dev-lang/perl:             5.36.0-r2::gentoo
dev-lang/python:           3.11.2_p2::gentoo
dev-util/cmake:            3.26.2::gentoo
dev-util/meson:            1.0.1::gentoo
sys-apps/baselayout:       2.13-r1::gentoo
sys-apps/sandbox:          2.30-r1::gentoo
sys-apps/systemd:          253.2::gentoo
sys-devel/autoconf:        2.71-r6::gentoo
sys-devel/automake:        1.16.5-r1::gentoo
sys-devel/binutils:        2.40-r4::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/gcc:             13.0.1_pre20230402::gentoo
sys-devel/gcc-config:      2.10::gentoo
sys-devel/libtool:         2.4.7-r1::gentoo
sys-devel/make:            4.4.1::gentoo
sys-kernel/linux-headers:  6.2::gentoo (virtual/os-headers)
sys-libs/glibc:            2.37-r1::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage/
    priority: -1000
    volatile: False
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: --human-readable --delete-before --progress --new-compress
    sync-rsync-vcs-ignore: true
    sync-rsync-verify-jobs: 1

added
    location: /var/db/repos/added
    masters: gentoo
    priority: 100
    volatile: False

newer
    location: /var/db/repos/newer
    masters: gentoo
    priority: 100
    volatile: False

pi
    location: /var/db/repos/pi
    masters: gentoo
    priority: 100
    volatile: False

Installed sets: @dev, @diff, @minimal, @pi
ACCEPT_KEYWORDS="arm64 ~arm64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -O2 -pipe -g0 -fomit-frame-pointer -fipa-pta -fno-semantic-interposition -fno-plt -fgraphite-identity -floop-nest-optimize -flto=auto -flto-compression-level=9 -fdevirtualize-at-ltrans"
CHOST="aarch64-unknown-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.2/ext-active/ /etc/php/cgi-php8.2/ext-active/ /etc/php/cli-php8.2/ext-active/ /etc/php/fpm-php8.2/ext-active/ /etc/php/phpdbg-php8.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d"
CXXFLAGS="-march=armv8-a+crc -mtune=cortex-a72 -O2 -pipe -g0 -fomit-frame-pointer -fipa-pta -fno-semantic-interposition -fno-plt -fgraphite-identity -floop-nest-optimize -flto=auto -flto-compression-level=9 -fdevirtualize-at-ltrans"
DISTDIR="/mnt/linux/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=30 --keep-going --with-bdeps=y --complete-graph --quiet-build=n --autounmask-write --exclude app-emulation/qemu"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg buildpkg-live candy cgroup config-protect-if-modified distlocks fail-clean fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news nodoc noinfo parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-filter unknown-features-warn unmerge-logs unmerge-orphans userfetch usersandbox usersync"
FFLAGS="-O2"
GENTOO_MIRRORS="https://gentoo.cweb.ro/ http://mirrors.xservers.ro/gentoo/ http://distfiles.gentoo.org/"
INSTALL_MASK="/usr/share/locale -/usr/share/locale/ro -/usr/share/locale/en -/usr/share/locale/en_US"
LANG="ro_RO.UTF-8"
LC_ALL="ro_RO.UTF-8"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--enable-new-dtags,--as-needed -flto=auto"
LEX="flex"
MAKEOPTS="-j24 --load-average=30"
PKGDIR="/mnt/pi/binpkgs/pi/"
PORTAGE_COMPRESS="zstd"
PORTAGE_COMPRESS_FLAGS="-T0 -19"
PORTAGE_CONFIGROOT="/usr/aarch64-unknown-linux-gnu/"
PORTAGE_RSYNC_EXTRA_OPTS="--human-readable --delete-before --progress --new-compress"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="acpi arm64 bzip2 crc32 crypt dbus edsp fam gdbm gpm iconv icu ipv6 libglvnd libtirpc lz4 lzma ncurses neon nls nptl openmp pam pcre pgo readline seccomp server ssl systemd test-rust thumb thumb2 udev unicode usb v4 v5 v6 v7 v8 vfp vfp-d32 vfpv3 vfpv4 zlib zstd" ADA_TARGET="gnat_2021" ALSA_CARDS="usb-audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_ARM="edsp thumb vfp vfpv3 vfpv4 vfp-d32 crc32 v4 v5 v6 v7 v8 thumb2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="ro en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" NGINX_MODULES_HTTP="access auth_basic autoindex browser brotli charset fastcgi geoip2 gunzip gzip limit_conn limit_req memcached realip referer rewrite split_clients ssi sub userid naxsi" NGINX_MODULES_STREAM="access geoip2 limit_conn realip" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" QEMU_SOFTMMU_TARGETS="x86_64 aarch64" QEMU_USER_TARGETS="aarch64" RUBY_TARGETS="ruby30" USERLAND="GNU" VIDEO_CARDS="v3d vc4" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

Please, consider the three questions. If more info is required, just ask.
Thank you!

[NeddySeagoon]

costel78,

1. What I am doing wrong ?

Nothing. Some packages are cross compile hostile. The build systems do not cater for cross compiling.
Portage itself has got a lot better with cross compiling over the last few years.

2. What can be improved ?

File bugs. With patches if that's within your skill set.

3. Is there a more elegant way to solve @diff set ?

Try --with-bdeps=y. Normally build time dependencies for cross compiling are only installed and used on the build host.
I think that --with-bdeps=y builds for the target too. Its not tested.

You may find my Pi4 BINHOST of interest.
Help yourself.

As you have a chroot qemu and a cross compile environment, you can use cross distcc from inside the chroot to the host (outside)
That avoids QEMU emulating gcc. Its described in this somewhat old wiki page Pi3 Build Root
Don't even think about using pump mode. Its no longer supported.

[szatox]

It worked quite well, except few and dev-perl/* and perl-core/* packages all together. Some of them (perl) does not compiled at all and some generate x86_64 code instead of aarch64.
I will post bellow the setup. Questions are:
1. What I am doing wrong ?
2. What can be improved ?
3. Is there a more elegant way to solve @diff set ?

AFAIK perl does not lend itself to cross-compilation easily.
I vaguely recall something about it building code it needs to execute later on during its build process, so it requires the build host and target to be compatible.

You can try working around it by installing qemu-user together with binfmt, which would allow you to transparently run the "wrong" code on an emulator at a huge performance cost.
This might still result in a net gain if the emulated part does not make a big part of the whole job. Whether it makes the build succeed or not depends on implementation details in the build process, so no guarantees there, but I think it's worth giving it a shot.

In the meantime, it's probably a good idea to just launch building those particular things on rpi instead. Also, running emerge on rpi you should still be able to use extra horsepower via crossdev and distcc.

[NeddySeagoon]

szatox,

perl builds mini-perl then uses that to build the real thing.

Its much like gcc doing a bootstrap when the build host and target are the same but skipping the bootstrap otherwise.
Either the perl build system or the ebuild does not do that, so cross compiling fails by design.

[costel78]

Thank you all!, for your time.

@NeddySeagoon
When I asked What I am doing wrong, I referred to setup, scripts, as a whole. I am sorry, not a native English speaker.
I choose to use default/linux/arm64/23.0/systemd profile because it seems that fewer packages fail with cross compile.
Will file a bug regarding dev-perl and perl-core packages using crossdev.
Also, will use distcc between host and qemu - great ideea . Yes, pump mode didn't work well, even when it was supported, if I recall correctly.
--with-bdeps=y make no difference. It is already in EMERGE_DEFAULT_OPTS.

@szatox
dev-lang/perl, itself, it is compiling fine, but without lto, using crossdev. dev-perl and perl-core packages are problematic. Already using qemu to compile them.
Compiling them in parallel on qemu is faster than installing resulted binary packages on Pi .
The main reason I avoid to compile on pi itself is because the only one available to development is my personal Pi, the others (nine pieces) are required on sites, as production systems. Availability is still a problem with those.

Recompiling the whole system (lto and graphite flags were added gradually) and kernel is done in under one hour. gcc itself take 5 hours on pi. Installing them (binary packages) on Pi, takes about 90 minutes, 30 minutes more. I am using an ssd with Argon One M.2 case. Also, on sites, ssd will be used, but with different, cheaper, cases.

[NeddySeagoon]

costel78,

My answer is not changed. Your setup is much the same as mine before I got an arm64 server.

For the bdeps and cross compiling, make a set with the missing packages, say at /usr/<tuple>/etc/portage/sets/bdeps
Unfortunately, you need to maintain it by hand.
Its a list of bdeps that you want to be installed into the target root.

Build the contents of the set using cross emerge @bdeps ... much like @world. It will be included in your @world set unless you use --oneshot.

You will find more expertise on Libera IRC in #gentoo-arm
The channel requires you to be registered to speak.

Someone there may know how to make portage spit out a list of built time dependencies.

[costel78]

I filled a bug and keep hoping regarding dev-perl/* packages.
Bug #903769

Keeping @diff set updated is not hard. You are right, maybe bdeps name is more apropiate.
I have two variants:

1. On Pi, update script is:

Code: Select all

#!/bin/bash
emerge -auvDN -kG world
emerge -auvDN world
emerge -av @preserved-rebuild
etc-update
emerge --deep --depclean -a
echo "Repornesc squashmount"
systemctl daemon-reload
systemctl restart squashmount
/root/tools/cron.sh
emaint --fix all

If something was missed on emerge -auvDN -kG world it will show up on emerge -auvDN world and I will know where to put that package(s).

2. Start with a empty Pi binpkgs folder and run recompile script. After that run

Code: Select all

ROOT=/usr/aarch64-unknown-linux-gnu/ aarch64-unknown-linux-gnu-emerge -pve world | grep ebuild

This will include dev-libs/cyrus-sasl mail-mta/postfix www-client/w3m www-servers/nginx sys-apps/i2c-tools dev-python/RPi-GPIO app-text/po4a app-admin/needrestart, but this is not a problem.

Thank you!

[ShorTie]

Can I make a sugguestion ??
To use the proper perl script "perl-cleaner --all"
Maybe

[costel78]

Thank you for your suggestion, but, here, does not apply.
perl-cleaner is recommended to be used when dev-lang/perl is updated.
The worst thing here with dev-perl and perl-core packages is that many of them are compiling fine, but for host arch (x86-64) not for Pi - arm64.

[ShorTie]

That sounds more like a cross compile issue.
Or your not chrooting in properly.
'perl-cleaner is recommended', does NOT mean it can not be used.
It is a great tool for checking/fixing perl issues.
I always run it on a fresh install to make sure portage works right.
But that's me, lol.

[costel78]

It is definitely a cross compile issue.
I run perl-cleaner during initial phase, because the crossdev is on ~arm64, so perl was upgraded.
The thing is that I was able to fully run it only on qemu chroot. It wasn't worked to the end using aarch64 emerge wrapper, due to compilations issues.