[SOLVED] Questions about maintaining binpkg chroot

[irets]

Hi,

I have a chroot for making binpkgs for my laptop which is a Lenovo T400 with Intel Core2 Duo processor.
The chroot was setup in this old thread -> viewtopic-t-1159789-start-0-postdays-0- ... ight-.html
My current host computer has an i7 4790k processor.

After running emaint sync --allrepos on my host computer, can I somehow make this up-to-date repository available to my chroot? So I wouldn't have to run that same command inside the chroot again to update both of my Gentoo installations.
Also, what's the recommended way to update a system which is dependent on this binpkg chroot? emerge -eav @world seems a bit wasteful as it creates a lot of read/write events on my laptop.

My second question is about the support of cross-compiling packages for my Core2 Duo on more modern processors.
I've been thinking about upgrading my aging parts and migrate my current system to new hardware - along with my binpkg chroot.
Will I have compability issues if I decide to buy a modern AMD or Intel? Can AMD produce code which Core2 duo can execute? What should I keep in mind when looking for a new processor?
Do I have to buy Intel in order to continue cross-compiling for my Intel laptop?

Cheers.

EDIT: Here's the make.conf of my binpkg chroot.

Code: Select all

# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
CHOST="x86_64-pc-linux-gnu" 
COMMON_FLAGS="-march=core2 -O2 -pipe"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
INSTALL_MASK="/lib/systemd/*/*.service /usr/lib/systemd/*/*.service"
FEATURES="buildpkg"
GRUB_PLATFORMS="pc"
VIDEO_CARDS="intel"
INPUT_DEVICES="libinput"
USE="X elogind pulseaudio -gnome -systemd"

CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 sse4_1 ssse3"

# NOTE: This stage was built with the bindist Use flag enabled

# This sets the language of build output to English.
# Please keep this setting intact when reporting bugs.
LC_MESSAGES=C

GENTOO_MIRRORS="ftp://mirror.netcologne.de/gentoo/ https://mirror.netcologne.de/gentoo/ http://mirror.netcologne.de/gentoo/ rsync://mirror.netcologne.de/gentoo/"

[pjp]

After running emaint sync --allrepos on my host computer, can I somehow make this up-to-date repository available to my chroot? So I wouldn't have to run that same command inside the chroot again to update both of my Gentoo installations.
Also, what's the recommended way to update a system which is dependent on this binpkg chroot? emerge -eav @world seems a bit wasteful as it creates a lot of read/write events on my laptop.

I have a T470. I don't use any hardware specific optimizations (my intent is to use common binaries for all my systems).

On my host, I sync portage and create a squashfs image of the repository. I then distribute that squashfs image to my chroot and when I update the laptop, to it as well. I have scripts to reduce the manual work, but I still do quite a bit of manual work. Well, more than I'd like. Others use NFS to distribute files. Using NFS probably makes more sense, but I have a strong dislike of NFS.

On the laptop, I update the squashfs repo (I now have 2 others besides ::gentoo, so that's still manual as I've only recently started using them). I then update /etc/portage from a tarball of the chroot.

I then fetch the binaries (distributed via http) using:

Code: Select all

emerge -auDgKf --with-bdeps=y --backtrack=42 --verbose-conflicts @world

That ensures I should have everything local to the laptop.

To install, using ONLY binary packages (same command, drop the 'f'):

Code: Select all

emerge -auDgK --with-bdeps=y --backtrack=42 --verbose-conflicts @world

The downside to this is that it will install "binaries" of virtuals and the new "alternatives" and any similar packages if there are others. I recently tried to allow those uncompiled packages to not require binaries, but couldn't get it to work.

If nothing else, that's one way you may not want to do it ;)

[NeddySeagoon]

Irets,

This comes up every now and again. As a result I wrote Build In A Chroot

It could do with some polish before being moved to the main Wiki.
Feedback here or on the Talk page would be good.

[irets]

Thank you for the replies.

The wiki article quickly covers that the host must CPU must not be a subset of the target CPU.
How can I know I get a compatible CPU in order to continue cross-compiling for Core2 Duo when buying a new host CPU?

[Genone]

Any CPU can generate code for any other CPU as long as the relevant software (like compiler) is available for it, after all it's just bits and bytes.
It just becomes (much) more complicated to setup when both architectures differ too much and the host CPU can't run binaries compiled for the target CPU.

With x86, newer CPUs are generally fully compatible with older models of the same product family. Only when mixing product families (like Core and Celeron) there can be issues with missing instructions sets (like AVX2). As long as you avoid Celeron and Pentium models you should be fine with basically any Intel or AMD CPU from the last 5 years.

[NeddySeagoon]

Irets,

The wiki page says that emitting incompatible instructions is OK.
Attempting to execute them is not. You will get an illegal instruction exception and the process will be killed.

Then you have to fix it and use package.env/ in the chroot to set CFLAGS/CXXFLAGS on a per package basis.
To be clear, this is only a (potential) problem on the build box, not on the target.

In practice, if the chroot gcc breaks and you need to do a suboptimal build to get it to work on the build host, it won't matter much as you don't intend to run it on the target if you can avoid it anyway.

[irets]

Thank you Neddy & Genone - I'll start looking for a "modern" processor & hardware and recompile both my host system and chroot once I get all my parts together.
I'll reference old forum threads when the time comes.

One last question popped up while I was maintaining my chroot; how do I properly unmount the chroot filesystem once I'm done with it?

I have the following script to enter the chroot:

Code: Select all

#!/bin/bash

chroot=/mnt/t400chroot

mount --rbind /dev "$chroot"/dev
mount --make-rslave "$chroot"/dev
mount -t proc /proc "$chroot"/proc
mount --rbind /sys "$chroot"/sys
mount --make-rslave "$chroot"/sys
mount --rbind /tmp "$chroot"/tmp
mount --bind /run "$chroot"/run

cp -v /etc/resolv.conf "$chroot"/etc
chroot "$chroot" /bin/bash

Trying to umount the chrooted filesystem(s) results in "target is busy" errors:

Code: Select all

navi /mnt # umount /mnt/t400chroot/*
umount: /mnt/t400chroot/bin: not mounted.
umount: /mnt/t400chroot/boot: not mounted.
umount: /mnt/t400chroot/dev: target is busy.
umount: /mnt/t400chroot/etc: not mounted.
umount: /mnt/t400chroot/home: not mounted.
umount: /mnt/t400chroot/lib: not mounted.
umount: /mnt/t400chroot/lib64: not mounted.
umount: /mnt/t400chroot/media: not mounted.
umount: /mnt/t400chroot/mnt: not mounted.
umount: /mnt/t400chroot/opt: not mounted.
umount: /mnt/t400chroot/root: not mounted.
umount: /mnt/t400chroot/sbin: not mounted.
umount: /mnt/t400chroot/stage3-amd64-openrc-20221211T170150Z.tar.xz: not mounted.
umount: /mnt/t400chroot/startchroot.sh: not mounted.
umount: /mnt/t400chroot/sys: target is busy.
umount: /mnt/t400chroot/usr: not mounted.
umount: /mnt/t400chroot/var: not mounted.

These paths are visible in output of mount:

Code: Select all

navi /home/irets # mount | grep t400
udev on /mnt/t400chroot/dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=2040174,mode=755)
devpts on /mnt/t400chroot/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /mnt/t400chroot/dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /mnt/t400chroot/dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /mnt/t400chroot/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /mnt/t400chroot/sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /mnt/t400chroot/sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /mnt/t400chroot/sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /mnt/t400chroot/sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /mnt/t400chroot/sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /mnt/t400chroot/sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /mnt/t400chroot/sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /mnt/t400chroot/sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /mnt/t400chroot/sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /mnt/t400chroot/sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /mnt/t400chroot/sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /mnt/t400chroot/sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /mnt/t400chroot/sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /mnt/t400chroot/sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /mnt/t400chroot/sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /mnt/t400chroot/sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /mnt/t400chroot/sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /mnt/t400chroot/sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
udev on /mnt/t400chroot/dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=2040174,mode=755)
devpts on /mnt/t400chroot/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /mnt/t400chroot/dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /mnt/t400chroot/dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /mnt/t400chroot/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /mnt/t400chroot/sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /mnt/t400chroot/sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /mnt/t400chroot/sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /mnt/t400chroot/sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /mnt/t400chroot/sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /mnt/t400chroot/sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /mnt/t400chroot/sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /mnt/t400chroot/sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /mnt/t400chroot/sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /mnt/t400chroot/sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /mnt/t400chroot/sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /mnt/t400chroot/sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /mnt/t400chroot/sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /mnt/t400chroot/sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /mnt/t400chroot/sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /mnt/t400chroot/sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /mnt/t400chroot/sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /mnt/t400chroot/sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
udev on /mnt/t400chroot/dev type devtmpfs (rw,nosuid,noexec,relatime,size=10240k,nr_inodes=2040174,mode=755)
devpts on /mnt/t400chroot/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /mnt/t400chroot/dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /mnt/t400chroot/dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /mnt/t400chroot/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /mnt/t400chroot/sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /mnt/t400chroot/sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
pstore on /mnt/t400chroot/sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /mnt/t400chroot/sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /mnt/t400chroot/sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /mnt/t400chroot/sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cpuset on /mnt/t400chroot/sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /mnt/t400chroot/sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /mnt/t400chroot/sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /mnt/t400chroot/sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /mnt/t400chroot/sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /mnt/t400chroot/sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /mnt/t400chroot/sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
net_cls on /mnt/t400chroot/sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
perf_event on /mnt/t400chroot/sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
net_prio on /mnt/t400chroot/sys/fs/cgroup/net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio)
hugetlb on /mnt/t400chroot/sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
pids on /mnt/t400chroot/sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)

[Hu]

The simple approach is to run the chroot in a separate mount namespace, so that when the last process exits, the mount namespace is destroyed and all its mounts are cleaned up automatically. Use unshare -m -- startchroot.sh instead of a bare startchroot.sh if you want to do this. You could arrange for the script to unshare itself, but that is slightly tricky since you need unshare to run a process with the new namespace, and that new process needs to run the mount --bind commands and then exec the chroot command.

[Genone]

Looks like the error only affects mount points that were marked as "rslave" before. Not very familiar with that option, but from what I understand (and can be seen in the mount output) there are mountpoints located within /dev and /sys which need to be unmounted first before you can unmount /dev and /sys themselves (but that's really just an educated guess).

[Hu]

mount --rbind recursively binds all mounts from beneath the specified source to the corresponding place beneath the specified destination. I think that the use of --make-rslave on the bad mounts is coincidental. /tmp was not affected because OP happens not to have any mounts beneath /tmp on the host, so for that path, --rbind and --bind produce the same result. I agree that the mounts beneath /dev and /sys are the problem, and they are only there to cause a problem because of the use of --rbind for /dev and /sys. Using a mount namespace as I described above would avoid the need to explicitly unmount anything, and thus sidestep the problem of manually unmounting in the right order.

[irets]

unshare -m -- ./startchroot.sh works well!
It'd be nice to include this in the script but I don't know how.

A dirty way could be to call ./startchroot from another script which includes unshare -m --.

I'll look into this.

Thank you, marking as solved.

[Hu]

The system call unshare only affects the caller, and processes it subsequently forks off. The program unshare therefore needs to run a child process in the new namespace. Since bash does not (as far as I know) have an intrinsic to call the system call unshare, you cannot avoid executing /usr/bin/unshare and letting it execute a shell with the target script. You can, if you want to be tricky, wrap this all in one script. For example (from memory, untested):

Code: Select all

#!/bin/bash

if ! [[ -v running_in_private_namespace ]]; then
    running_in_private_namespace=1 exec /usr/bin/unshare -m -- /bin/bash "$0" "$@"
fi

# Remainder of script, as normal

This tests for an environment variable and, if absent, delegates to unshare to re-run this with the variable set. This has the minor risk that if your guard variable happens to be set in the interactive shell, this script will get confused and not re-exec when run from the interactive shell, even when it should. Therefore, you should pick a guard variable that the interactive shell will never have set.

[alexander-n8hgeg5e]

Irets,

This comes up every now and again. As a result I wrote Build In A Chroot

It could do with some polish before being moved to the main Wiki.
Feedback here or on the Talk page would be good.

I wonder if my problem is also something others encountered.
Point 8 is why the chroot way doesn't work for me.
I have the problem that I want to build with a compiler
compiled itself with native optimizations for another machine that is also x86_64
but with different cpu features, some that are not available on the build machine.
I know about these 3 ways :

1. crosscompiling with emerge-x86_64-turtoise-linux-gnu (made by crossdev)
2. emerge --root=/usr/turtoise/ --sysroot=/usr/turtoise
3. chroot as described in the wikipage

The problems I had:
with 1. : crossdev refused to build a cross compiler with the same tuple.
Then I used another tuple like : cross-x86_64-turtoise-linux-gnu-gcc

with 2. : some packages built with the wrong flags and illegal instructions were embedded.
I think sqlite was the culprit.

with 3. : what is described under point 8. "What Goes Wrong"

Way 1 seems to work.
My hope is that the packages respect the flags more strictly if they think
that they are cross-compiling because the CHOST and CBUILD tuples are different
with way 1.
For the binpkgs to be accepted by the target host I also need to change the target hosts
CHOST.
But this seems natural to me because actually they are different cpus so why not different tuples.
Maybe "pc" should be for "mtune=generic" ... ?
Maybe these new precompiled x86-64-v3 binaries
should be "pc3".
This would prevent that they are accidentally used on the wrong system.