~Package updating via binary host that is stable

[LIsLinuxIsSogood]

Is it possible/or wise to attempt to make a stable branch system (amd64) a host for binary packages to a unstable ~amd64? I know that I've heard mixing branches is generally not a good idea, and this appears to be a sort of similar issue here with dependencies that would be required at build time but not a runtime. I have two live operating systems at the moment and both are working great, but it's just that every time I update I get the sense that it could be done better if all the building was manageable from one central place, which just happens to be where I have stable branch.

So is that a big no--no building experimental packages for other systems to access from that machine if those packages are already installed stabled there on the binary package host?

And if so I can forego the entire idea in favor of another solution to the situation that just involves some minor inconveniences like downtime of a virtual host in order to update more efficiently on the experimental branch and its software and hardware.

Looking forward to getting this one figured out, so I can move on with no worry of corrupting or destroying my stable branch system!!! Or the experimental one, he heh

[krinn]

It is, but you'll see that technically, it's not.

When a stable tree wish build an unstable binary it must build ALL binaries that unstable package depends on, and because the stable host doesn't have them install, it mean the stable tree will have to nearly always build a lot (and really a lot) of packages.

[LIsLinuxIsSogood]

Thaks krin that is a huge help for what it is I want to try. As for the toolchain and kernel, would it be a good thing to consider something like a small chroot environment where I might actually enter before passing the actual command you put here. That way maybe if I can fool the operating system to think it is already an experimental branch (from inside a chroot) then it will have the ability to install a second kernel there and toolchain. Does that make sense to attempt?

[krinn]

if you use a chroot, then nothing prevent you from doing what you wish in it. You don't even need anymore the --buildpkgonly as you don't care about what is install or not.

[LIsLinuxIsSogood]

Thats right, ha

[LIsLinuxIsSogood]

So just got done setting up the chroot it was not a lot of work, but somehow a lot more still than I remember it being in past, but I think because in the past I was not trying to rebuild and update everything like I am now going to have to do so that the latest experimental packages are "able" to be built in one place and then shared to another machine. So far I have upgraded kernels on both machines to ensure that is not an issue. I have copied many of the package.* files from /etc/portage from one machine to the other, and then I just was going to run a --deep --update command when I noticed some other things about beingn in a chroot that aren't too cool.

Like this:

[LIsLinuxIsSogood]

Nevermind a quick search for chroot missing /dev/pts turned up a debian link about making sure to separately bind mount this filesystem location

[LIsLinuxIsSogood]

It's nice to be actually compiling on a faster machine for a change, since my laptop usually runs very slow due to the number of tasks always working and memory being used, etc.

Based on the kernel compilation time it was like 1/2 the time, and that is comparing a Pentium IV to a virtual machine running on top of the core i3...my desktop compiled the kernel (which is about 11M large) in roughly 15m, which took double the time on the virtual machine. We will see how that goes once I upgrade the ram on both machines and I have feeling that shouldn't make any impact HERE.

[LIsLinuxIsSogood]

Just finished updating glibc so that is good, and when I went into now update the rest of the toolchain it auto-merging perl upgraded version. Gentoo is not for the weak hearted. Wow I think this deserves another post in the portage section about why does Perl package get in the way of system update, and then suddenly needed by just something so important like the toolchain...weird

[LIsLinuxIsSogood]

Is there going to be a known issue with trying to compile the toolchain with makeopts set to higher than -j1 like -j3 or -j4???

[krinn]

[Hu]

For packages with a correct build system, the only limit on -j is whether your build system has sufficient resources to handle all the concurrent processes. Depending on source code language and enabled features, some packages can require substantial amounts of RAM for each individual file. Setting an excessively high -j might run you out of RAM and cause the build to fail. In extreme cases, it might have negative side effects on the host (OOM killer, etc.). In no case should RAM exhaustion result in a package that finishes, but produces incorrect code. However, if you overheat the system to the point that the CPU malfunctions, there is no way to predict what would be corrupted, so you cannot trust the built package. Ensuring adequate cooling should be your top priority.

[LIsLinuxIsSogood]

What I was referring to or my intention with the question was not actually specific to CPU or RAM issues exactly. I happen to recall at times from people that using higher settings for cores was the cause of packages failing to build...so is that logic wrong? Maybe they were referring to something else like emerge jobs IDK.

[NeddySeagoon]

LIsLinuxIsSogood,

You can set -j to more than (logical) cores +1.

It may cause cache thrashing, where a context switch results in data being evicted from the CPU cache that with a lowel -j would have remained in cache.
This is harmless but results in additional much slower fetcher from main RAM, which costs extra time.
You can also run into swapping. This hurts performance a lot. That does not mean that swap space is used. The kernel will drop code that has a permanent space on HDD and reload it when its needed again. Swap is only ever used for dynamically allocated RAM, as it has no permanent home on HDD.

None of this changes the output code any, you are just using extra non productive CPU cycles.

Having built Gentoo on a system with 96 cores and 128G RAM, I can say that setting anything over -j50 is as good as unlimited.
The only way I have been able to get the load average over 100 on this system is to run emerge with --jobs=8 and MAKEOPTS="-j50".
Even then, it was 'only' using 60G RAM.

The kernel and glibc are good tests as both will spawn lots of parallel jobs if you ask.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[Hu]

There are plenty of packages with incorrect build systems that incorrectly assume a build product will be available when they have not told the build tool to ensure that to be the case. In serial or low-parallelism builds, missing dependency statements can often go unnoticed because the earlier steps will have completed before the result is needed. In a perfectly parallel build, where every object is built as soon as its stated dependencies are satisfied and it has no access to outputs not declared as its prerequisites, those mistakes break the build.

[LIsLinuxIsSogood]

Granted there are, but I would assume that the more central or core the package is to the operating system, say the toolchain then that probably means something that the developers who are also responsible for the packaging have thought of more (if not everything)...since clearly they can't account for everything and anything, but I thought for some reason that parallel builds was somehow a "hard" no for some of the packages...anyway thanks for the information it is very helpful.

I have another unrelated question about no-multilib versus multilib, since what I want to know could stir up more discussion hopefully it is ok to just post here, the following:

Which packages requiring 32-bit libraries are there that won't build on the 64-bit pure no multilib system? And is there some way to tell from the package database or search via emerge if the package requires the 32-bit support...I would imagine it has to be somewhere in the ebuild of every package right? ABI keyword? I am trying to figure out why a particular package isn't building, but I've posted and can seem to figure it out. Anyway, since it is also related to a larger question about kernel upgrades to virtual machines, for which this is definitely new territory for me and I want to take my time to make sure that I don't skip important steps. I'm right now on the guest additions installation and it is throwing me this weird kbuild kmk missing or KBUILDDIR variable or something, here's the failed build log:

https://paste.pound-python.org/show/K2sZqrN2zAJUIhVpVq3h/

[NeddySeagoon]

LIsLinuxIsSogood,

When you set a /no-multilib/ profile the packages that are 32 bit only are masked by the package.mask from the profile.
Profiles use inheritance from one level to the next, which makes working out what is actually package.masked quite difficult.

The package.mask isn't always right either. Packages change, package.mask lags behind.

There is always one more bug, so software is never perfect.

Packages that won't parallel build usually have broken build systems. Devs forcing MAKEOPTS="-j1" are waiting for upstream to fix it.
Its not something users should ever see.

There are a few packages like rust, that do use parallel make correctly but still have a long single threaded part. Its just that parallelism is not possible everywhere.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.