More atomic upgrades with portage

[mid-kid]

Hello, I rarely write on this forum, but I felt this was the right way to discuss this, so here I am.

As of the recent years, I've been upgrading my machine on about a monthly basis. This allows me to be sure that no unexpected changes or breakage happens during the month, and allows me to coordinate my upgrades with free time to spend fixing any possible breakage/new bugs in new software. Unfortunately, this also means each upgrade is *long*, usually ranging from 12 to 24 hours, depending on what packages decide to upgrade. There's a lot of time where I'll want to use my machine during the upgrade.

Portage only ensures system consistency before and after an upgrade, but during an upgrade, it's anyone's game. There's many packages that don't deal with this well, one very annoying example being Qt. If you update dev-qt/qtcore, all Qt applications will stop working until all of the Qt components they use have been upgraded as well. This happens with things like PYTHON_TARGETS as well, where the gentoo news item will generally recommend rebuilding twice to prevent breakage, by first enabling the new python target, and afterwards disabling the old one. There's other packages with this issue, but less frequently so.

Looking at today's upgrade, I'm seeing dev-qt/qtcore will be upgraded as the 59th package, yet dev-qt/qtgui is package 323, somewhere near the end of the upgrade. For a huge portion of the coming 12 to 24 hours, I won't be able to use any Qt application, making me unable to use my password manager, among other things.

Granted, Gentoo isn't the only distribution with this problem. Every single binary distribution deals with this as well, and some have even decided to deal with this by forcing the user to reboot their machine to apply any non-flatpak update (kde neon, gnome software center). The big difference between those and Gentoo is that the time between the start of an upgrade and the point where the system is fully consistent again, is much, much shorter.

This is why I was thinking, if there'd possibly a way to improve this situation in Gentoo. If not make upgrades fully atomic, then at least increase atomicity by shortening the timespan during which the system is broken.

Gentoo has binpkgs, and I feel that leveraging this feature, or otherwise delaying the package install stage to when everything is finished building, would be key to improving this. emerge --buildpkgonly complains about not all dependencies being satisfied before build, and portage has no way of retaining the "image" directories until the transaction is done, so those are out of the question. Two solutions I can think of, are either somehow leveraging --quickpkg-direct to install every currently installed package into a chroot, then building binpkgs in there, and afterwards installing them on the host, OR, setting up an overlayfs chroot, and building inside of that such that installed files end up in the overlay instead of the real system, and afterwards installing the binpkgs on the host.

There's drawbacks to both of them, --quickpkg-direct won't copy over all configuration files, and an overlayfs chroot is hard to get right in terms of mounts, especially if the host system is split up over multiple drives or partitions.
I'm wondering, has anyone tried anything like this before? What are your thoughts? Maybe something like this could be implemented in portage itself?

[Banana]

I do updates more often and thus the amount of changes is smaller.
Maybe using https://wiki.gentoo.org/wiki/Binary_package_guide provided by another machine could solve your case? This way you make sure the updates a working on a not-so-important machine.

[sublogic]

I wonder if a solution can be cobbled together using LVM snapshots ? (LVM= Logical Volume Manager).

1. create a snapshot of root (and /var, if a separate filesystem)
2. convince portage to update the snapshot(s)
3. merge the snapshot(s)

I use the LVM but I haven't played with snapshots. No experience in the matter. Especially step 3.

Maybe safer,

1. create snapshot(s)
2. start emerge
3. meanwhile, set up a chroot to the snapshot(s) and work from there
4. delete the snapshot(s)

[Goverp]

I build all the binaries in a chroot, and then install them into the main system. That's not to make the main system updates more atomic, it's because I also load up a laptop from the same binaries. a few packages (currently one on each system) in fact get compiled natively on the main and laptop boxes because of differing CPU_FLAGS_X86 optiizations.

I set up the chroot as you describe, using quickpkg; the config files aren't a problem - the ones put in the package are the ones you want anyway. More to the point, you should only be installing binary packages that have been updated, and so they should contain the latest configs. In fact, I don't think you need to populate the chroot with old binaries.

My setup uses bind mounts to share bits of the portage configuration between the chroot and the main system. Basically /etc/portage, /var/db/repos and /var/lib/portage/world. You may as well include /var/cache/distfiles and /var/cache/binpackages too. That way any alterations you need to get the compilation to work in the chroot propagate to the main system.

[Dr.Willy]

Have you considered distributions that do atomic upgrades out-of-the-box, like Guix or Nix?

[mid-kid]

Thanks for the replies, this is all very interesting. Unfortunately, I currently don't have the time to try different approaches, so it'll remain mostly a thought experiment for now.

I think the LVM solution is very interesting, but I'm not sure it won't have any drawbacks, aside from the obvious part where I'll have to reformat my drive with LVM.
I'll have to play with it sometime.

I set up the chroot as you describe, using quickpkg; the config files aren't a problem - the ones put in the package are the ones you want anyway. More to the point, you should only be installing binary packages that have been updated, and so they should contain the latest configs. In fact, I don't think you need to populate the chroot with old binaries.

I'm not sure config files are not a problem, when critical files such as /etc/profile are excluded from a --quickpkg-direct=y install of sys-apps/baselayout. And that's without mentioning auxiliary files in e.g. /etc/kernel, /etc/eselect, /etc/env.d and similar that affect a build and may not be set up the same in a fresh chroot. If I were to follow a chroot approach, I'd love to be able to destroy the chroot after every upgrade, in order to ensure a consistent state compared to my host system.
I understand that having a clean system to build everything from is the right solution when you're maintaining multiple devices, but it sounds like an extra headache in my current "single gentoo" situation.

Have you considered distributions that do atomic upgrades out-of-the-box, like Guix or Nix?

I have, but both of those have serious drawbacks for me, including but not limited to having to rebuild the entire reverse dependency tree whenever a package has been modified in any way, shape or form.
I wish there'd be a distribution with all the advantages of Guix and Nix, like being able to have several versions of the same dependency installed, and being able to have reduced environments with only a selection of packages installed, and temporarily install packages, without the strict reproducibility requirement.

[Goverp]

...

I set up the chroot as you describe, using quickpkg; the config files aren't a problem - the ones put in the package are the ones you want anyway. More to the point, you should only be installing binary packages that have been updated, and so they should contain the latest configs. In fact, I don't think you need to populate the chroot with old binaries.

I'm not sure config files are not a problem, when critical files such as /etc/profile are excluded from a --quickpkg-direct=y install of sys-apps/baselayout. And that's without mentioning auxiliary files in e.g. /etc/kernel, /etc/eselect, /etc/env.d and similar that affect a build and may not be set up the same in a fresh chroot. If I were to follow a chroot approach, I'd love to be able to destroy the chroot after every upgrade, in order to ensure a consistent state compared to my host system.
I understand that having a clean system to build everything from is the right solution when you're maintaining multiple devices, but it sounds like an extra headache in my current "single gentoo" situation.
...

I think you're misunderstanding. Emerging from a binpkg server obeys normal emerge rules - existing config files don't get overwritten (at least, not unless you specifically ask for that). The only time the config files from a quickpkg binary get used is if you install a new package not previously on the target machine.

I was misunderstanding your use of --quickpkg-direct - it's not an option I've every noticed, but it's pretty much the same as building binpkgs with quickpkg. Either way, though, I think the results are the same. If you built your chroot packages without changing their config files from the default, emerging from the chroot as a binpkg server will act just as emerging from source in the usual way.

The thing to do is a i-time build of your packages in the chroot from scratch without making any config changes. That's to say, ignore using quickpkg to populate it, and instead copy your /var/lib/portage/world file, and do an

Code: Select all

emerge -e @world

Then all the config files in the chroot are unmodified, and normal emerge rules should apply when you emerge from the chroot (assuming you specify all the right options - with binpkgs, that happens automatcally; I guess the various --quickpkg-direct options would cover that too).