How many Gentoo boxes do you maintain?

[Ralphred]

My server (public facing) syncs daily, the rest of the machines use it to sync daily. They all also run a -p update command and email the expected updates to me. As soon as I see something potentially problematic or beneficial in the lists I do the actual update. I still spend less time updating Gentoo than I used to spend making windows behave.

[cboldt]

Mail, print, gentoo server (screenless laptop, Intel) faces the public on some ports
File, music, photo server (medium tower, AMD) faces the public on some ports
Laptop

Either the tower or the laptop are the workstation of the moment, depending on where I feel like sitting.

All three are updated once a week. Most weeks are uneventful. I use ssh and screen to "stuff" esync command onto slave machines. For gentoo purposes, the gentoo machine is the master, the other two are the slaves. The `esync` command is a short script in .bashrc that does `emerge sync` in screen window 1, and `emerge -puDNv @world` in screen window 2.

By commanding `esync` on the master machine, all three machines are synced, and all three report what they would do to update world. I may make some manual changes to package.use - this goes machine by machine. When the pretend updates are agreeable, each machine is individually commanded to `emerge.new`. This is another short bash script that does `emerge -qUDN;emerge @preserved-rebuild;emerge --depclean; and revdep-rebuild --quiet --ignore` The make.conf contains the --keep-going switch.

All of this can be run and monitored from one machine, and I usually do so from the laptop. Have a terminal open to gentoo server, and a terminal to the file server (tower). The terminals contain the instance of screen that show the updates progress.

After the system update (including config changes, restarting services indicated by lib_users) I run rkhunter and tripwire to find the material changed on purpose, then update the rkhunter and tripwire databases. Then I update the backup drives that are external to the computers. All those drives are connected to the file server, and update is a cronjob that runs if the drive is powered up and the backup partition is mounted.

An easy week is half hour, totally routine. Many weeks are hours long, but routine. Long compile for example firefox. Many weeks include simple config changes, and these are spread across machines. Once in a blue moon gentoo throrws me a challenge. Most recent was the glibc/openntpd issue.

System is mostly stable. A few unstable. In the mix of update maintain mktwpol, a package I composed to automate creating tripwire config files. More particularly, I add and remove package names from the mktwol config file, following the contents of the gentoo portage tree as it is revised over time. This too is usually uneventful (zero action needed), but once a month or so adds ten minutes to half an hour.

[Goverp]

Depends how you count mine. I've an hp laptop with an AMD bdver4 cpu, and a desktop with an AMD ryzen2 cpu.
I have 3 Gentoos: one is a chroot where CFLAGS is the greatest common divisor of bdver4 and ryzen2 , which is bdver4 less a number of flags (fma4 and several others). It's a binpkg host.

The two real boxes have -march=native, and the appropriate CPU_FLAGS_X86; this means a few packages get built from source for native, where the CPU_FLAGS_X86 differ from those in the chroot, the rest just get installed as binpkgs from it.

This isn't as successful as it sounds; few packages declare specific CPU_FLAGS, so most get installed as the bdver4-less-some-other-optimizations binaries. But I guess that's still much better than a generix X86_64.
_________________
Greybeard

[Anon-E-moose]

As far as time spent with gentoo, fixing blocks, why doesn't it do this, etc, isn't that much in the scheme of things. Not talking about compile/emerge times, because that's wildly dependent on hardware and other things and will definitely vary from machine to machine, much less person to person.

Having said that, I've always disciplined myself to pay attention to what the "emerges" are doing. I always run with -p (pretend) first, and keep on with that until I get a clean, no-block output, then I may run the real emerge, or I might wait until later in the day or even the next, if the updates are really minor. I also remove apps and stuff that I haven't used for at least six months and keep the number of packages down, which keeps conflicts down, etc. My package count on this machine (main desktop) is 870 or so packages. I also, at least at the moment, have only gtk2 apps, no gtk3 yet, and thus no gtk3 libs, etc. I do like to run a modernish gcc, but no necessary the lastest and greatest, I'm running 9.3.0 at the moment, but I lag behind on things glibc, where I'm still running 2.29, because I like very stable for that. I also don't run the latest kernel when it comes out, unless it has something that I really, really, really need at that moment and I manage the kernel outside of the portage tree.

Updates on major things, like gcc, glibc, etc I try not to jump on the bandwagon and be the first to be bit by problems, I give it anywhere from a few days to a few months for many "potential" issues to shake out. Then I consider when to update those pieces. I also like to only keep one version of things, one each of python2/3, one ruby, etc. I don't like things like multiple interpreters on my system, they're fine for devs and testing, but really shouldn't be shoved onto regular users (who then start lots of threads about problems from those multiple interpreters., yada, yada, yada)
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[Buffoon]

Five. And sixth which I use as a backup server, I do not update it, it has some super old 6 GB SSD for OS which probably would die if I started compiling routinely. Never understood complaints about time-consuming Gentoo updates. I use SSH and screen, I believe my time spent on upkeep of 5 boxes is not in excess of 15 minutes per week. All boxes are ~amd64.

[pa4wdh]

I have 5 physical systems (desktop, laptop, RPi2B+, router, server), 9 lxc's on the server and a qemu based VPS.

I usually update all my systems every 3 months and give them the TLC they need. Sometimes (with important security updates for example) i to a extra update.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com

[DancesWithWords]

[eccerr0r]

I don't think any of us watch emerge output, or at least sit there and use it as a primary activity. Also I am ignoring compilation time, all I'm concerned is time/effort you spend per computer to solve merge conflicts. Eventually you will still get to a point where you run out of time per day with enough uniquely configured computers. Running emerge -p, waiting for it to fail, doing something to try to fix it, running emerge -p again, etc., etc., - this discovery loop can take a significant amount of time - the reason why I tried to ignore replicated machines is if one machine has all its problems solved, the other machine (rsynced, usebinpkg, etc.) by the transitive property also has its problems solved.

Not only after you do get a complete solution to an emerge problem, there's packages like Locale-Gettext, intltool, setuptools, etc. that should be remerged before another package fails because those packages didn't get set up properly despite being installed, and a whole bunch of random stuff including a recent rdoc failure that didn't set the current ruby properly after ruby got upgraded. All tend easy stuff to fix but the machines just sit there waiting for you, even if you --keep-going and that build happens to be a bottleneck build.

At least when I first running Gentoo, a merge problem's debug-repair loop seemed to take a lot less time in the past. I would like to think that I've learned more about the system, and even armed with this knowledge, these problems are getting more difficult to solve. I'm not sure which is worse - portage is giving more and more misleading diagnostic, or simply complexity is getting higher and higher. I think it's mostly the USE flag mess that's now causing a majority of conflicts, but should it be this case?

Based on some other threads, it seems some people have resorted to deleting a good portion of their boxes in order to figure out some of these problems - so at least my experience does not seem to be isolated. I would say the option to delete packages, emerge @system, portage tree rollbacks, dumping over wit a fresh stage 3's, etc., should be of last resort yet people seem to be quickly suggest these options, which is alarming. I hope these people were just pulling my leg instead of seriously suggesting these options.

And as I thought, at least for these recent issues I had, I discovered that indeed there was no need to resort to the drastic options. Like some of the circular dependency issues, some workarounds may have been needed but ultimately portage was coaxed to a solution using the latest portage tree. But it took a long time to find these issues, and emerge -p itself takes several minutes before it reports whether it found a solution or not. I mean I think I spent at least a day or so total (adding up all the time here and there) to coax just one machine to solve. And this information was not directly portable to the other machines suffering a similar issue - every machine had a unique issue.

And since the number of machines I have is also increasing, this leads to an quadratic increase in time used to keep the machine army up to date -- this is my main concern here... This leads me to believe there's really some limit to the number of machines one can truthfully keep up to date without using replication or making installs generic as this becomes a full time job or more... This poll was intending to find just that.

BTW,

[Tony0945]

I don't think it increases quadraticly unless you have radically different installations. Yesterday I took from breakfast to 8:00PM to update my server, which unfortunately is the slowest modern machine because I deliberately chose of low power APU for power consumption savings on a machine that's on 24/7.
The biggest culprit was python. About two hours of that was building during which I watched Waiting for God streaming from Britbox via Roku. A very funny series even though now that I'm the same age as the protagonists, some things don't seem so funny anymore. OTOH, my wife likes it better. I think she identifies with Diana Trent.

The advice in these threads https://forums.gentoo.org/viewtopic-t-1129728-highlight-.html and https://forums.gentoo.org/viewtopic-t-1129728-highlight-.html was invaluable. After that, the next box was easier but some --depcleaning and use flag additions solved that in about an hour. As you say, most of the time is waiting for portage calculations. Since the algorithm is non-polynomial, it makes sense to run it in chunks. if you can build @system, the python problems are solved or 99% solved.

[eccerr0r]

Well, the machines' setups are different enough to warrant their own installs instead of sharing one machine for two functions...
But it is true sometimes data about one problem found can be used to help out another. I think this time around, emerge -ep @world found a lot of the issues that I had in my world and /etc/portage/package.{use|accept_keywords|mask|unmask} - though even with it, careful checking of the diagnostic output and cleaning up the config files eventually helped portage find a solution, as well as a bit of encouragement from fgo. The previous update I had found that increasing backtracking helped but not this time around.

I worry the next major update, which may only be in 3 months just because I'm an unlucky person, all learned from this event will be useless once more, plus I'll have another machine on the docket. Time to figure out emerge conflicts increases linearly, number of machines increases linearly, sounds a lot like a quadratic increase of time spent... at least it's not exponential ...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Tony0945]

I usually have similar setups. same tools, all OpenRC, same GUI, all either refind if UEFI, or grub-legacy if not UEFI, same browsr, thunderbird or no mail client. Actually Thunderbird-bin so emerging is rarely (I can't recall ever) a problem. This go around I added python 3.7, 3.8 and 3.9 to python targets. Just keep them all until they pass out of portage. Had to drop 2.7 (still in overlay but causes blockers). that required dropping some overlay packages. I've managed to avoid rust, lua, go, ruby, and chromium. Setup differences are minor between machines. Some have apache, most do not. Two have dnsmasq, one with dhcp, and one without. Virtualbox only on the Ryzen. Most differences are local, i.e. not under the control of portage, installing into /usr/local/lib64 and /usr/local/bin.
I have the least problems with ebuilds in overlay where I wrote the code because there is no python. I only program in C/C++ I haven't written any assembly code for over twenty years, but it's like riding a bike. you never really forget, just get rusty, but it all comes back with a little practice.

Some like to experiment with langages and different GUI's, Plasma on this one , Mate on another ... I've settled on openbox plus pacmanfm but one machine boots openbox or mate. I solved a nagging focus problem that was caused by openbox configuration. I run some mate applets and mate-terminal on openbox which generates dbus error messages but they work ok. Just don't understand why everything wants to talk to dbus. Seems like sloppy design.

[eccerr0r]

I suspect it's the mixture of programs that cause issues. A lot of them that I use end up wanting ~amd64/~x86 versions installed, and oddly enough I keyword them because they frequently merge correctly unlike the stable versions. However as portage moves on, these ~unstable versions are quickly put out to pasture, and I tend to keyword exact versions to reduce version thrash...

... which probably is misguided as these are the ones that also end up causing merge issues as they stay stealthily installed and go out of date. Along with the /usr/local/portage local overlays that don't get updated along with the rest of portage and limited backward compatibility (not just portage but libraries, etc., too)...

Sigh. Well, looks like after a reboot my PVR machine is up to date and working properly once more. Now time to migrate it to 17.1...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[saellaven]

[eccerr0r]

[389292]

I maintain only one, but very frustrated even with that, the reason is how manual my update procedure is. (though it isn't really a foul of gentoo, but my own desire to make things different, ~secure~ etc.). My /boot partition is encrypted and detached (to the external flash drive), so each kernel update I have to connect my flash drive, enter decryption password, mount, compile kernel, compile initrd, update grub, sync, luksClose, umount and finally detach the flash drive. Overall it's 10-15 commands, spread throughout 15-20 minutes. Another issue is that my initrd image is very messy, I have ~3 different keyfiles and 3 different luks headers (which are also detached to the same flash drive), I also use btrfs for root, mounting it is more confusing than mounting regular drives or luks. So I'm kind of stuck with this messy setup, I also don't want to bake this messy setup into a single script to make it easier, because it would mean that it will be like that forever, I need to tide it up first, than make a script. I also thinking about switching to Artix, but also don't have courage because of how messy my partition scheme currently is.

[Anon-E-moose]

Most stuff in my local could be consider old or depreciated, but 90% were stable when they moved to local.

Just like this morning, gtkdialog wanted to update from what it had to a new version, I didn't like what the new one wanted to install, so I copied it to local. Not an unstable package, just a superseded one. Now eventually the ebuild might need some work, when the depreciate the EAPI level it's at, but until then, it simply works.

I don't have problems with PYTHON or RUBY, because I don't let things devolve to more having more than one version installed, keeping multiple versions around, especially if you don't need them, is a recipe for a submarine danger, ie you'll be bit by something you didn't expect.

I do keep python2.7 but it's not an env nor is it used by portage for any calculations other than a small handful of packages.

I have things set like this,
PYTHON_SINGLE_TARGET="python3_7"
PYTHON_TARGETS="python3_7"
RUBY_TARGETS="ruby25"
and for python2.7 I use a use package flag if needed

[eccerr0r]

Yay I just got torpedoed by a submarine while migrating to 17.1 ... another 5 minutes lost...

I had app-accessibility/pocketsphinx installed along with its dependencies. Apparently it didn't cause any issues on the upgrade to latest despite it and its dependencies no longer being in portage, but having 32-bit libraries lying around the 17.1 migration procedure failed the package as it couldn't find the ebuild for rebuilding the dependency libraries back into the proper 32-bit locations...

...whether to --unmerge the packages and lose that software is the question...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[Anon-E-moose]

If you haven't upgraded all your old systems, you might be able to find what you need on those systems.

Barring that you can "cheat" and hand move those dependency libs from lib32 to lib, which is what will happen if it gets recompiled.
AND you might have to modify the CONTENTS file matching that in /var/db/pkg/<whatever>. Not what I recommend, but it's certainly doable, just depends on how much effort you want to put into it.

Question though, aren't the newer sphinx's just upgrades to pocketsphinx? I don't use it, so I have no real knowledge of it.

Edit to add: If packages are still installed, but the ebuild has been removed from portage, the ebuild that was used to build every package is under /var/db/pkgs/<whatever>/<whatever>/<whatever>.ebuild. You might have to scrounge around for any patches that were in the files directory of the ebuild.
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

[eccerr0r]

I was *that* tempted to write such a script to do 32-bit auto migration with mucking with the databases, though I'm sure there are some hardcoded lib32s in some of the library files and won't fully work...

Alas this system with the submarine is the only system with this (technically I don't *need* this program, was experimenting with it as part of another project - was trying to get this computer to do voicemail to text ...) I might have to go to the full version, but for now I skirted the issue by --excluding these packages, that's my 5 minutes.

---

Just started updating my 64-bit laptop after finally getting portage to find a solution to merge. 826 packages slated for merge, let's see how many fail... and thank goodness that distcc is currently working for me...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[dmpogo]

I have just 4 machines ( laptop and desktop at home, desktop and an old server in the office). Run 99.9% stable on all of them. Not much problem updating. Currently just polkit is hanging around since I do not to update spidermonkey.

I use Gentoo update as a zen relaxing activity, when I am tired of working, and thus it does not bother me having to solve some rare puzzles it throws at me.

[figueroa]

You don't want to update spidermonkey? Why not? It just comes with the territory. Yes, it's weird that it needs updating so often. Twice in November, twice in December, once in January, not yet in February.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

[eccerr0r]

Now that spidermonkey requires rust (since it's using the spidermonkey in firefox) I can understand the aversion to upgrade it.

However I'd be worried about not upgrading it too...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[flysideways]

I am presently actively maintaining 4.

One each arm64 and ~arm64 Pi4 w/8gb, they build natively and are on external ssd's. I initially intended to chroot those drives from the 3900X, and did for a while, but have found it more interesting to just stay with native builds for now. There are some more Pi's but they are derived from these.

One Ryzen 3900X that has chroots that I am not counting because they are more look see and not actively maintained. It has an AMD video card and is entirely open source. Mixed stable and ~amd64. It may eventually be dual boot, stable and testing.

One FX8350 with an Nvidia Quadro with the Nvidia binary. Trying to keep this one stable only.

I have a MythTV computer that has not been Gentoo for the last 5 years because I got tired of the changing support for MythTV. I had maintained a Gentoo MythTV box for 10 years prior to that point. This computer was used by the family, but Netflix and YouTube have overtaken MythTV's utility and I am not sure of this computer's future.

I also have a 4th generation i7 box that is not being used presently. This one may end up with a Gentoo install in the near future.

Gentoo is a distraction for me, nothing I do with it is required for daily use.

[Ionen]

[dmpogo]