[split] upgrade philosophies

[pjp]

Split from How to avoid upgrading an installed Python 2.7 package? --pjp

At the end, only very few infrequent updaters remain, as the majority of them turns to alternatives.

I'd be curious to know what those alternatives are which provide the flexibility of Gentoo without the update frequency. I might switch! :)

Good luck with your progress. Let us know how it went.

[segmentation-fault]

At the end, only very few infrequent updaters remain, as the majority of them turns to alternatives.

I'd be curious to know what those alternatives are which provide the flexibility of Gentoo without the update frequency. I might switch!

We are going to dilute this thread a bit with the question you asked, but since you asked...

When people switch (linux distributions, political parties and even partners), they often might do so not because the new alternative is "better" - they will do it because the old one has become unbearable. The following quote sums it up very well:

I finally realized my time is worth something, and I’d rather use my OS than…with it all the time, so I went to...

(from: Why did Gentoo Linux fade into obscurity?)

That's how a system "selects" its users.

To paraphrase Linus: If you treat your users like frequent updaters, then sooner or later only frequent updaters will be your users.

Not counting some die-hard "severe outliers", that is...

[pjp]

Good point. That's partly why I'm working to reduce friction of how long updates take. With your comment about choosing something else, I was thinking "Option B does closer to what I want, so I'll use that" rather than "Gentoo doesn't do some of the things I'd like, so I'll go back to Option C which does few or none of them."

Given the frequency of kernel updates, that's an amusing rephrasing of that quote :)

[figueroa]

For one thing, Gentoo Linux did not fade into obscurity and is not in the process of doing such fading.

For two, time invested in updating/maintaining Gentoo is a red herring issue. Update with reasonable frequency given the pace of upstream development and the nature of Gentoo, and very little actual user time needs to be invested. Don't watch the window in which emerge is running, and just go about your business. Come back to that window after in completes it's work, at your convenience, look over the results, and usually no further action is required. The amount of user time spent solving problems related to updates and maintenance is more than proportional to the length of excess time in between updates.

Stable Gentoo installations are not the same as running something like Debian stable. A bog standard Gentoo installation is near bleeding edge upstream. and a ~ Gentoo installation is more like extreme bleeding edge. There is no version of Gentoo that is slowpoke or "sub-stable" or partially "sub-stable." Failure to stay reasonably up-to-date across the board, daily increases the user's chances that the many sources of upstream development become incompatible or mutually-exclusive to at least part of an old, neglected, out-of-date Gentoo system. Excessively infrequent updates are undertaken at your own risk.

Further, the problems faced through infrequent updates are exacerbated (made worst) when the user's portage become mis-configured, especially with pollution of strange USE flags and the system's /var/lib/portage/world.

[segmentation-fault]

Given the frequency of kernel updates, that's an amusing rephrasing of that quote

Actually, you can say it about (almost) everything:

If you treat your X as Y, then sooner or later your will only have Y as X.

X: a noun,
Y: an adjective, or some construction with an adjective

Linus said it for X="users" and Y="idiots", I used the same X, but put "frequent updaters" for Y. Try it with e.g.

X="students"
Y="idiots"

or

X="citizens"
Y="incapable"

It works, because in any interaction of X and Y, where Y needs X and X imposes some conditions on Y, there is (by the nature of "interaction", whatever this interaction might be) a "diffusion phenomenon". Ideas, behavior "flow" like heat. It's heat diffusion in networks. At the end, X and Y have the same "temperature". X and Y have "come together", they exchanged ideas and behavior as if it were heat - so now they think and behave similarly. I didn't check, but I would conjecture that this is the mechanism by which the species (the Y) adapt to their environment (the X) - evolutionary adaptation! - that's why I used the term natural selection to describe the way a system "chooses" its users.

[segmentation-fault]

For one thing, Gentoo Linux did not fade into obscurity and is not in the process of doing such fading.

I didn't say that - even if this could be the end result of the "heat diffusion" process I described above. pjp asked a question, which I tried to answer, giving an example to illustrate my point. However, even if this may not be the situation right now, it is the situation Gentoo is heading to.

Why do I say this?

Think about the process I described above: a system necessarily chooses its users. You cannot evade this (well, by definition, I evaded this fate, but this makes me a "severe outlier" (in Hu's words), so I am the exception that proves the rule). And you all here have admitted that you have to be a "frequent updater" to be able to use Gentoo - without the kind of friction that I encounter, that is. Therefore, by the Generalized Linus Postulate (let's give that proposition above, with the X and Y in it, this prominent name for the sake of discussion ), in the limit ("sooner or later") you will only have frequent updaters as users.

But what percentage of the totality of users are frequent updaters? A very small one.

Now combine the above and you must come to the conclusion that Gentoo is heading to a situation where its user base makes up only a tiny percentage of the totality of users - that's the same as saying that in the limit Gentoo will fade into obscurity.

The purpose of this whole thread (besides getting practical help for my concrete problem at the start, which I got and am very thankful for this) is to show you where Gentoo is heading if you don't change this situation (the "conditions that X imposes on Y", here: having to update frequently, or else...hell will break loose!).

Now you have the mathematical proof that Gentoo will fade into obscurity.

Don't watch the window in which emerge is running, and just go about your business. Come back to that window after in completes it's work, at your convenience, look over the results, and usually no further action is required.

If you look at the start of this thread, you will see that my problem was that I could not move past the first message - I do let it do its job, when it finally gets to compile something, that's not my problem.

Failure to stay reasonably up-to-date across the board, daily increases the user's chances that the many sources of upstream development become incompatible or mutually-exclusive to at least part of an old, neglected, out-of-date Gentoo system.

I consider this an urban myth. Logic does not support what you say - and portage is ultra-logical. portage will take a set (any set!) and update it to the versions in its tree IF (and, unfortunately I might add, ONLY IF...) there is a way to fulfill ALL dependencies of ALL packages in the set given, including (but not restricted to) python targets, slot and USE flag dependencies. Therefore, an "old, neglected, out-of-date Gentoo system", as you call it CAN be updated with portage and portage will do its job as in any other case.

But if it is so, then where is the problem?

The problem is that the set you give portage to update must be what I call updatable. If, for example (to pick again the subject of this thread and try to stay on-topic), some packages of the set (installed or not, doesn't matter) contain a dependency like

Code: Select all

>=dev-lang/python-exec-2[python_targets_python2_7(-),-python_single_target_python2_7(-)]

and python-exec2 has been removed from the python2_7 target, then this dependency is by construction unfulfillable - and the set becomes NON-updatable, since portage insists that ALL dependencies be fulfilled.

The problem is not "time elapsed since last update" - although, on the surface, it looks like it is - the problem is how portage, or the user, deal with dependencies. Since portage will not change its mode of operation for a "severe outlier" like me, the ball is in my field and I have to find a subset of the world file that is updatable according to portage's rules. Then portage will dutifully update that one and there will be no difference to a frequently-updated system.

The difference will be in the sets that get updated. In the frequently-updated system, the user updates world, so world must be updatable, always, after each sync. This forces the user to unmerge software that is not in the portage tree. Sometimes this makes sense, in other situations it is totally counter-productive - a true catastrophe.

In the seldom-updated system, the user updates the maximal subset of the world file that is updatable. If done judiciously, this can help the user to both update what can be updated and keep what makes sense for him to keep.

Think about it in "sets that you update", that's all there is to it. portage works the same for all sets.

Excessively infrequent updates are undertaken at your own risk.

In view of the above, this just creates the impression that being a "frequent updater" is somehow an inherent prerequisite of Gentoo. If Gentoo does not want to fade into obscurity, as shown further above, it must put some effort into designing methods, tools and strategies to define maximally updatable subsets of the world file.

It's not as difficult as it looks. I will write something about it when my updates are finished.

Further, the problems faced through infrequent updates are exacerbated (made worst) when the user's portage become mis-configured, especially with pollution of strange USE flags and the system's /var/lib/portage/world.

This is another urban myth. There is no such thing as "strange USE flags" - the user should be able to use as many (all, if he likes) USE flags, as he wishes. It's portage's job to find a solution. If there is none, because A needs B, B needs C through a USE flag and C needs some huge library D, which you cannot upgrade for some other reason - well, then portage should offer an option to ignore C (ignore dependencies based on a USE flag).

However everybody here keeps saying portage will not do it. In that case, it's not a problem of the user having "too many" or "strange" USE flags - it's portage that is incapable of offering at least some solution, even if partial.

[eccerr0r]

Really it's masks (both package.mask (blacklist) and accept_keywords (whitelist) that makes the life of portage harder. Also implicit masks like not having the package in portage whether it was deleted or simply doesn't exist (like saying you want a version that never existed) are no different than masking.

Ultimately if a program simply requires something not available, whether it be a dependency or the author simply has no time to update every potential conflict - you can't write an ebuild to work around this...

[figueroa]

segmentation-fault -- I should know better than to continue to engage in vain disputes, but it seems interesting seeing you tie yourself into knots about this.

I didn't originate this line, but Gentoo isn't really a distribution. It's a tool kit with which a user can create their own distribution following Linux the kernel and GNU upstream. As upstream ratchets upwards, certain combinations become difficult, implausible, even impossible. When a user does not want to follow the general upstream thread, move the program(s) into the local repository, and , if necessary, write ebuilds or install outside of portage. The user definitely should not whine loudly when portage will no longer support their out-of-mainstream quirks.

Upstream sets the pace, and Gentoo gets pulled along. In fact, the Gentoo developers and maintainers do an awesome job of protecting its users when upstream goes off the rails, and I am thankful.

I get irritated sometimes when a program I'm using forces me to update and removes the version I'm using from the tree with no apparent reason. Then I acknowledge that I am not the volunteer who is doing the work, and move along.

Keep up or get left behind. Consider Debian stable (not really). It's been a fun rant to follow. Next.

[pjp]

The amount of user time spent solving problems related to updates and maintenance is more than proportional to the length of excess time in between updates.

That's your experience, not mine. Your experience does not make the experience of someone else a red herring issue.

segmentation-fault's issue in this thread is absolutely brought on by their choices , and in my opinion is not "relevant" to normal / reasonable Gentoo updating procedures.

For me, I'll need to make meaningful progress by mid-year in reducing time involved with updates or I'll need a different solution.

[segmentation-fault]

...to continue to engage in vain disputes...

...but Gentoo isn't really a distribution...

...Upstream sets the pace, and Gentoo gets pulled along.

...Keep up or get left behind.

All these platitudes are irrelevant to what I have been saying here - nice half-truths pulled into the discussion in order to justify easy decisions about complex problems. If you can't imagine other than that as a true solution to the problem(s) of this thread then, indeed, there is no point in reiterating urban myths.

This discussion started from a concrete problem, to which we found a solution that pleases both myself and portage and all I am trying to tell you is that portage can do much more than what you are accustomed to, that you should not fear to abandon old habits (e.g. frequent updating), that you should embrace the idea of updating a maximal updatable subset of world, instead of world itself , and that if you don't, Gentoo is heading into obscurity - to which I even gave you the outline of a mathematical proof.

Nothing of what is quoted above really touches what I'm saying.

[segmentation-fault]

For me, I'll need to make meaningful progress by mid-year in reducing time involved with updates or I'll need a different solution.

As soon as I finish with my updates, I will gather all my experiences and solutions and will put them into a new thread to discuss. It will not be a rant, it will be guide like "how to resolve this, or that". And I will tell you how to construct such a maximal updatable subset of world, like my world-updatable file. My intention is to show how one can, with the help of such a file and a few techniques, overcome those seemingly impossible blockers on the way to a solution.

Maybe this will be a solution for you too. Stay tuned.

[segmentation-fault]

If you treat your X as Y, then sooner or later your will only have Y as X.

X: a noun,
Y: an adjective, or some construction with an adjective

...

It works, because in any interaction of X and Y, where Y needs X and X imposes some conditions on Y, there is (by the nature of "interaction", whatever this interaction might be) a "diffusion phenomenon". Ideas, behavior "flow" like heat. It's heat diffusion in networks. At the end, X and Y have the same "temperature". X and Y have "come together", they exchanged ideas and behavior as if it were heat - so now they think and behave similarly. I didn't check, but I would conjecture that this is the mechanism by which the species (the Y) adapt to their environment (the X) - evolutionary adaptation! - that's why I used the term natural selection to describe the way a system "chooses" its users.

This is what I called the Generalized Linus Postulate. I am not sure you understand the implications of this - they are far-reaching and, yes, even scary. So let me expand a bit:

The proof of the Postulate is based of these ideas:

• A network is a kind of graph.
• Heat diffusion in a network is heat diffusion on a graph.
• Ideas and behavior spread like heat on a network of individuals.

Therefore, ideas and behavior spread like heat on a graph. As a consequence, in the limit, a system "chooses" its users.

Let's see:

That a network is a kind of graph is easy to see. But what is "heat diffusion on a graph"? I suggest you search for some examples of heat diffusion on graph.

The interesting question is: do ideas and behavior really spread like heat on a network of individuals? This could be the subject of a PhD in sociology. The point is: do they spread like heat, or like waves - or are networks of individuals somehow "elastic" and behave as if the "edges" were made of rubber, where ideas just pose some kind of "strain"? Parabolic, hyperbolic, or elliptic?

I say ideas and behavior spread like heat, so they diffuse like heat on a graph. Now let's look at Diffusion on a Graph, on p.20:

In the long term, each node in the connected graph gets the same share of the dye (or whatever is diffusing), which is equal to the total amount initially present divided by the number of nodes.

That is, the system arrives a long-term equilibrium (the "steady state"), where ideas are homogeneously shared among the individuals (the "nodes"). In that state, no individual disagrees with another - they all share the same ideas, the same views!

That's how a system "chooses" its users.

Sounds familiar?

A developer who read my woes here might say:

"Dear S-F, I would love to implement what you are asking, but you are the only one who is asking for it! No one else feels the pressure for this! If I make a poll, the overwhelming majority of our users will say it is not interested. I am sorry, but I can't invest time on projects that are not on our users' wish list. "

...and he will be right! Gentoo has already chosen its users!

Don't be fooled. We are all trapped into "systems" that have chosen us as their "users".

• A university "chooses" its students.
• A firm "chooses" its employees.
• A neighborhood "chooses" its residents.
• A state "chooses" its citizens.

People that have "different heat conductivity" than the rest, i.e. people who are stubborn enough to refuse to accept the conditions, ideas and behavior imposed from a system on its users, these people leave the system sooner or later, in one way or another: they either really leave, or they live in some kind of "internal resignation" (for firms), or "internal exile", like monks or heremites (for states) - or they become "dissidents", with all implications.

This is the never-ending story of the battle between adaptation and mutation. Between the "Keep up or get left behind" and the "Give us the tools (the possibilities, the rights...), or you will fade into obscurity". A wise system knows when to enforce adaptation by all means - and when it's time to mutate.

[Leonardo.b]

There is a book by Richard Dawkins, about diffusion and evolution of ideas, compared to natural selection.
It was a good book, but I do not remember the title.

[pjp]

Maybe this will be a solution for you too. Stay tuned. :)

I look forward to it. I suspect our challenges are sufficiently different, but that doesn't mean I won't learn something or at least find it interesting.

My main point of friction is frequency and potential disruption due to a significant change.

Frequency could be alleviated by some sort of automatic compiling setup. I've never used one, and have no idea what would be appropriate without adding "enterprise" complexity.

A somewhat minor example of disruption would be the recent change to shadow / util-linux regarding su. Other than PAM being sufficiently convoluted such that it is itself an ongoing security concern, I don't have a strong opinion on using it. But the change does justify a need for more care during that update. I have no idea how to work around this type of issue. More importantly, they need to be known in advance of any automated build environment, creating a circular dependency for a solution to these issues.

A developer who read my woes here might say:

More importantly, would they reject code from a capable developer who wanted to add those features? A mechanism that could manifest packages soon to be unsupported and allow for external maintenance of that package might be a good start. Something like "This package claims whole or partial ownership of these files. Here's a tarball of those files which may be manually installed elsewhere, such as /usr/local."

A wise system knows when to enforce adaptation by all means - and when it's time to mutate.

A system itself is does not have the ability to gain or demonstrate wisdom. For better or worse, open source projects exist because of those who contribute code. Either that code comes from being paid to work on that task, or any given developer finds working on that task to be sufficiently rewarding. The choosing seems largely based on whether or not the individuals get along sufficiently well and whether or not they share enough values about the direction of the project. Most would probably agree that Portage doesn't need its own editor (vi, etc.) built-in. Given its support for acct-user and -group, it already has the basics for a configuration management tool :)

[figueroa]

pjp -- I shared the scripts that I use to semi-automate the update process on this thread:
viewtopic-t-1146998.html

The result is that I get an email each morning showing me what is available then to update after that morning's sync, which will also show any potential points of trouble (not very often). I don't automate the actual emerge/compile process, but when I do decide to pull the trigger, I just enter "emerge -uDU @world -a" so in less than a minute on a faster machine I can push the window into the background and get on with other work. These small bites, usually taken daily, are rarely a problem and requires maybe a total of three to five minutes of my actual time, mainly because I am a slow and careful worker.

On any subsequent machines (usually just my collocated server), I'll take half that time inside an ssh shell.

I almost never have to invest any notable time to fix system issues related to upgrades. The su USE flag change was a non-issue. I've done that on four machines and the last remote server is compiling the two related packages as I'm writing this during today's update which involves updating 30 packages, a week's worth of accumulated upgrades. I do this machine at least once a month these days, sometimes more or less weekly, depending on the rate of development during the preceding time frame.

Your mileage may vary, but my good experience is partly because I have invested a lot of effort in order to be well configured (also non-exotic and stable) in /etc/portage/make.conf and package.use, and especially /var/lib/portage/world.

[pjp]

Thanks, I'll definitely review them. I also avoid exotic as I neither need it nor want the hassle associated with it. (Edit: I'm not sure an email gains me anything. After a sync, I review output of an update command. Putting it in an email would mainly eliminate color. Still, I've made a note about it as something to consider.)

Other than some sort of deny list or watch list of packages that need closer inspection, I don't know what would address the su type of issue. While the updates went well for you, they've obviously not gone well for everyone. I prefer to avoid being in the latter category, so I'm perhaps overly cautions about updates. Some of that probably comes from the day job and not liking to pull all-nighters in the event something goes wrong.

[figueroa]

...
(Edit: I'm not sure an email gains me anything. After a sync, I review output of an update command. Putting it in an email would mainly eliminate color... .) ...

Doing the sync via crontab and emailing yourself the output of the subsequent emerge command takes zero effort beyond reading your own email. When the emerge -p output is only a few lines long and seldom includes any issues (thanks to frequent updates), color becomes nice to have fluff. The average number of packages in a daily update is somewhere around 4-5. I usually pull the trigger. Next day, rinse and repeat. Yawn! Wonderfully boring. Every few months I'll build and reboot to the latest stable kernel holding in the 5.10 series.

A few months ago (like maybe six) I had to revert an updated nfs-utils to the previous version while something was fixed upstream. That's about as exciting as it gets.

I'm doing things on the computer every day, so why not do the updates? It makes life easier. I'm definitely an evangelist for frequent updates. I have a few machines that I only update more or less monthly. They do fine. Monthly updates are then something like 50-100 packages. THAT makes me nervous, though it seems to always be fine, so I think monthly is fine too. Two or three months, then those updates start to become a big deal and a real drag. We live in the supercomputer age. Upstream development can be like a blur. But, I can't stop upstream, so I keep up instead.

[segmentation-fault]

Well, for one thing, rebooting is not an option for me, as it means to close 100s of open windows. Last time I checked, during a 300-day uptime period, I had more that 900 processes open. It usually takes me a week to find myself back at where I was, if I manage at all.

Yes, monthly updates are easy and weekly ones are easier and daily ones are a breeze and hourly - well, hourly updates are the hit!

But just because X-ly updates work for you, the system should not behave in a way that it practically says that ONLY X-ly updates are welcome - or else hell will break loose. That's when the system chooses its users - however, it seems the Gentoo ecosystem has chosen frequent updaters as its users already.

In a way you're saying it yourself:

Two or three months, then those updates start to become a big deal and a real drag.

so imagine it had gone without updates for one year and longer...Things must become better there, or users will be worn and will ultimately turn to alternatives (an equivalent formulation of the Linus Postulate).

[pjp]

...

We just have different philosophies. I'm not patching and rebooting every day, so the email would only be a nuisance. The system works for me, I don't work for it. When i want to do something on the computer, I want to do that, not be disrupted by updates. Forced disruptive updates is why I replaced Windows 11 with Gentoo.

If I see too big of an update, I grab a snapshot from 1/2 the time since my last sync. Rinse, repeat.

I look for a few things when doing updates. Delaying large packages (gcc, firefox, ?) until a more convenient time. Anything for which the update might cause problems accessing the system (ssh, pam, sudo, su, ...). And then anything that catches my eye. New packages stand out. Depending on my mood, I look into why there's Yet Another Dependency being added.

Although I very much appreciate early adopters. I've become aware of many gotchas over the years from people posting about them.

[figueroa]

segmentation-fault -- Your just a messy user (100 windows open). I imagine Gentoo must seem hopeless to you. But, who reboots. My desktops and servers run 24/7, but I do log out almost every night.

pjp -- Maybe you misunderstand. On most days my daily emerge email reads something like this:

Code: Select all

emerge -uDU @world -p

These are the packages that would be merged, in order:

Calculating dependencies  ...... done!
[ebuild     U  ] media-libs/mesa-21.3.5 [21.3.4] USE="egl%* gbm%*" 
[ebuild     U  ] net-libs/neon-0.32.2 [0.31.2] USE="expat* -test%" 

Yesterday there was one. On Sunday, there were none. In small bites like this, with everything else up-to-date, updates break things rarely; very rarely. Typically, breakage with portage happens due to infrequent updates leaving one's core system somewhat vulnerable and somewhat unstable for whatever equilibrium has been achieved by upstream.

That is all. Each makes their choices and rolls their own dice. I think my dice are loaded in my favor.

[pjp]

I think it was clear. The only time I can recall an update breaking something **knock on wood** is when upstream changes to X resulted in my mouse not working. That was sometime around 2008. The next time I used Gentoo as a "desktop" was around 2018.

I do see app-admin/needrestart and app-admin/restart-services. Apparently zypper can also identify services needing a restart. I might play around with that sometime. needrestart doesn't "ship with hooks" that support Gentoo, and restart-services is apparently by a Gentoo dev.

[figueroa]

...
I do see app-admin/needrestart and app-admin/restart-services. Apparently zypper can also identify services needing a restart. I might play around with that sometime. needrestart doesn't "ship with hooks" that support Gentoo, and restart-services is apparently by a Gentoo dev.

What services to restart hasn't been much of an issue here. I think when services need a restart that there should be an elog, but not not all developers/maintainers do that for us. But, I pretty much know anyway.

Please post any good or bad experiences you have with needrestart and restart-services, and maybe even zypper. I'm not using any of those now.

[pjp]

zypper is a SuSe tool. It came up when looking for needing to reboot. I generally prefer to reboot, though may put it off for a short time. Rather than "know" what needs restarting, I like the idea of something being able to tell me when I ask. Search results also suggest that might be determined by contents of /proc/<pid/maps, so I'm curious to learn more about how the concept works, and if there is an option that can work on BSD as well. It's not a short list priority, but I may include it in my mid-year objective for an improved update process.

[figueroa]

My computers run 24/7 and only run Gentoo. I only reboot for kernel upgrades about every 100 days or so. Kernel upgrades would be more often if I followed gentoo-sources stable without any restriction, but I mask gentoo-sources higher than the major version I'm sticking with, which is 5.10 (currently 5.10.93), which makes kernel upgrades routine and predictable. Since I do upgrade packages daily as a practical matter, it's easy to know when a service needs to restart or I need to log out and log back in (which is about as rare as needing to reboot).

I spend quite a bit of time in the forums participating with other users about their problems, but I rarely experience those problems.

On a tangent: My other current experience is maintaining the desktop computers at the school I support (very remote; over 500 miles to the north). I have them mostly migrated to MX-Linux which runs on a core of Debian stable. There, the updater, which runs "apt-get full-upgrade" automatically restarts services on the fly, and informs the user (me) in standard output of the need to reboot. I don't let the users do any updating, which prevents problems. I do all the updates in the evening or weekends. At the school, I maintain each users' installation right up to EOL, so re-installation of the OS may be required only once every 3 or 4 years, more or less. But, when I set up a new machine at the school, it will get the current version of MX-Linux, so currently some machines are running MX-18.3 and others MX-19.4 and two others are still running Mint till this summer and then those will get MX-21-~. Being the system administrator of these 10 (more or less) machines is very easy and the only problem machines every are two still running Mint, which is why I started migrating them to MX-Linux in the first place. The truth is, the users have no idea what's under the hood. They point and click and don't care.

I'm still running one headless server with Gentoo at the school. I don't see that changing.

About 12 years ago (don't hold me to that) I masked Python 3 when it first came out, and also GCC shortly thereafter, which prevented a lot of other upgrades. Eventually, I had dug that system into such a hole that upgrades that I needed to make could not be made without major surgery, and this mission-critical machine wasn't going to be able to be screwed around with in the middle of the school year. Ultimately, I upgraded it from a stage4 clone from a compatible machine I maintain at home. Reboot was uneventful and it only had to be off-line for about two minutes.

On all the machines I support at home or at school, I always set them up with at least one, usually three spare partitions set aside to be used in future full-system upgrades. All machines create nightly backups of users' files, plus weekly full-system backups (stage4 equivalent) to secondary hard drives, with additional copies to that gentoo server from which off-site encrypted backups are also created. Live without good backups is an anathema.

[pjp]

Also only Gentoo on my systems, 24/7, with 30-90 day updates / reboots, give or take. Although I'm hoping to go with a WOL build host and migrate some file serving to lower powered devices. For now, I have no better idea that RasberryPi's (the Zero seems interesting). If that goes well, I'd only have a single, power-hungry server for backups and large data.

The ability to have a lot of systems that are identical makes upgrading nice. You can generally update one and verify it, then do the others all at once. I've never directly supported that kind of environment. I've supported single / few similar systems, or servers within an Application "cluster" hat had minimal degrees of similarity. I've also only read about Unicorn environments where everything is automated and "easy" (I think those mostly describe "web-scale" companies. The easiest servers I recall having to upgrade were simple web servers).