Gentoo routine update/upgrade strategies?

[jamtat]

This query will relate to a Gentoo system I recently set up as a base on which to run MythTV. So this is a system that, once set up and running (as it is currently) is unlikely to have additional software added or have any use flag changes. I will hopefully only be performing software updates, and maybe rarely a kernel re-compile. I should mention that this system has fairly modest specs. I would be especially interested in hearing, from other Gentoo/MythTV users, feedback on the questions posed below.

I come mainly from an Arch/Debian (unstable) background, though I did run Gentoo for a brief period about a decade ago. I abandoned Gentoo fairly quickly since it was a bit too complex and demanding for my abilities at that time.

I later gravitated to Arch because of its rolling-release character and, once I'd come up with an update strategy that worked for my circumstances, things came together well. I initially had a bit of difficulty remembering to update/upgrade which, with a distro like Arch (and to some extent with with Debian unstable), can get you into trouble: the further your system gets out of date, the more difficult it becomes to upgrade. In fact, as I discovered, there comes a point with Arch where updating an out-of-date system demands about as much time and energy as doing an installation from scratch. I assume things are similar in the Gentoo world?

The update/upgrade strategy I came up with for Arch was fairly simple but has proved quite effective. First, I subscribed to the Arch news feed so as to keep track of major developments. Next I was able with some help to cobble together a script that would run the equivalent of emerge --sync, then run a command similar to emerge --update --ask --deep @world, in an xterm the script would open, awaiting user input. Using cron, I set that script to run that a few times a week and my update/upgrade woes largely disappeared. So this is a semi-automated approach that uses the computer to prompt me to do updates regularly but that doesn't actually update any software without my input/agreement.

I've just done my first manual Gentoo update/upgrade after having installed and set up the system a few months ago and things went fairly smoothly. Of course it takes much longer than an Arch update/upgrade, since you're only replacing binaries in that distro, whereas in Gentoo you're recompiling most of the outdated software on the system. Of course as with Arch, I prepared for this Gentoo upgrade by reading news items. I used a combination of directives I found on the wiki (https://wiki.gentoo.org/wiki/Handbook:X86/Working/Portage) and on-screen prompts to do this update/upgrade. That involved, incidentally, the additional step of running emerge --update --deep @world, then emerge --update --deep --with-bdeps=y @world (that latter took over 9 hours to complete on this system).

So I'm starting to think about future upgrades/updates and how to handle them on this Gentoo system. It would be wonderful if I could semi-automate the process as I've done with Arch. I've looked around a bit but have so far mainly turned up abandoned update/upgrade scripts for Gentoo (like this one from 2009 http://www.linux.com/community/blogs/130-distributions/12457 and this one from 2010 https://forums.gentoo.org/viewtopic-t-827398-start-0.html). So, can anyone point out to me a semi-automated utility of this sort for Gentoo?

Just as importantly, I need help determining how often I should update/upgrade this system. With Arch, a once-monthly update/upgrade is probably about as infrequently as you'd want to do this: I believe some even perform this task daily. The few-times-a-week scenario I use on my Arch system would obviously not be a reasonable strategy for my Gentoo/MythTV system. I would guess running emerge --sync on a weekly basis might be advisable and I do plan to set up a cron job to do something like that. But actually upgrading software on the system should be done a lot less frequently, I would think--maybe once per month or once every couple of months? Anyone have any feedback on that?

Then, there are various levels of updating/upgrading to be done on a Gentoo system: emerge --update --deep @world, I would think, would need to be run more frequently than emerge --update --deep --with-bdeps=y @world--correct? As an example, if emerge --update --deep @world were to be run once per month, perhaps emerge --update --deep --with-bdeps=y @world would then be run only once every three or maybe even six months. What are peoples' thoughts on these matters?

Then, of course, there are the additional commands that sometimes need to be run in connection with an update/upgrade, like emerge --depclean, emerge @preserved-rebuild, etc-update, and so on. Perhaps because of the complexity, then, semi-automating this process under Gentoo will not even be possible? I did just now find this http://weaver.gentooexperimental.org/update.html , which looks close to what I'm after. But the matter of frequency/infrequency of updating/upgrading the system does not seem to be addressed so, even if I were to use something like that, i would still want to get feedback on frequency.

Be that as it may, input on the questions of semi-automating the update/upgrade process, frequency of update/upgrades, and all related issues will be greatly appreciated, as will relevant advice of most any kind.

AFTERTHOUGHT: here's another related forum thread https://forums.gentoo.org/viewtopic-p-7674310.html

[ct85711]

[The Doctor]

Long post, but I think I can help with a few points, even though I don't run MythTV. First, Gentoo should be updated monthly. If you run something like KDE or GNOME that changes things quickly to change things, weekly. If you go too much past that you get people showing up on the forums requesting help.

Second, backups. There is an extremely good one, namely rsnapshot. The reason it is so good is it is configurable. It takes backups when you want them of the parts of the system you want them, and it only saves changes and hardlinks the rest. Naturally, this is a huge space saver if you have 30 copies of your /home directory! It is super easy to restore from, so you can be sure it is actually a backup, not a fancy way to use up an external drive or where ever you save your backups.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.

[jamtat]

Thanks for the input so far in this thread. I'm leaning toward a once-monthly, or perhaps once-every-two-months, semi-automated upgrade/update schedule. Looking over the scripts I'm finding, this one is the most comprehensible to my quite-rudimentary bash-scripting sensibilities:

[Section_8]

Sometimes automated updates like this can cause problems. One thing to watch out for is if an update emerges an updated kernel source, with the "symlink" use flag on, AND an updated package with a kernel module at the same time. This bit me one time.

So a single world update first emerged an updated gentoo-sources with USE=symlink (which updated the /usr/src/linux symlink) and then after that, an updated nvidia-drivers - which built against the just emerged gentoo-sources instead of the running kernel. I didn't catch that this had happened, & after the next reboot, nvidia-drivers failed & X wouldn't start until I installed the new kernel. After that, I turned off the symlink use flag.

[jamtat]

Thanks for your input, section_8. Learning about potential gotchas like the one you've described is part of the reason I wanted to inquire here on the forums about my desire to set up semi-automated system updates/upgrades on this installation.

[jamtat]

[ian.au]

Gentoo boxes need to be updated regularly in my experience, and it's much less work to do a bit of regular admin than sort out the results of leaving it too long.

Upgrading every two weeks or so by shelling in from some home device and running a few reasonably simple commands gets you a pretty robust and secure system. Install all the tools (sshd gentoolkit etc..) you have to read messages generated by installs, occasionally (admittedly more rarely these days) ensure your configs aren't going to be overwritten on etc-update/dispach-conf.

I don't think there is any way around a certain amount of time investment in any Gentoo box, the pay-off really occurs once the box is tuned to the task at hand, and that can take a while.

Scripting updates on old hardware seems hard and risky to me. I think it'd be bound to break something.

It takes less time to install and learn a couple of good, stable portage tools (eix, equery for sure) that will make life easier in the long run the portage universe a lot more human-readable. Kernel upgrades on older hardware, especially re graphics require *careful* consideration. On an old machine I would remain on a nice stable long-supported kernel and always scan the forums quickly to see if anyone is howling about breakage first, that saves grief as does always running emerge (and especially emerge --depclean) with the -pv switch and carefully reading the output before committing the job.

I think you need to consider setting up a crossdev environment to handle your upgrades on that hardware and really investigate building up a lean system (maybe from a seed). That would probably be a better investment in time savings overall than trying to automate nine-hour compiles on old hardware. With a really stripped down setup, you could push the updates out to monthly or more, but I've always found more often is better than too long between updates, it's just how Gentoo rolls.
Distcc: https://wiki.gentoo.org/wiki/Distcc/Cross-Compiling Kernel Seeds https://wiki.gentoo.org/wiki/Kernel/Configuration/Kernel_Seeds

My .02, can't help on mythTV

[jamtat]

Thanks for the further input, ian.au.

I've gone ahead an implemented part of a possible solution to the semi-automated update scheme I hope to set up on this machine. For now, that just involves a cron job that opens a new tmux tab/window on a weekly basis, doing there an emerge --sync, followed by an eselect news list.

Had a problem with the eselect news list part of that since, after that command runs, the tmux tab/window wants to immediately close. I resolved that by piping the eselect part through less, which ends up being kind of convenient since all I have to do is hit "q" to close the tab/window after I've looked at news item titles. It's inconvenient in that I can't actually read any news items using less. But if I see any headlines of interest I can always just issue eselect news list in a remaining open tmux tab/window for reading further. So this is a fairly serviceable step that takes me partway along the path to possibly semi-automating updates/upgrades for this system.

Meantime, I continue studying up on and asking about scripts I've found.

[cwr]

I wouldn't go more than a couple of weeks between updates - more than three months or so and a re-install
would be easier, or so I've found in the past. OTOH my desktop is running a 2010.0 install (which has moved
between various HDs and has no internet connection). It still does everything I want it to do.

My laptops are now rebuilt annually, with some scripts I've developed to ease the process, and it usually takes
a couple of days; one to build the system and one to configure it. That's as much time as I want to spend on
maintaining Gentoo, and it seems to work quite well.

Will

[jamtat]

Thanks for continuing interest and input in this thread. My most recent update/upgrade occurred almost exactly 3 months after my last update/upgrade. There was a minor glitch, but that seems to have gotten resolved by running emerge --update --deep --with-bdeps=y @world. But I don't intend to go that long between updates/upgrades in the future, so point taken on that matter, cwr.

One question I'd had about whether an upgrade could be segmented somehow was recently answered by someone who pointed out specifying "sets" to emerge. That individual specifies first the "system" set, then the "world" set, when doing upgrades, so as to break things up into more manageable units. That's nice to know about, but I'm not sure it would work so well for my intended semi-automated scheme.

That individual also recommended against automated updates/upgrades--not sure whether it was understood that what I'm after is really a semi-automated scheme. The difference between those two, as I've explained here, is that user input is required for any upgrading to really happen in the semi-automated scheme I'm contemplating. In this scheme, the computer initiates the process, thus serving the function of reminding that it's time to upgrade, but it will not complete the process unless the user gives the ok, the --ask switch ensuring that, as I envision it.

That individual did have a good point about taking some time to inspect the proposed upgrade, though. I thought of a way to make that happen as well. I could, for example, run the upgrade command with the --pretend switch a day or so before the scheduled upgrade and cat its output to a file. I could then cause the system to e-mail me that file so as to allow me to more carefully review what would be happening on the next upgrade.

One thing I'm still a bit unclear about though is the following. In the wiki, three upgrade commands are presented and explained: 1) emerge --update --ask @world, 2) emerge --update --deep @world, and 3) emerge --update --deep --with-bdeps=y @world. About the third command, the wiki says "it is recommended to run this command once in a while." The question I have is, given my hope of setting up a semi-automated upgrade scheme, which of the first two commands should I be running, say, once per month? I would think number two, since it's more comprehensive than number one. I'm not sure, then, why running number one would even be needed in such a scheme as I hope to implement. Can anyone offer feedback on that?

Instead, in my scheme I would probably wind up running emerge --update --deep --ask @world most regularly (maybe with emerge --update --deep --pretend @world the day before, as explained above). Then, every three months or so, I'd run emerge --update --deep --ask --with-bdeps=y @world. I've still got to work out how I'd introduce other commands that are often needed such as emerge --depclean, emerge @preserved-rebuild, etc-update, and so on.

Further input will be appreciated.

[Tony0945]

[trismo]

i prefer every 2 Weeks update, few min real work + compile time never had big problems with it the last few years on Stable
but if you run service that a available public weekly check of update maybe better and have a eye on Security database rss feed

[SiliconFiend]

Hi James,

You had asked me to comment about this so here goes. I also run MythTV on Gentoo, but I do not treat it like an appliance. I update every day, but just because I like to. Your proposed weekly maintenance is probably fine. I use eix so my maintenance is typically 1. sudo eix-sync (runs emerge --sync and then shows a list of updated packages) 2. sudo emerge --update --newuse --deep @world (shortened to emerge -uND @world), but only if eix indicates there is anything to update. I have EMERGE_DEFAULT_OPTS="--ask --verbose". I had never used --with-bdeps=y before, but now I will periodically to clean up those lingering packages that eix-test-obsolete shows me. I don't run revdep-rebuild as a matter of course, and newer versions of portage are much better at preventing that sort of breakage and will now tell you to run emerge @preserved-rebuild to link affected packages to the new library versions and clear out the old library versions. And regarding emerge --depclean, that's something I like to run manually every month or so just to clear out old cruft.

I run mostly stable packages with just a few keyworded as necessary. Even though I update quite frequently I'd rather let bleeding-edge testers find the problems (the WAF can suffer otherwise). But I think it's very helpful to update frequently so you can deal with any potential breakage in small increments instead of one big batch. My system was originally installed in late 2005 and although it's migrated to new hardware I've never needed to reinstall. I really appreciate that about Gentoo, especially after reading about the major headaches people using other distros have to go through when installing a new version. Generally the only reason I reboot is to install a new kernel, which I typically do as they are released to stable.

I don't know if my comments are helpful since I run a mostly manual update process, but maybe you might find my experience valuable.

Karl

[Tony0945]

[trismo]

ARCTIC Freezer under 8 cores use a model with 3 heatpipes
and 8 core cpu 4 heatpipes

i use ARCTIC Freezer A30 but is big check it

Dimensions 100 (L) x 139 (W) x 161 (H) mm

[Tony0945]

[C5ace]

I do a weekly semi automated update.

- open a terminal
- su to root

- cut and past text from a file "Update.txt" on my desktop into the Xterminal:

emerge --sync
emerge --update --deep --with-bdeps=y --newuse --backtrack=300 --keep-going=y @world -va

- check and if necessary read news, add and/or change use flags and keywords.
- if no change required, hit Enter

- else
emerge --update --deep --with-bdeps=y --newuse --backtrack=300 --keep-going=y @world -va
- Hit enter

- cut and past text from a file "Update.txt" on my desktop into the Xterminal:
emerge --depclean
revdep-rebuild

- DONE

Maybe one of these days I will create a script.

[Tony0945]

[Buffoon]

Upgrading Gentoo in routine manner, say once a week is not a time consuming task, I run it in screen session.

1. Run emerge-webrsync as a cronjob (followed by eix-update if you use eix), mine runs every day 0608 for whole network - my portage is on NFS (your time spent - 0);
2. run your emerge world update command and minimize the terminal window or put it in background (your time spent - 5 seconds);
3. do whatever you need to do and check back in a while, you should see emerge asking your Y for upgrades, if everything is as you like hit CR (your time spent - usually less than a minute);
4. some time later check back to see if there were any problems, if no log off (your time spent - usually 5 seconds).

See, in most cases you spend about a minute of your time on upgrading and you are doing it interactively, this is the only safe way to do it.

[ct85711]

I've been doing a similar routine as your Buffon, for me, I sync and update on the weekend once a week, otherwise I use -pv instead of -a so my involvement is a little more, but still very little over all.

[jamtat]

[jamtat]

Below I'll describe the semi-automated update/upgrade scheme I've come up with for this Gentoo box. It's a bit less semi-automated than what I set up for my Arch system, but that's partly because Gentoo is a distro that merits a bit more user involvement. It should nonetheless prove workable for me and should serve its intended function of prompting me to upgrade on a regular basis. I've done a bit of testing and things look like they'll work, but until I fully implement this scheme I won't know whether I've got everything configured correctly.

Mainly I'm using cron jobs, most of which will pop up new tmux tabs/windows. One cron job will run weekly and do a sync as well as show me news headlines. There will be 6 monthly cron jobs: two of those save, to a file, output from the impending update/upgrade, and two others will mail me that output, keeping me apprised of what is to be done in the upcoming upgrade. The other two will run either emerge --update --deep --ask @world or emerge --update --deep --ask --with-bdeps=y @world, depending on the month (the latter in place of the former every four months). I won't be trying to automate the further steps of running things like emerge --depclean, emerge @preserved-rebuild, etc-update, and so on. Instead, since the tmux tab/window where the update/upgrade commands have run will remain open, I will simply follow on-screen instructions I will see in those tabs/windows, after the update/upgrade has completed (that latter being the less automated aspect of this scheme).

Below I'll include the cron jobs I intend to set up. First, the weekly sync/news cron job:

[Roman_Gruber]

after fast reading this topic..

i try to upgrade this box on a daily basis

the other box when i need it, 4-8 weeks.

a weekly update routine in the past was a good cycle but you keep using packages with security risks when you do update once a month.

and best way to upgrade is to let portage run in the background and watch the output

emerge --keep-going ... keeps portage going, thats very essential flag to use.

and there are so many blockers in past 6 months taht I highly doubt that you can automate portage and rely that these blockers are solved themself. architecute changes / philosophy changes regarding ffmpeg / libav ... perl upgrades, / poppler / raptor / liborcus ... and so on

I wish you the best that you can generate a script and cut down the regular tasks but I highly doubt in my expierence.

[Buffoon]

If the problem is remembering let the box email you every time the weekly sync runs. I have a local mail server in the router for this purpose. It can even receive outside mail. It's fun actually, you can create as many email addresses you want and you can ban anybody you want. Say, you find an one time deal on the net, you get the item and of course they start spamming you, all cheap stores are spammers. Here's where the ban hammer comes in.