Nighly emerge, best practice?

[pgu]

I could not find this in the FAQ even though it probably pops up from time to time.

What is the best practice when it comes to nightly (or regular) updates from cron?

Is there a way to get a parser friendly (e.g. json) output where different levels of severity are tagged?

Also is it possible to set up a cache which can be used for syncing on the local network to reduce the load on the upstream?

[freke]

Look here for setting up a local portage-mirror - https://wiki.gentoo.org/wiki/Local_Mirror

Also if your systems is quite similar, you could consider using a binhost - https://wiki.gentoo.org/wiki/Binary_package_guide
I do that for 5 similar small headless servers.

I'm using something like

Code: Select all

PORTAGE_ELOG_SYSTEM="save mail"
PORTAGE_ELOG_CLASSES="warn error log qa"
PORTAGE_ELOG_MAILURI="<receiver> <account>:<password>@<mailserver>:<port>"
PORTAGE_ELOG_MAILFROM="<sender>"

in /etc/portage/make.conf - I *think* it can be used without app-portage/elogv

But I'm not letting cron doing the actual emerges, though - I want to audit what will happen myself.

[Leonardo.b]

Automatic upgrades from cron are risky: they may fail, leaving the system in a broken state.

Sharing some directories over NFS is the easiest way to share files in a local network.

[pgu]

Thanks. The systems are somewhat different: headless servers, desktop and embedded headless systems. That's why I was thinking of some kind of cache which could pick whatever it had in the cache which is common.

I do use NFS. But what port of the three should be shared over NFS? Both /etc/portage and /var/lib/portage will contain unique settings for each system.

If it's not safe to do emerge sync/world updates from cron it would have been nice to do a pretend type of update to get any warnings (sorted by severity) up front and then commit the update manually.

[freke]

Emerge sync should be quite safe (and you could setup a local mirror for that, so you only sync upstream once).
You could do an emerge in cron with --pretend/-p option and get that output mailed to you - to verify before doing an actual manual emerge.

[figueroa]

Emerge sync should be quite safe (and you could setup a local mirror for that, so you only sync upstream once).
You could do an emerge in cron with --pretend/-p option and get that output mailed to you - to verify before doing an actual manual emerge.

That's exactly what I do. The output of sync and update -p are both mailed to me from both server and clients.

On the server "jeremiah" at 0600 local from root's crontab I run the scrip emerge.scr:

Code: Select all

#!/bin/sh
emerge --sync
echo "emerge -uDU world -p" > /home/USERNAME/bin/emergeauto.txt
emerge -uDU world -p >> /home/USERNAME/bin/emergeauto.txt 2>&1

Then, at 0650 local from the user's crontab I run the script mailx.scr:

Code: Select all

#!/bin/sh
mail -s "jeremiah emerge output" USERNAME < /home/USERNAME/bin/emergeauto.txt

My /usr/bin/mail is from mail-client/mailx. On this box I'm running a full mail server with postfix, courier, and other miscellaneous supporting packages.

Roughly an hour later, the equivalent scripts can and do run on the primary client, my main desktop machine "bethel" with the client getting it's portage tree from the server "jeremiah." On other clients, I run the scripts manually.

Almost daily on "jeremiah" and "bethel" I'll manually run "emerge -uDU @world -a" after reading the output that I'll look at with my first cup of coffee. I type well, and the update command flows out of muscle memory, but I LOOK at it before pressing enter to be sure I don't make a mistake. I don't watch the terminal that is running the update command, but I will look in at it from time-to-time and I will read any elog messages before considering myself done. Takes me maybe two minutes of my time, and errors and exceptions are rare these days.

[Leonardo.b]

I do use NFS. But what port of the three should be shared over NFS?

The ::gentoo repository, as an alternative to a local mirror.

[forrestfunk81]

You could use the --buildpkgonly or -B option for a save nightly compile.

Creates binary packages for all ebuilds processed without actually merging the packages. This comes with the caveat that all build-time dependencies must already be
emerged on the system.

Then you only have to install the binary packages and update configs whenever you have time for the actual updates.

Something like this:

Code: Select all

emerge -uDUBv world
emerge @preserved-rebuild --usepkg=n -B
eclean-pkg
eclean-dist -d

[pgu]

Thanks for the input.

It could be nice to have an output mode of emerge which is suitable for parsing (e.g. json) and stable across different revisions so a program could parse it and try to determine if it's safe to update and present this as e-mail (like you do) or render a html file for a web server or similar. Also any problem could be looked up in an error database (or in a section on this forum) so any potential issues could be resolved across all gentoo users using the same solution.

[marduk]

I don't do nightly builds from cron, but I do regular (more than nightly) builds from CICD (Jenkins) and export ("publish") the binpkgs (and the exact portage tree used to build them) so that my "real" machines can use them. This is done on another system (actually in a container) and for all different types of machines (machine types). I've been doing this or something similar for years and for me it has worked out pretty well. I call this setup "Gentoo Build Publisher" and you can read about it here.

I'm curious if this method might work for others as well.