| View previous topic :: View next topic |
| Author |
Message |
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1993
|
 Posted: Wed Mar 03, 2021 12:39 pm Post subject: |
 |
|
Updated 2021/03/14
Here's mine to grab the latest snapshot, using the "current" symlink, fetching both the current snapshot and the sha512sum.txt file from the same server, to avoid problems with pooled servers out of sync. It no longer maintains its own timestamp.
It keeps the synchronized snapshot and the current sha512sum.txt file in "/var/db/snapshot/".
It checks the validity of the sha512sum.txt file, and that the sha512sum for the snapshot agrees with that expected.
(The validity check demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.)
It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It no longer writes messages to syslog.
If the snapshot was mounted before you invoke the script, it gets remounted after it runs, and it prints the timestamp from its Metadata file.
It works with dash, and so it should work with any shell worthy of note.
Enjoy.
| Code: |
#!/bin/sh
### Update a squashfs portage snapshot from an rsync mirror
# Version 0.2 Paul Gover 2021/3/14
set -u -e # Catch typos and unchecked commands
# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync.uk.gentoo.org"
SYNCDIR="gentoo/snapshots/squashfs"
# Defintions for what and where to store the snapshot. Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
TYPE="lzo.sqfs"
NAME="gentoo-current"
NDIR="$DIR.new"
ODIR="$DIR.old"
### Utility functions
# Attempt a command, but don't exit just because it failed
tryto() {
$@ || echo "Command $* failed - continuing."
}
# Issue an error message to stderr, tidy up, then exit
die() {
echo "Error: %s\n" "$*" >& 2
tryto rm -r "$NDIR"
exit 1
}
# Run a command. If if fails, exit with an apprpriate error message
command() {
$@ || die "Command $* failed - exiting."
}
# Swap new for current directories safely
update() {
[ -e "$ODIR" ] && command rm -r "$ODIR"
command mv "$DIR" "$ODIR"
command mv "$NDIR" "$DIR"
command rm -r "$ODIR"
}
### Mainline code
tryto mkdir "$NDIR" # Assume fails means NDIR is left from before.
tryto cp -a "$DIR/*" "$NDIR" # Assume fail means DIR is not yet populated - i.e. this is first run
command rsync --copy-links --verbose --update "$SYNCURI::$SYNCDIR/$SUMS" "::$SYNCDIR/$NAME.$TYPE" "$NDIR"
command gpg --verify "$NDIR/$SUMS"
cd "$NDIR" # sha512sum --check has to be run from the same relative directory used to create the checksums
command sha512sum --check --ignore-missing --status "$SUMS"
if mountpoint -dq "$PORTDIR"
then
command umount "$PORTDIR"
update
command mount "$PORTDIR"
grep -F "TIMESTAMP" "$PORTDIR/Manifest"
else
update
fi
chgrp -R portage /var/db/snapshot
|
_________________
Greybeard
Last edited by Goverp on Sun Mar 14, 2021 11:36 am; edited 2 times in total |
|
| Back to top |
|
 |
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1993
|
| Posted: Thu Mar 04, 2021 1:01 pm Post subject: Problems with gentoo rsync mirrors |
|
|
OK, now I can see what's going wrong. It's not the scripts, it's not the files themselves, it's the rsync server pools being inconsistent.
My script stopped this morning with the message:
| Code: | | No newer snapshot on rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashf than 20210302. Sync aborted |
which was odd, as it created that yesterday.
Thinking it was a problem with rsync, I tried wget instead. That got gentoo-20210303.lto.sqfs, but its checksum was not in sha512sum.txt, which only held entries up to 20210302. Digging shows that the UK Bytemark mirror has the files up to 3rd March, whereas the others in the UK pool only go up to 2nd March. rsyncing twice, once to get the checksums to see if there was any point in doing an update, and then separately for the gentoo-current file, depends on which server it gets each time.
Possible fixes:
(a) rsync both the sha512sum.txt and gentoo-current.lto.sqfs files (use --existing on the rsync, or use a list of filenames)
(b) use "rsync --debug=connect2", which gives the IP address used on the first sync, then use that instead of the pool name on the second sync. Hopefully the files are consistent on the same server.
(c) somehow scan the pool to find which mirror in the pool has the most up-to-date data, and again use it's IP address. Except I don't know how to do this! Can anyone help?
(d) use an rsync server nearer the gentoo infrastructure master, but that's more than a little against the spirit of mirroring servers.
Note that all this means that any sync method may get inconsistent results if the mirrors are out of date. That's obvious, but it might mean that downloading the tarball and then syncing might not do what you hoped.
An aside: my script ought to include a "gpg --verify sha512sum.txt". Interestingly, that demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.
_________________
Greybeard |
|
| Back to top |
|
|
user
Apprentice
Joined: 08 Feb 2004
Posts: 200
|
| Posted: Thu Mar 04, 2021 1:17 pm Post subject: |
|
|
cron job like
- download if changed
- verify gpg checksum file
- verify checksum
- umount
- replace squashfs file
- mount
| Code: | bash -c 'umask 022 && \
cd /var/cache/portage/squashfs/ && \
wget -q --secure-protocol=PFS --timestamping https://<<your mirror>>/gentoo/snapshots/squashfs/{gentoo-current.xz.sqfs,sha512sum.txt} && \
gpg --verify sha512sum.txt && \
sha512sum --check --ignore-missing sha512sum.txt && \
sudo umount /var/db/repos/gentoo && \
cp -p gentoo-current.xz.sqfs gentoo-portage.xz.sqfs && \
sudo mount /var/db/repos/gentoo'
|
fstab entry
| Code: |
/var/cache/portage/squashfs/gentoo-portage.xz.sqfs /var/db/repos/gentoo squashfs auto,ro,loop,nodev,noexec,nosuid 0 0
|
|
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6097
Location: Dallas area
|
| Posted: Thu Mar 04, 2021 1:23 pm Post subject: |
|
|
I noticed the inconsistent results from the squashfs directory, then I swapped a dir up (snapshots) and it was still inconsistent from one day to the next.
So now, I'm back to rsync portage and then create my own squashfs (~4 seconds using 16 cores).
Note: I'm not using emerge --sync, but directly rsync'ng against a mirror and then run gemato separately (better control over the process)
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
| Back to top |
|
|
Leonardo.b
Apprentice
Joined: 10 Oct 2020
Posts: 294
|
| Posted: Mon Mar 08, 2021 1:12 am Post subject: |
|
|
A note about the use of cron jobs, I just realized.
I had the sync job running nightime, and the system upgrades too.
I guess it may be a problem if the cron job remounts /usr/portage while Portage is doing an upgrade.
Maybe nothing bad would happen, but I changed the sync time because I don't want to know it's not. |
|
| Back to top |
|
|
elover
Apprentice
Joined: 20 Nov 2019
Posts: 159
Location: Spain
|
| Posted: Sat Mar 13, 2021 2:32 pm Post subject: |
|
|
| Goverp wrote: | And here's mine to grab the latest snapshot, ignoring the "current" symlink.
It keeps the synchronized snapshot, the current sha512sum.txt file, and the timestamp in a file, in "/var/db/snapshot/
if the latest snapshot on the rsync sha512sum.txt is no later than the timestamp, it skips further processing.
It checks the sha512sum for the snapshot agrees with that expected.
It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It's set to write a few messages to syslog, but you can turn that off.
It works with dash, and so it should work with any shell worthy of note.
Enjoy.
| Code: | #!/bin/sh
### Update a squashfs portage snapshot from an rsync mirror
set -u -e # Catch typos and unchecked commands
# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs"
# Defintions for what and where to store the snapshot. Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
PREFIX="gentoo-"
TYPE="lzo.sqfs"
NAME="gentoo-snapshot"
STAMP="timestamp"
# If Gentoo change the filename convention from "gentoo-YYYYMMDD.foo", we're SCREWED
PATTERN="${PREFIX}[0-9]{8}[.]$TYPE" # Strictly we should escape any .'s in TYPE, but why bother?
# Set non-null to write messages to syslog
LOG="log"
NDIR="$DIR.new"
ODIR="$DIR.old"
### utility functions
# Issue and log a message; first parameter is log facility.level
say () {
[ "$LOG" ] && logger --priority "user.info" "squashsync: $*"
printf "%s\n" "$*"
}
warn() {
[ "$LOG" ] && logger --priority "user.warn" "squashsync: $*"
printf "Warning: %s\n" "$*"
}
# Attempt a command, but don't exit just because it failed
tryto() {
$@ || warn "Command $* failed - continuing."
}
# Issue an error message to stderr, tidy up, then exit
die() {
[ "$LOG" ] && logger --priority "user.err" "$*"
printf "Error: %s\n" "$*" >& 2
tryto rm -r "$NDIR"
exit 1
}
# Run a command. If if fails, exit with an apprpriate error message
command() {
$@ || die "Command $* failed - exiting."
}
# Application functions
# Print the checksum and date for the most recently dated record from stdin (an sha512sum.txt file)
# The parameter is the date of the previous snapshot already downloaded
# If no records are later than that, prints a null string
latestsum() {
local line date lline ldate
lline=""
ldate="$1"
while read -r line
do
# Keep only the latest date and line
date="${line#*$PREFIX}"
date="${date%%.*}"
if [ "$date" -gt "$ldate" ]
then
ldate="$date"
lline="$line"
fi
done
printf "%s" "$lline"
}
# Swap new for current directories safely
update() {
[ -e "$ODIR" ] && command rm -r "$ODIR"
command mv "$DIR" "$ODIR"
command mv "$NDIR" "$DIR"
command rm -r "$ODIR"
}
### Mainline code
say "Starting portage tree snapshot update with $SYNCURI"
# Get the date of the current snapshot from our timestamp file (if any, otherwise use the epoch start).
cdate=$(cat "$DIR/$STAMP")
[ "$cdate" ] || cdate="19700101"
# Sync a new SUMS file into the NDIR. Start with a copy, so the original remains if thing go wrong
tryto mkdir "$NDIR" # Assume fails means NDIR is left from before.
tryto cp "$DIR/$SUMS" "$NDIR/$SUMS" # Assume fail means DIR is not yet populated - i.e. this is first run
command rsync --verbose "$SYNCURI/$SUMS" "$NDIR/$SUMS"
# Ignore the line for the new "current" snapshot; it tends to be out of date and hence with the wrong checksum
# Instead, grep SUMS for the right type and select the line with the latest date after the current snapshot
line=$(grep -E -o "[[:xdigit:]]+[ *]+$PATTERN" "$NDIR/$SUMS" | latestsum $cdate)
[ "$line" ] || die "No newer snapshot on $SYNCURI than $cdate. Sync aborted"
nsum="${line%%[ *]*}"
nfile="${line##*[ *]}"
ndate="${line#*$PREFIX}"
ndate="${ndate%%.*}"
# OK, according to SUMS there's a later snapshot. Sync a new copy into NDIR.
target="$NAME.$TYPE"
tryto cp "$DIR/$target" "$NDIR/$target"
command rsync --verbose "$SYNCURI/$nfile" "$NDIR/$target"
# Check the sum is correct
shasum=$(sha512sum "$NDIR/$target")
shasum="${shasum%%[ *]*}"
if [ -z "$shasum" ]
then die "Unable to calculate sha512sum for current snapshot $NDIR/$target"
elif [ "$shasum" != "$nsum" ]
then die "Checksum $shasum for $NDIR/$target differs from $N$SYNCURI/$nfile."
else
printf "%s" "$ndate" > "$NDIR/$STAMP" # Create the timestamp file
# Replace the current snapshot with the new. If portage tree is mounted, unmount and remount it
if mountpoint -dq "$PORTDIR"
then
command umount "$PORTDIR"
update
command mount "$PORTDIR"
else
update
fi
say "Successful sync to portage tree snapshot dated $ndate."
fi
|
|
Hi, great job
I am a normal user who does not understand scripts. (Sorry)
I get this error when I run it:
| Code: | sudo sh /home/elover/Documentos/actualizar.sh ─╯
Starting portage tree snapshot update with rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/sha512sum.txt': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/sha512sum.txt /var/db/snapshot.new/sha512sum.txt failed - continuing.
-========== B Y T E M A R K H O S T I N G M I R R O R ==========-
sha512sum.txt
sent 43 bytes received 29,716 bytes 19,839.33 bytes/sec
total size is 29,627 speedup is 1.00
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/gentoo-snapshot.lzo.sqfs': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/gentoo-snapshot.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - continuing.
rsync: link_stat "/snapshots/squashfs/gentoo-20210312.lzo.sqfs" (in gentoo) failed: No such file or directory (2)
sent 8 bytes received 131 bytes 278.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1816) [Receiver=3.2.3]
Error: Command rsync --verbose rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/gentoo-20210312.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - exiting. |
I guess then you put it in cron, how do you configure it.
Sorry for asking so much, but my SSD has already died. |
|
| Back to top |
|
|
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1993
|
| Posted: Sun Mar 14, 2021 9:30 am Post subject: |
|
|
elover,
Ah, sorry, you're being caught by the same problem described in previous posts on this thread. That script can go wrong if you are using an rsync URI for a server pool (and most of those listed on the Gentoo download pages are pooled). The script gets the sha512sum.txt file (whiich lists the available snapshots) and chooses the latest, but when it goes back to get the snapshot, it gets a different server which is out of date from the pool. (Why some of the pool's servers are a day out of date is probably down to the various mirror service providers, and may be beyond the power of the internet to get it fixed everywhere.)
I have a new version of the script that's a lot simpler, and gets the sha512sum.txt file and the current snapshot in one operation, so it will use the same server and get a consistent result. I'm on the wrong PC at the moment, but I'll post the new version this morning (UK time).
_________________
Greybeard |
|
| Back to top |
|
|
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1993
|
| Posted: Sun Mar 14, 2021 11:35 am Post subject: |
|
|
OK, I've updated my script. I edited the post containing the script, rather than leave the broken one in the thread.
_________________
Greybeard |
|
| Back to top |
|
|
geki
Advocate
Joined: 13 May 2004
Posts: 2387
Location: Germania
|
| Posted: Sun Mar 14, 2021 12:02 pm Post subject: |
|
|
JFYI, I recommend you to check your local university FTP site for a gentoo rsync mirror. In germany, we got plenty of them. No more out-of-date pools. Like so:
https://<your favorite FTP site>/gentoo/ or https://<your favorite FTP site>/pub/gentoo/
I use, for example, https://ftp.fau.de/gentoo/. But there are a dozen others in germany, too.
Goverp
Maybe better to add a config variable to be set by the user of the script.
_________________
hear hear |
|
| Back to top |
|
|
Goverp
Veteran
Joined: 07 Mar 2007
Posts: 1993
|
| Posted: Sun Mar 14, 2021 2:03 pm Post subject: |
|
|
| geki wrote: | ...
Goverp
Maybe better to add a config variable to be set by the user of the script. |
That's SYNCURI, unless I'm misunderstanding you.
_________________
Greybeard |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
