Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Discussion & Documentation Documentation, Tips & Tricks
  • Search

The really simple way to use the portage tree on squashfs ?

Unofficial documentation for various parts of Gentoo Linux. Note: This is not a support forum.
Post Reply
Advanced search
41 posts
  • Previous
  • 1
  • 2
Author
Message
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Wed Mar 03, 2021 12:39 pm

Updated 2021/03/14

Here's mine to grab the latest snapshot, using the "current" symlink, fetching both the current snapshot and the sha512sum.txt file from the same server, to avoid problems with pooled servers out of sync. It no longer maintains its own timestamp.

It keeps the synchronized snapshot and the current sha512sum.txt file in "/var/db/snapshot/".
It checks the validity of the sha512sum.txt file, and that the sha512sum for the snapshot agrees with that expected.
(The validity check demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.)

It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It no longer writes messages to syslog.
If the snapshot was mounted before you invoke the script, it gets remounted after it runs, and it prints the timestamp from its Metadata file.
It works with dash, and so it should work with any shell worthy of note.

Enjoy.

Code: Select all

#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror
# Version 0.2 Paul Gover 2021/3/14

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync.uk.gentoo.org"
SYNCDIR="gentoo/snapshots/squashfs"

# Defintions for what and where to store the snapshot.  Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
TYPE="lzo.sqfs"
NAME="gentoo-current"

NDIR="$DIR.new"
ODIR="$DIR.old"

### Utility functions

# Attempt a command, but don't exit just because it failed
tryto() {
        $@ || echo "Command $* failed - continuing."
}

# Issue an error message to stderr, tidy up, then exit
die() {
        echo "Error: %s\n" "$*" >& 2
        tryto rm -r "$NDIR"
        exit 1
}

# Run a command.  If if fails, exit with an apprpriate error message
command() {
        $@ || die "Command $* failed - exiting."
}

# Swap new for current directories safely
update() {
        [ -e "$ODIR" ] && command rm -r "$ODIR"
        command mv "$DIR" "$ODIR"
        command mv "$NDIR" "$DIR"
        command rm -r "$ODIR"
}

### Mainline code

tryto mkdir "$NDIR"                     # Assume fails means NDIR is left from before.
tryto cp  -a "$DIR/*" "$NDIR"           # Assume fail means DIR is not yet populated - i.e. this is first run

command rsync --copy-links --verbose --update "$SYNCURI::$SYNCDIR/$SUMS" "::$SYNCDIR/$NAME.$TYPE" "$NDIR"
command gpg --verify "$NDIR/$SUMS"

cd "$NDIR"                              # sha512sum --check has to be run from the same relative directory used to create the checksums
command sha512sum --check --ignore-missing --status "$SUMS"

if mountpoint -dq "$PORTDIR"
then
        command umount "$PORTDIR"
        update
        command mount "$PORTDIR"
        grep -F "TIMESTAMP" "$PORTDIR/Manifest"
else
        update
fi

chgrp -R portage /var/db/snapshot
Last edited by Goverp on Sun Mar 14, 2021 11:36 am, edited 2 times in total.
Greybeard
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

Problems with gentoo rsync mirrors

  • Quote

Post by Goverp » Thu Mar 04, 2021 1:01 pm

OK, now I can see what's going wrong. It's not the scripts, it's not the files themselves, it's the rsync server pools being inconsistent.

My script stopped this morning with the message:

Code: Select all

No newer snapshot on rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashf than 20210302.  Sync aborted
which was odd, as it created that yesterday.
Thinking it was a problem with rsync, I tried wget instead. That got gentoo-20210303.lto.sqfs, but its checksum was not in sha512sum.txt, which only held entries up to 20210302. Digging shows that the UK Bytemark mirror has the files up to 3rd March, whereas the others in the UK pool only go up to 2nd March. rsyncing twice, once to get the checksums to see if there was any point in doing an update, and then separately for the gentoo-current file, depends on which server it gets each time.

Possible fixes:
(a) rsync both the sha512sum.txt and gentoo-current.lto.sqfs files (use --existing on the rsync, or use a list of filenames)
(b) use "rsync --debug=connect2", which gives the IP address used on the first sync, then use that instead of the pool name on the second sync. Hopefully the files are consistent on the same server.
(c) somehow scan the pool to find which mirror in the pool has the most up-to-date data, and again use it's IP address. Except I don't know how to do this! Can anyone help?
(d) use an rsync server nearer the gentoo infrastructure master, but that's more than a little against the spirit of mirroring servers.

Note that all this means that any sync method may get inconsistent results if the mirrors are out of date. That's obvious, but it might mean that downloading the tarball and then syncing might not do what you hoped.

An aside: my script ought to include a "gpg --verify sha512sum.txt". Interestingly, that demands what I think is an obsolete key "Gentoo ebuild repository signing key (Automated Signing Key) <infrastructure@gentoo.org>". It's no longer listed amongst the signing keys, but Googling shows it was used in 2018. It's also the key used for the taballs in gentoo/snapshots.
Greybeard
Top
user
Apprentice
Apprentice
Posts: 246
Joined: Sun Feb 08, 2004 4:57 pm

  • Quote

Post by user » Thu Mar 04, 2021 1:17 pm

cron job like
- download if changed
- verify gpg checksum file
- verify checksum
- umount
- replace squashfs file
- mount

Code: Select all

bash -c 'umask 022 && \
cd /var/cache/portage/squashfs/ && \
wget -q --secure-protocol=PFS --timestamping https://<<your mirror>>/gentoo/snapshots/squashfs/{gentoo-current.xz.sqfs,sha512sum.txt} && \
gpg --verify sha512sum.txt && \
sha512sum --check --ignore-missing sha512sum.txt && \
sudo umount /var/db/repos/gentoo && \
cp -p gentoo-current.xz.sqfs gentoo-portage.xz.sqfs && \
sudo mount /var/db/repos/gentoo'
fstab entry

Code: Select all

/var/cache/portage/squashfs/gentoo-portage.xz.sqfs              /var/db/repos/gentoo squashfs auto,ro,loop,nodev,noexec,nosuid 0 0
Top
Anon-E-moose
Watchman
Watchman
User avatar
Posts: 6566
Joined: Fri May 23, 2008 7:31 pm
Location: Dallas area

  • Quote

Post by Anon-E-moose » Thu Mar 04, 2021 1:23 pm

I noticed the inconsistent results from the squashfs directory, then I swapped a dir up (snapshots) and it was still inconsistent from one day to the next.
So now, I'm back to rsync portage and then create my own squashfs (~4 seconds using 16 cores).
Note: I'm not using emerge --sync, but directly rsync'ng against a mirror and then run gemato separately (better control over the process)
UM780 xtx, 6.18 zen kernel, gcc 15, openrc, wayland
minixforum m1-s1 max -- same software as above but used for ai learning


Zealots are gonna be zealots, just like haters are gonna be haters
Top
Leonardo.b
Guru
Guru
Posts: 314
Joined: Sat Oct 10, 2020 9:24 pm

  • Quote

Post by Leonardo.b » Mon Mar 08, 2021 1:12 am

A note about the use of cron jobs, I just realized.

I had the sync job running nightime, and the system upgrades too.
I guess it may be a problem if the cron job remounts /usr/portage while Portage is doing an upgrade.
Maybe nothing bad would happen, but I changed the sync time because I don't want to know it's not.
Top
elover
Apprentice
Apprentice
Posts: 181
Joined: Wed Nov 20, 2019 12:32 pm
Location: Spain

  • Quote

Post by elover » Sat Mar 13, 2021 2:32 pm

Goverp wrote:And here's mine to grab the latest snapshot, ignoring the "current" symlink.
It keeps the synchronized snapshot, the current sha512sum.txt file, and the timestamp in a file, in "/var/db/snapshot/
if the latest snapshot on the rsync sha512sum.txt is no later than the timestamp, it skips further processing.
It checks the sha512sum for the snapshot agrees with that expected.
It's tailorable for file names, directories, and whether you want lzo or xz compressed snapshots.
It's set to write a few messages to syslog, but you can turn that off.
It works with dash, and so it should work with any shell worthy of note.

Enjoy.

Code: Select all

#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
PORTDIR="/var/db/repos/gentoo"
SYNCURI="rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs"

# Defintions for what and where to store the snapshot.  Note that rsync mirrors offer a choice of .lzo and .xz compressed SHOTs.
DIR="/var/db/snapshot"
SUMS="sha512sum.txt"
PREFIX="gentoo-"
TYPE="lzo.sqfs"
NAME="gentoo-snapshot"
STAMP="timestamp"

# If Gentoo change the filename convention from "gentoo-YYYYMMDD.foo", we're SCREWED
PATTERN="${PREFIX}[0-9]{8}[.]$TYPE"     # Strictly we should escape any .'s in TYPE, but why bother?

# Set non-null to write messages to syslog
LOG="log"

NDIR="$DIR.new"
ODIR="$DIR.old"

### utility functions

# Issue and log a message; first parameter is log facility.level
say () {
        [ "$LOG" ] && logger --priority "user.info" "squashsync: $*"
        printf "%s\n" "$*"
}

warn() {
        [ "$LOG" ] && logger --priority "user.warn" "squashsync: $*"
        printf "Warning: %s\n" "$*"

}

# Attempt a command, but don't exit just because it failed
tryto() {
        $@ || warn "Command $* failed - continuing."
}

# Issue an error message to stderr, tidy up, then exit
die() {
        [ "$LOG" ] && logger --priority "user.err" "$*"
        printf "Error: %s\n" "$*" >& 2
        tryto rm -r "$NDIR"
        exit 1
}

# Run a command.  If if fails, exit with an apprpriate error message
command() {
        $@ || die "Command $* failed - exiting."
}

# Application functions

# Print the checksum and date for the most recently dated record from stdin (an sha512sum.txt file)
# The parameter is the date of the previous snapshot already downloaded
# If no records are later than that, prints a null string
latestsum() {
        local line date lline ldate
        lline=""
        ldate="$1"

        while read -r line
        do
                # Keep only the latest date and line
                date="${line#*$PREFIX}"
                date="${date%%.*}"
                if [ "$date" -gt "$ldate" ]
                then
                        ldate="$date"
                        lline="$line"
                fi
        done

        printf "%s" "$lline"
}

# Swap new for current directories safely
update() {
        [ -e "$ODIR" ] && command rm -r "$ODIR"
        command mv "$DIR" "$ODIR"
        command mv "$NDIR" "$DIR"
        command rm -r "$ODIR"
}

### Mainline code

say "Starting portage tree snapshot update with $SYNCURI"

# Get the date of the current snapshot from our timestamp file (if any, otherwise use the epoch start).
cdate=$(cat "$DIR/$STAMP")
[ "$cdate" ] || cdate="19700101"

# Sync a new SUMS file into the NDIR.  Start with a copy, so the original remains if thing go wrong
tryto mkdir "$NDIR"     # Assume fails means NDIR is left from before.
tryto cp  "$DIR/$SUMS" "$NDIR/$SUMS"    # Assume fail means DIR is not yet populated - i.e. this is first run
command rsync --verbose "$SYNCURI/$SUMS" "$NDIR/$SUMS"

# Ignore the line for the new "current" snapshot; it tends to be out of date and hence with the wrong checksum
# Instead, grep SUMS for the right type and select the line with the latest date after the current snapshot

line=$(grep -E -o "[[:xdigit:]]+[ *]+$PATTERN" "$NDIR/$SUMS" | latestsum $cdate)

[ "$line" ] || die "No newer snapshot on $SYNCURI than $cdate.  Sync aborted"

nsum="${line%%[ *]*}"
nfile="${line##*[ *]}"
ndate="${line#*$PREFIX}"
ndate="${ndate%%.*}"

# OK, according to SUMS there's a later snapshot.  Sync a new copy into NDIR.
target="$NAME.$TYPE"
tryto cp "$DIR/$target" "$NDIR/$target"
command rsync --verbose "$SYNCURI/$nfile" "$NDIR/$target"

# Check the sum is correct
shasum=$(sha512sum "$NDIR/$target")
shasum="${shasum%%[ *]*}"

if [ -z "$shasum" ]
then die "Unable to calculate sha512sum for current snapshot $NDIR/$target"
elif [ "$shasum" != "$nsum" ]
then die "Checksum $shasum for $NDIR/$target differs from $N$SYNCURI/$nfile."
else
        printf "%s" "$ndate" > "$NDIR/$STAMP"   # Create the timestamp file

        # Replace the current snapshot with the new.  If portage tree is mounted, unmount and remount it
        if mountpoint -dq "$PORTDIR"
        then
                command umount "$PORTDIR"
                update
                command mount "$PORTDIR"
        else
                update
        fi

        say "Successful sync to portage tree snapshot dated $ndate."
fi
Hi, great job

I am a normal user who does not understand scripts. (Sorry)

I get this error when I run it:

Code: Select all

sudo sh /home/elover/Documentos/actualizar.sh                                                                                                                              ─╯
Starting portage tree snapshot update with rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/sha512sum.txt': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/sha512sum.txt /var/db/snapshot.new/sha512sum.txt failed - continuing.

 -========== B Y T E M A R K   H O S T I N G   M I R R O R  ==========- 

sha512sum.txt

sent 43 bytes  received 29,716 bytes  19,839.33 bytes/sec
total size is 29,627  speedup is 1.00
cp: no se puede efectuar `stat' sobre '/var/db/snapshot/gentoo-snapshot.lzo.sqfs': No existe el fichero o el directorio
Warning: Command cp /var/db/snapshot/gentoo-snapshot.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - continuing.
rsync: link_stat "/snapshots/squashfs/gentoo-20210312.lzo.sqfs" (in gentoo) failed: No such file or directory (2)

sent 8 bytes  received 131 bytes  278.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1816) [Receiver=3.2.3]
Error: Command rsync --verbose rsync://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/gentoo-20210312.lzo.sqfs /var/db/snapshot.new/gentoo-snapshot.lzo.sqfs failed - exiting.
I guess then you put it in cron, how do you configure it.

Sorry for asking so much, but my SSD has already died.
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Sun Mar 14, 2021 9:30 am

elover,

Ah, sorry, you're being caught by the same problem described in previous posts on this thread. That script can go wrong if you are using an rsync URI for a server pool (and most of those listed on the Gentoo download pages are pooled). The script gets the sha512sum.txt file (whiich lists the available snapshots) and chooses the latest, but when it goes back to get the snapshot, it gets a different server which is out of date from the pool. (Why some of the pool's servers are a day out of date is probably down to the various mirror service providers, and may be beyond the power of the internet to get it fixed everywhere.)

I have a new version of the script that's a lot simpler, and gets the sha512sum.txt file and the current snapshot in one operation, so it will use the same server and get a consistent result. I'm on the wrong PC at the moment, but I'll post the new version this morning (UK time).
Greybeard
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Sun Mar 14, 2021 11:35 am

OK, I've updated my script. I edited the post containing the script, rather than leave the broken one in the thread.
Greybeard
Top
geki
Advocate
Advocate
User avatar
Posts: 2387
Joined: Thu May 13, 2004 8:59 pm
Location: Germania
Contact:
Contact geki
Website

  • Quote

Post by geki » Sun Mar 14, 2021 12:02 pm

JFYI, I recommend you to check your local university FTP site for a gentoo rsync mirror. In germany, we got plenty of them. No more out-of-date pools. Like so:

https://<your favorite FTP site>/gentoo/ or https://<your favorite FTP site>/pub/gentoo/

I use, for example, https://ftp.fau.de/gentoo/. But there are a dozen others in germany, too.

Goverp
Maybe better to add a config variable to be set by the user of the script.
hear hear
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Sun Mar 14, 2021 2:03 pm

geki wrote:...
Goverp
Maybe better to add a config variable to be set by the user of the script.
That's SYNCURI, unless I'm misunderstanding you.
Greybeard
Top
NiXZe
Tux's lil' helper
Tux's lil' helper
User avatar
Posts: 78
Joined: Thu May 22, 2003 5:53 pm
Location: Uppsala Sweden
Contact:
Contact NiXZe
Website

  • Quote

Post by NiXZe » Wed Aug 14, 2024 7:34 pm

Goverp wrote:Yup, trying my new script, the gentoo-current link seems to be one day behind - I wonder if the process that builds the snapshots directory sets the link BEFORE it creates the new snapshot, maybe for "safety", and then somehow fails to update it for the new one. Whatever. To add to the fun, the TIMESTAMP in the Manifest file is yet another day back. So today I have

from the sha512sum.txt file entries for .lzo files:

Code: Select all

d94925dce716d81f025031387e63debdd5b4dbdd00de63ee033434b4a20cdc4b2ae4b8106e697aa686e29a12280b574c7c3b3bd96839e90967ec0859eb478b96  gentoo-20210227.lzo.sqfs
f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-20210228.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-20210301.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-current.lzo.sqfs
so the current claims to be 2021/03/01
but with my latest rsynced gentoo-current.lzo.sqfs, sha512sum returns

Code: Select all

f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-current.lzo.sqfs
which you can see is the one for gentoo-20210228.lzo.sqfs, which the web page for ftp://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/ claims was created: 2021-03-01 01:45 (which is reasonable if the snapshot creation started at midnight on 02/28), but having mounted the snapshot, the Manifest file within it says

Code: Select all

grep TIMESTAMP /var/db/repos/gentoo/Manifest
TIMESTAMP 2021-02-27T01:38:33Z
so it seems to be a day older again!

OK, I'm not too bothered about being a day (or even 2) out, but clearly summat is going weird. I'll see if I can work out who to contact in the infrastructure world.
Trying to get this working reliably I ended up opening a Bug due to the inconsistencies between checksum and actual files.
The reason (at least currently) is CDN and caching, which gives a possible out of sync view of these files.
Details explained in the response on the bug: https://bugs.gentoo.org/937906#c1

rsync on main mirror would probably work most of the time.
What? Where? Oh! this one, it's Gentoo.
If you don't have anything constructive to say, you might want to consider staying quiet instead.
To many destructive comments, which in no way will help making the Gentoo community better.
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Thu Aug 15, 2024 9:04 am

I stopped using the rsync method of updating my squashfs snapshot about a year ago. For some reason I couldn't figure out, rsync was no longer saving any IO compared with simply downloading the snapshot. So now I use a simple wget. It also means I changed the mirrors from which I download the snapshot - now I get it from the usual Gentoo source mirrors (which also have snapshot directories) instead of the Gentoo rsync mirrors.

That's not directly linked to the issue of the snapshots and checksum files being out of sync for the "current" snapshot, except that I've not had that problem since I started using wget! I presume that's because of using the source mirrors, rather than the use of wget. (The snapshots on the rsync mirrors appeared to be second-class citizens - about once a month I'd find them days out of date, whereas the source mirrors' snapshots have so far always been up to date. That might just be luck.) Whatever.

When I'm back on the right machine, I'll upload my current script in case anyone's interested.
Greybeard
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Thu Aug 15, 2024 11:48 am

As promised:

Code: Select all

#!/bin/sh

### Update a squashfs portage snapshot from an rsync mirror
# Version 0.4 Paul Gover 2024/2/1
# Changes:
# Use wget instead of rsync, allowing the use of source mirrors instead of rsync mirrors.
# In theory, rsync gave I/O reductions, but since about the start of 2023 this was no longer true.
# Trim the mirror name of protocol and stuff, to allow copy and paste from mirror lists.
# Ensure the snapshot directory and files are in portage group
# Leave the snapshot.old directory as a fallback.  Not something portage can do for you!

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
portage="/var/db/repos/gentoo"

mirror="http://www.mirrorservice.org/sites/distfiles.gentoo.org/snapshots/squashfs/"

# Calculate the mirror server and directory
mirror="${mirror#*://}"         # strip protocol prefix, if any
mirror="${mirror%/}"            # strip trailing /, if any
mirrorDirectory="${mirror#*/}"
mirrorURI="${mirror%/"$mirrorDirectory"}"

# Defintions for what and where to store the snapshot.
currentDir="/var/db/snapshot"
checksums="sha512sum.txt"
shapshotType="lzo.sqfs"
shapshotName="gentoo-current"

newDir="$currentDir.new"
oldDir="$currentDir.old"
snapshot="$shapshotName.$shapshotType"

# Swap new for current directories safely
update() {
        [ -e "$oldDir" ] && rm -r "$oldDir"
        mv "$currentDir" "$oldDir"
        mv "$newDir" "$currentDir"
}

### Mainline code

printf 'Starting portage tree synchronization\n'

# Ensure there's a nice clean and tidy new directory
[ -d "$newDir" ] && rm -r "$newDir"
mkdir "$newDir"         # Ensure there's a new directory

cd "$newDir"    # In the new directory

wget "$mirrorURI/$mirrorDirectory/$checksums"
wget "$mirrorURI/$mirrorDirectory/$snapshot"

gpg --verify  "$checksums"
sha512sum --check --ignore-missing "$checksums"

cd

[ "$newDir/$snapshot" -nt "$currentDir/$snapshot" ] || { printf 'Newly-synchronized snapshot from %s is no newer than current.\n' "$mirrorURI" ; exit 1 ; }

if mountpoint -q "$portage"
then
        umount "$portage"
        update
        mount "$portage"
        printf 'Portage tree synchronized %s.\n' "$(grep -F "TIMESTAMP" "$portage/Manifest")"
else
        update
fi

chgrp -R portage "$currentDir"
Greybeard
Top
NiXZe
Tux's lil' helper
Tux's lil' helper
User avatar
Posts: 78
Joined: Thu May 22, 2003 5:53 pm
Location: Uppsala Sweden
Contact:
Contact NiXZe
Website

  • Quote

Post by NiXZe » Thu Aug 15, 2024 7:13 pm

Goverp wrote:I stopped using the rsync method of updating my squashfs snapshot about a year ago. For some reason I couldn't figure out, rsync was no longer saving any IO compared with simply downloading the snapshot. So now I use a simple wget. It also means I changed the mirrors from which I download the snapshot - now I get it from the usual Gentoo source mirrors (which also have snapshot directories) instead of the Gentoo rsync mirrors.

That's not directly linked to the issue of the snapshots and checksum files being out of sync for the "current" snapshot, except that I've not had that problem since I started using wget! I presume that's because of using the source mirrors, rather than the use of wget. (The snapshots on the rsync mirrors appeared to be second-class citizens - about once a month I'd find them days out of date, whereas the source mirrors' snapshots have so far always been up to date. That might just be luck.) Whatever.

When I'm back on the right machine, I'll upload my current script in case anyone's interested.
If you are using one and the same mirror that would probably be more reliable than the distfiles CDN.
But the same should also hold true most of the time for any one and same rsync mirror.

Not recommending, but after getting the explanation in the bug report the below thisbeen reliable

Code: Select all

rsync --progress -v --copy-links rsync://gentoo.osuosl.org/gentoo/snapshots/squashfs/sha512sum.txt rsync://gentoo.osuosl.org/gentoo/snapshots/squashfs/gentoo-current.xz.sqfs  .
grep gentoo-current.xz.sqfs sha512sum.txt | sha512sum -c 
I think the most important part is to always download both files at the same time, in the same connection, and not using the CDN.
What? Where? Oh! this one, it's Gentoo.
If you don't have anything constructive to say, you might want to consider staying quiet instead.
To many destructive comments, which in no way will help making the Gentoo community better.
Top
NiXZe
Tux's lil' helper
Tux's lil' helper
User avatar
Posts: 78
Joined: Thu May 22, 2003 5:53 pm
Location: Uppsala Sweden
Contact:
Contact NiXZe
Website

  • Quote

Post by NiXZe » Sun Jun 01, 2025 8:02 pm

I was alerted that some checks i did was vulnerable to padding attacks https://github.com/ASoft-se/Gentoo-HAI/ ... 2294995050
But it took some time before I actually understood, but here is an example:

Code: Select all

# grab current checksum file as a base
checksums="sha512sum.txt"
wget "https://gentoo.osuosl.org/snapshots/squashfs/$checksums"
# create an "attack" file, and checksum append/pad the existing file
touch gentoo-current.xz.sqfs
sha512sum gentoo-current.xz.sqfs >> "$checksums"

# gpg verification fails?
gpg --verify "$checksums" && echo WHOOPS!
sha512sum --check --ignore-missing "$checksums" || echo At least good here
Instead only use verified part of csum file:

Code: Select all

[ -f "sha512sum.verified.txt" ] && rm "sha512sum.verified.txt"
echo y | gpg \
  --trusted-key ABD00913019D6354BA1D9A132839FE0D796198B1 \
  --output "sha512sum.verified.txt" \
  --verify "$checksums" || (echo gpg verification failure; exit 1)
sha512sum --check --ignore-missing "sha512sum.verified.txt" || (echo sha512 sum verification failure; exit 1)
In regards to mismatch, I stopped using gentoo-current and instead grabbed all date files from csum, sorted them and took the latest only.
But will be moving to this logic:

Code: Select all

DISTMIRROR=https://distfiles.gentoo.org
TRUSTKEY=ABD00913019D6354BA1D9A132839FE0D796198B1
curl -L -C - --remote-name-all --parallel-immediate --parallel \
  https://qa-reports.gentoo.org/output/service-keys.gpg \
  ${DISTMIRROR}/snapshots/squashfs/sha512sum.txt || exit 1

# gpg import, trust starting with Gentoo L1 signing key
gpg \
  --trusted-key $TRUSTKEY \
  --import service-keys.gpg && rm service-keys.gpg
echo "Ensure Gentoo infrastructure gpg key ..."
gpg --locate-key infrastructure@gentoo.org

# Validate sha512sum
[ -f sha512sum.verified.txt ] && rm sha512sum.verified.txt
gpg \
  --trusted-key $TRUSTKEY \
  --output sha512sum.verified.txt  \
  --verify sha512sum.txt \
  && rm sha512sum.txt || exit 1

[ -f sha512sum.verified.txt ] || exit 1
# WARNING: prepare for this file to change format in future to BSD-like tagged checksum
expected_checksum_and_file=$(awk  '/\<gentoo-[0-9]*\.xz\.sqfs/{l=$0}END{print l}' sha512sum.verified.txt)
SNAPSHOT=${expected_checksum_and_file//* }
curl -C - --remote-name-all "${DISTMIRROR}/snapshots/squashfs/$SNAPSHOT" || exit 1
snapshot512=$(sha512sum $SNAPSHOT | awk '{print $1}')

echo -e "\e[93mSnapshot  SHA512 $snapshot512 ...\e[0m"
echo -e "\e[93mExpecting SHA512 ${expected_checksum_and_file// *}\e[0m"
grep $snapshot512 sha512sum.verified.txt && echo -e " \e[92m - OK\e[0m" || (echo " \e[91mnot found in sha512sum.txt\e[0m"; exit 1)
rm sha512sum.verified.txt
Questions:
  • Any experiences with remounting squashfs while it is in use?, That is update existing squashfs without full umount?
  • Any work on a Portage Sync module? https://wiki.gentoo.org/wiki/Project:Po ... n_sync_API
Top
Goverp
Advocate
Advocate
User avatar
Posts: 2404
Joined: Wed Mar 07, 2007 6:41 pm

  • Quote

Post by Goverp » Thu Sep 04, 2025 7:15 pm

NiXZe, thanks for the warning about padding attacks.
For anyone using a script based on mine, it's worth making a small change: instead of doing the gpg and sha512 checksums independently, it's better to pipe the gpg-verified checksums to sha512sum, which should be safe against padding. My script today looks like the following:
#!/bin/sh

Code: Select all

### Update a squashfs portage snapshot from an rsync mirror
# Version 0.5 Paul Gover 2025/8/28
# Changes:
# Pipe gpg-verified checksums to sha512sum
# Use wget instead of rsync, allowing the use of source mirrors instead of rsync mirrors.
# In theory, rsync gave I/O reductions, but since about the start of 2023 this was no longer true.
# Trim the mirror name of protocol and stuff, to allow copy and paste from mirror lists.
# Ensure the snapshot directory and files are in portage group
# Leave the snapshot.old directory as a fallback.  Not something portage can do for you!

set -u -e       # Catch typos and unchecked commands

# Portage definitions - basically from /etc/portage/make.conf and/or /etc/portage/repos.conf/gentoo
portage="/var/db/repos/gentoo"
mirror="https://www.mirrorservice.org/sites/distfiles.gentoo.org/snapshots/squashfs/"
#mirror="http://mirror.bytemark.co.uk/gentoo/snapshots/squashfs/"


# Calculate the mirror server and directory
#mirror="${mirror#*://}"                # strip protocol prefix, if any
mirror="${mirror%/}"            # strip trailing /, if any
mirrorDirectory="${mirror#*/}"
mirrorURI="${mirror%/"$mirrorDirectory"}"

# Defintions for what and where to store the snapshot.
currentDir="/var/db/snapshot"
checksums="sha512sum.txt"
shapshotType="lzo.sqfs"
shapshotName="gentoo-current"

newDir="$currentDir.new"
oldDir="$currentDir.old"
snapshot="$shapshotName.$shapshotType"

# Swap new for current directories safely
update() {
        [ -e "$oldDir" ] && rm -r "$oldDir"
        mv "$currentDir" "$oldDir"
        mv "$newDir" "$currentDir"
}

### Mainline code

printf 'Starting portage tree synchronization\n'

# Ensure there's a nice clean and tidy new directory
[ -d "$newDir" ] && rm -r "$newDir"
mkdir "$newDir"         # Ensure there's a new directory

cd "$newDir"    # In the new directory

wget "$mirrorURI/$mirrorDirectory/$checksums"
wget "$mirrorURI/$mirrorDirectory/$snapshot"

# PMG 25/08/28 fix verification to ensure sha512sum only processes the verified lines
gpg --output - --verify "$checksums" | sha512sum --check --ignore-missing
# gpg --verify  "$checksums"
# sha512sum --check --ignore-missing "$checksums"

cd

[ "$newDir/$snapshot" -nt "$currentDir/$snapshot" ] || { printf 'Newly-synchronized snapshot from %s is no newer than current.\n' "$mirrorURI" ; exit 1 ; }

if mountpoint -q "$portage"
then
        umount "$portage"
        update
        mount "$portage"
        printf 'Portage tree synchronized %s.\n' "$(grep -F "TIMESTAMP" "$portage/Manifest")"
else
        update
fi

chgrp -R portage "$currentDir"
Greybeard
Top
Post Reply

41 posts
  • Previous
  • 1
  • 2

Return to “Documentation, Tips & Tricks”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy

 

 

magic