The really simple way to use the portage tree on squashfs ?

[Goverp]

IMHO the portage tree doesn't sit well on an SSD, nor indeed on rotating rust - too many small files, to many changes in each sync. So I read up on using Squashfs to hold it. Most of solutions get involved in ways to update the squashed file system after an "emerge --sync", and so get involved in overlayfs and the like. Too complex for a bear of little brain such as me.

I notice the portage rsync mirrors contain snapshots apparently taken a bit after midnight each day, with the latest called "current". There's also an sha512sum.txt file of checksums. So I wondered if you can replace "emerge --sync" with rsync of the squashfs snapshot, and then an sha512sum to check it. Yes, it works, just like that. rsync is clever with large files, and just transfers changed blocks, so the sync is damn fast.

OK, you get a read-only portage tree, and syncing during the day won't work, but my weekly "emerge --update" should be fine. And you need to setup the PACKAGES and DISTFILES directories in their new /var locations, not the portage tree.

Here's some output, and timings for today's snapshot versus yesterday's, and an emerge --sync today versus a sync yesterday (note these are not particularly comparable, but they give some idea):

Code: Select all

packager@ryzen ~ $ rsync -v --copy-links rsync://mirrors.gethosted.online/gentoo/snapshots/squashfs/gentoo-current.xz.sqfs Downloads/gentoo-current.xz.sqfs
sent 51,703 bytes  received 29,611 bytes  32,525.60 bytes/sec
total size is 54,431,744  speedup is 669.40

# I should add an rsync update on sha512sum.txt - to be investigated - below, I just downloaded it

packager@ryzen ~ $ sha512sum -c --ignore-missing sha512sum.txt 
gentoo-current.xz.sqfs: OK
sha512sum: WARNING: 22 lines are improperly formatted

This compares with (to skip rather a lot of rsync output)

Code: Select all

packager@ryzen ~ $ emerge --sync
...
Number of files: 146,441 (reg: 119,956, dir: 26,485)
Number of created files: 75 (reg: 75)
Number of deleted files: 225 (reg: 215, dir: 10)
Number of regular files transferred: 1,068
Total file size: 209.09M bytes
Total transferred file size: 8.16M bytes
Literal data: 8.16M bytes
Matched data: 0 bytes
File list size: 3.36M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 51.82K
Total bytes received: 9.99M

sent 51.82K bytes  received 9.99M bytes  647.86K bytes/sec
total size is 209.09M  speedup is 20.82

So, sent amount is about the same (~52KB), but the received is over 300 times less!

I'll try this all again next week, when there will be more to compare.

[pjp]

I notice the portage rsync mirrors contain snapshots apparently taken a bit after midnight each day, with the latest called "current".

Do you download the file named with "current" or do you download the file with the timestamp in the file name? I briefly looked at using those snapshots, but I want the version with the timestamp in the filename. It wasn't immediately obvious to me how that could be automated.

Having the timestamp versions of snapshots makes it easier to attempt incremental updates on systems that have not been updated in months (or worse).

Most of solutions get involved in ways to update the squashed file system after an "emerge --sync", and so get involved in overlayfs and the like.

I use squashfs, but not unionfs. When I update, I extract my current ::gentoo squashfs to a tmpfs (currently 800M works), sync to that tmpfs, then create a new squashfs.

[Goverp]

... Do you download the file named with "current" or do you download the file with the timestamp in the file name? I briefly looked at using those snapshots, but I want the version with the timestamp in the filename. It wasn't immediately obvious to me how that could be automated. ...

The --copy-links means it gets the target of the "current" version (which is a symbolic link on the mirror). Without it, rsync just barfs at reading the non-regular file. You can work it out from the sha512sum.txt file - the "current" snapshot has the same sum as the file it points at (which happens to the the penultimate group of entries in the sha512sum.txt file).. Alternatively, the file "Manifest" in the snapshot contains a datestamp as well as other checksums.

... I use squashfs, but not unionfs. When I update, I extract my current ::gentoo squashfs to a tmpfs (currently 800M works), sync to that tmpfs, then create a new squashfs.

Doing it my way, I just rsync the snapshot file itself; that gives the new squashfs; no need to extract it, just mount it; something like:

Code: Select all

mount -t squashfs -o loop Downloads/gentoo-current.xz.sqfs /usr/portage

(you can even have the appropriate entry in /etc/fstab to mount it at boot).
Obviously, you need to unmount it during the rsync, and remount it afterwards.

[pjp]

Interesting, thanks. I had only tried with wget.

If I read how you used rsync, I read too quickly. That looks a LOT easier. I'll have to try that as a default.

I don't recall the last time I "needed" to sync at some random time, so I doubt I'll need my current method. But it isn't too difficult to fall back on if it is needed.

I think this will solve a not so insignificant problem I had. Thanks again!

[Anon-E-moose]

Thanks, I changed the way I was syncing.

I added this to crontab (replacing a regular emerge --sync) could have more checks in it.

Code: Select all

#!/bin/bash
  
dout=`date "+%Y-%m-%d"`

umount /usr/portage

rsync -v --copy-links rsync://mirrors.gethosted.online/gentoo/snapshots/squashfs/gentoo-current.xz.sqfs /var/portage
rc_sqfs=$?

rsync -v --copy-links rsync://mirrors.gethosted.online/gentoo/snapshots/squashfs/sha512sum.txt /var/portage
rc_sha=$?

if [ $rc_sqfs -eq 0 ] && [ $rc_sha -eq 0 ]
 then
  cd /var/portage

  sha512sum -c --ignore-missing sha512sum.txt --status
  rc=$?
 else
  rc=1
fi

mount /usr/portage

if [ $rc -eq 0 ]
 then
  /usr/bin/eix-update -q

  /usr/bin/rsync -aix /var/portage/gentoo-current.xz.sqfs nas:/mnt/backup/bkup/$dout.portage.xz.sqfs
fi

exit $rc

Note: nas is my backup server for portage related stuff

fstab entry for squashfs

Code: Select all

/var/portage/gentoo-current.xz.sqfs	/usr/portage	squashfs	user,loop	0 0

[pjp]

could have more checks in it.

Code: Select all

egrep -v '^$|^#|^\s+#' pdm |wc -l
133

Includes "help", timestamp checks / creation, unmounting, temporary write space, unsquashfs, sync, (re)squashfs, re-linking to the generic mountpoint (timestamp vs. fstab no timestamp), and remounting the squashfs repo. :) I still had some cleanup to do and was considering a rewrite.

[geki]

Thanks Goverp, I changed from git to your squashfs type of sync. Git is eating gigs and started getting on my nerves; backup size was doubling in size lately and just then I see your post to save my day. Though, I had to change my rusty but working hacks to /etc/portage/ voodoo and one dummy package for mysql-connector-c[0] to be prepared for readonly portage. Thanks again, man.

[0] https://github.com/geki-yaba/gekis-play ... 999.ebuild

[Goverp]

Glad to be of service!

[Anon-E-moose]

I did my first pull of sqfs this morning (automated), check, and save sqfs file to my backup area.

Syncing the sqfs was indeed faster than a regular portage rsync (even on a tmpfs which I was using) so I'm happy.


I'm going to ask that this thread be moved to Documentation/tips. I think it belongs there.

[NeddySeagoon]

Moved from Portage & Programming to Documentation, Tips & Tricks.

[Goverp]

I looked at writing a suitable module for emaint, but my python fu lacks, and I'm sure I couldn't generate as much code as /usr/lib/python3.8/site-packages/portage/sync/modules/rsync/rsync.py !
Anon-e-mouse's script looks more my level.

[Leonardo.b]

Nice, thanks; even here on the other face of the planet, this is going to be useful.

[Goverp]

FWIW: A bit of digging on t' web shows that lzop compression uses significantly less resource than xz; the Snap guys noticed significant startup delays for images using xz compression. Of course, if you're short of "disk" space, xz is better.

[Anon-E-moose]

FWIW: A bit of digging on t' web shows that lzop compression uses significantly less resource than xz; the Snap guys noticed significant startup delays for images using xz compression. Of course, if you're short of "disk" space, xz is better.

My squashfs loading seems pretty fast, but I did set things in the kernel this way

Code: Select all

CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_CACHE=y
CONFIG_SQUASHFS_DECOMP_MULTI=y
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZ4=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_ZSTD=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3

[mv]

FWIW: A bit of digging on t' web shows that lzop compression uses significantly less resource than xz

If you have a choice, I would always prefer zstd: Almost the quality of xz and speed in the order of lzop.

[Goverp]

If you have a choice, I would always prefer zstd: Almost the quality of xz and speed in the order of lzop.

The only choice in the mirrors is .lzop and .xz!

(For expansion, I agree. However AFAIR lzop is faster for compression, so I use if for filesystem dumps to USB drives, anything else is slower than the IO saved; lz4 is possibly about as good)

[Anon-E-moose]

For compression one could use pxz/pixz (parrallel xz) same compression, faster end result (albeit with more memory/processors)

I wonder if lzop uses multiple cores?

Edit to add: It looks like lzop is similar to gzip, not terribly great on compression, but pretty fast (de)compession

[mv]

(For expansion, I agree. However AFAIR lzop is faster for compression, so I use if for filesystem dumps to USB drives, anything else is slower than the IO saved; lz4 is possibly about as good)

On my system, for mksquashfs, lz4 is fastest, and then lzo or zstd are similar with lzo being only slightly faster.
Once I had made a table https://github.com/vaeth/squashmount/bl ... mpress.txt, but apparently I did not add zstd; probably because my hardware had changed since then.
Note that squashfs uses parallelization by default.
Anyway, one must be careful with such comparisons since lzo and zstd have different compression levels which can easily influence the speed up to a factor of 10.
While the best compression level on lzo hardly saves any space, the best compression level on zstd is about the quality of xz or even brotli in some cases.

Oh yes: When doing comparisons, the disk cache has an enormous influence. It seems even drop_caches does not really drop all read caches (maybe also local caches on the harddisk itself play a role), so you have to repeat in many different orders of compression algorithms and take the median.

[Anon-E-moose]

Some comparisons

https://community.centminmod.com/thread ... etc.17259/ --- missing lzo but does look at parallel compressors

https://catchchallenger.first-world.inf ... LZ4_vs_LZO --- no tests of parallel compressors

https://larryreznick.com/2019/09/19/com ... mpression/ --- some parallel compressors but does mention how many cores per compressor/results

And I've seen a few others with various tests and various compressors.

What I take away, for single (non-parallel) lzo/lz4 best in speed, xz best in compression
For parallel, it changes a lot, even gzip gets spirited when done in a group.

[mv]

What I take away, for single (non-parallel) lzo/lz4 best in speed, xz best in compression

At least in the first test, zstd with highest level is about the same as xz with highest level. This corresponds to my experience.
It is strange that brotli at highest level is worse than xz at highest level. This is unusual and can indicate that the test data archive was unusual in the sense that e.g. it contained some identical or almost identical large files which did fit into the window of xz and zstd but just not into the window of brotli. Or it contained a lot of clearly binary files (like sound or video) on which the plain-text based pre-initialization from brotli does more harm than good. I had a few such cases, but these are rare: Usually, brotli is the best in compression.

[Goverp]

Warning:

I've been beefing up my current "squashfs-sync" script to check dates and so forth.
Today's gentoo-current.lzo.sqfs turns out to be linked to gentoo-20210226.lzo.sqfs which is not the most recent file: gentoo-20210227.lzo.sqfs.
If your scripts assume that gentoo-current points to the most recent file, you'd be disappointed.

I'll test mine a few times before posting it.

FWIW, the rsync between gentoo-20210220.lzo.sqfs and gentoo-20210226.lzo.sqfs is about 50MB, so the saving's not as good as I'd hoped, though probably still worthwhile. It's still simpler than setting up unionfs and running a full emerge --sync.

[Anon-E-moose]

I find that just downloading the squashfs and mounting it, provides a lot of benefits (over the way I did have it)

As far as dates, it's easy enough to grab todays date and apply that to the fetch so that you pull the right date, then you can just rename it current on your end.


something like
dout=`date "+%Y%m%d"`

rsync fetch gentoo-$dout.lzo.sqfs

rename gentoo-$dout.lzo.sqfs gentoo-current.lzo.sqfs

ETA: This morning the tarballs were out of date, but the squashfs was newer ~le sigh~
So I think I'll go back to a modifed old setup, rsync portage tree (on tmpfs), and from that create a backup xz compressed tarball and a squashfs to use for /usr/portage. Little bit more trouble than just downloading a tarball/sqfs but at least I know it's up to date, and I do run it from cron in the middle of the night.

[Goverp]

Yup, trying my new script, the gentoo-current link seems to be one day behind - I wonder if the process that builds the snapshots directory sets the link BEFORE it creates the new snapshot, maybe for "safety", and then somehow fails to update it for the new one. Whatever. To add to the fun, the TIMESTAMP in the Manifest file is yet another day back. So today I have

from the sha512sum.txt file entries for .lzo files:

Code: Select all

d94925dce716d81f025031387e63debdd5b4dbdd00de63ee033434b4a20cdc4b2ae4b8106e697aa686e29a12280b574c7c3b3bd96839e90967ec0859eb478b96  gentoo-20210227.lzo.sqfs
f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-20210228.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-20210301.lzo.sqfs
a30995ba0a9011fce03eb80331033549582e5669ae3c9c8f29439fb7fcc25266bd7e1ed161214c6207d918ad9038133b89d058c846390dc30c3e010a77c97e0e  gentoo-current.lzo.sqfs

so the current claims to be 2021/03/01
but with my latest rsynced gentoo-current.lzo.sqfs, sha512sum returns

Code: Select all

f54d7cfe625dadb8bca084e1b724947d06dd4666f59b13a6e838f8a7195e85326b49c3d81a3197764a1af5272418ad2131e09377ab260e1bca01012843c7fcb6  gentoo-current.lzo.sqfs

which you can see is the one for gentoo-20210228.lzo.sqfs, which the web page for ftp://rsync.uk.gentoo.org/gentoo/snapshots/squashfs/ claims was created: 2021-03-01 01:45 (which is reasonable if the snapshot creation started at midnight on 02/28), but having mounted the snapshot, the Manifest file within it says

Code: Select all

grep TIMESTAMP /var/db/repos/gentoo/Manifest
TIMESTAMP 2021-02-27T01:38:33Z

so it seems to be a day older again!

OK, I'm not too bothered about being a day (or even 2) out, but clearly summat is going weird. I'll see if I can work out who to contact in the infrastructure world.

[Anon-E-moose]

I was looking at that this morning, and that's when I decided to rewrite my script, to just do a regular rsync (against tmpfs) and then make a squashfs from that for mounting as /usr/portage.

[Anon-E-moose]

Ok, first test of new rewrite from last night, worked fine.

Code: Select all

#!/bin/bash

# Paths for different things
# /n/tmp/nas   -- save/read dir on hd
# /tmp/portage -- tmpfs for updating portage tree
# /usr/portage -- where squashfs is mounted
# /var/portage -- where the squashfs file is kept
# nas related  -- the backup box

exit_msg() {
 echo $2
 exit $1
}

main() {
cur_date=`date "+%Y-%m-%d"`
cur_txz="$cur_date.portage.txz"
yday_date=`date -d "yesterday" '+%Y-%m-%d'`
yday_txz="$yday_date.portage.txz"
file_save_path="/n/tmp/nas"
tmp_port_path="/tmp/portage"

# check to see if todays file has been copied to nas bkup
if [ "$1" != "f" ]
 then
  ssh nas "ls /mnt/backup/bkup/$cur_txz" >/dev/null 2>&1
  if [ $? -eq 0 ]; then exit_msg 0 "$cur_txz exists"; fi
fi

# check for existence of ydays tarball and untar it to $tmp_port_path
if [ ! -e $file_save_path/$yday_txz ]; then exit_msg 1 "$file_save_path/$yday_txz doesn't exist !!!"; fi

mkdir $tmp_port_path
tar xfC $file_save_path/$yday_txz $tmp_port_path || exit_msg $? "problem untarring $file_save_path/$yday_txz to $tmp_port_path"

# rsync the latest portage against $tmp_port_path -- limit of 5 retries
rm -f $file_save_path/rsync[1-5].log

count=0

while [ $count -lt 5 ]; do
 rc=-1
 /usr/bin/rsync -ai --delete --timeout=300 rsync://rsync.gentoo.org/gentoo-portage $tmp_port_path >$file_save_path/rsync$count.log 2>&1 
 rc=$?

 if [ $rc -eq 0 ]; then break; fi

 let count=count+1

 sleep 60
done

# check Manifest(s)
if [ $rc -ne 0 ]
 then
  exit_msg $rc "rsync failed after $count tries"
 else
  chown -R portage:portage $tmp_port_path
  echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-" >> $file_save_path/rsync$count.log 2>&1
  gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc $tmp_port_path >> $file_save_path/rsync$count.log 2>&1
fi

# change to /tmp to make tarballs/sqfs
cd /tmp

# create xz compressed tarball for backup server
{ tar cfC - /tmp/portage . | pixz > $cur_txz ; } || exit_msg $? "couldn't create $cur_txz"

# copy to hd for tomorrows run
cp $cur_txz $file_save_path

# sync to backup server
/usr/bin/rsync -aix $cur_txz nas:/mnt/backup/bkup/ || exit_msg $? "*** rsync of $cur_date.portage.xz.sqfs to nas has failed***"

# check for shits and giggles
ssh nas "ls /mnt/backup/bkup/$cur_txz"

# create sqfs for mounting on /usr/portage
mksquashfs portage gentoo-current.xz.sqfs -comp xz -no-progress || exit_msg $? "error creating sqfs for mounting on /usr/portage"

# set up for mounting new /usr/portage 
umount /usr/portage && mv gentoo-current.xz.sqfs /var/portage && mount /usr/portage && /usr/bin/eix-update -q
rc=$?

if [ $rc -eq 0 ] 
 then
  rm -r portage $cur_txz $file_save_path/$yday_txz || echo could not remove portage files/dir in /tmp
 else
  exit_msg $rc "problems with unmount/move/mount of /usr/portage or eix-update failed"
fi
}

time main
exit $rc

With these results

Code: Select all

Wed 03 Mar 2021 02:30:04 AM CST
-- just the rysnc against gentoo on tmpfs
Wed 03 Mar 2021 02:30:09 AM CST

-- copy file to backup
<f+++++++++ 2021-03-03.portage.txz
/mnt/backup/bkup/2021-03-03.portage.txz

-- pixz of /tmp/portage into xz.sqfs
Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on gentoo-current.xz.sqfs, block size 131072.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 52880.24 Kbytes (51.64 Mbytes)
	24.93% of uncompressed filesystem size (212072.62 Kbytes)
Inode table size 1165960 bytes (1138.63 Kbytes)
	24.92% of uncompressed inode table size (4679570 bytes)
Directory table size 1336310 bytes (1304.99 Kbytes)
	34.49% of uncompressed directory table size (3874653 bytes)
Number of duplicate files found 9270
Number of inodes 146104
Number of files 119610
Number of fragments 1482
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 26494
Number of ids (unique uids + gids) 1
Number of uids 1
	portage (250)
Number of gids 1
	portage (250)

-- time for main() total run time minus the original sourcing of the shell script.
real	0m42.140s
user	2m24.132s
sys	0m6.701s

all in all it worked well, now I'm current (as of the rsync) and I've got the benefits of a sqfs.