TIP: Compressing portage using squashfs: initscript method

[Khumarahn]

I am interested in squashfs primarily for a system with little memory, namely Teres 1 arm laptop. It has 2G ram and 16G MMC. With these little resources, optimizing portage from 1G to 50M is important. Then, the sync should ideally be also efficient... May not have enough ram free to keep whole portage tree.

[pjp]

tmpfs is optional. I apparently edited that out of my previous post. My squashfs images are ~59M.

[mv]

What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync?

You could use squashmount and call

Code: Select all

squashmount remount [repos mount point]

in an /etc/portage/postsync.d hook.

[Khumarahn]

You could use squashmount and call

Code: Select all

squashmount remount [repos mount point]

in an /etc/portage/postsync.d hook.

Thanks! This worked beautifully. I get a strange error on squashmount remount:

Code: Select all

mount: /usr/portage: wrong fs type, bad option, bad superblock on overlayfs, missing codepage or helper program, or other error.

But despite the error, it works correctly. Is there an easy way to see where the error comes from?

My config:

Code: Select all

#!/usr/bin/perl (this is only for editors)

@order = qw(overlay overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);

$rm_changes = $rm_workdir = $rm_readonly = 0;

my $defaults = {
	COMPRESSION => 'xz',
	COMPOPT_XZ => ['-Xbcj', 'arm'],
};

@mounts = (
	standard_mount('portage', '/usr/portage', $defaults),
);

1;  # The last executed command in this file should be a true expression

Code: Select all

#!/bin/sh
# This file remounts the squashmount mount-point "gentoo" after each syncing
# of the "gentoo" repository.
set -u

repository_name=$1
sync_uri=$2
repository_path=$3

# Run only for repository gentoo:
[ x"$repository_name" = x'gentoo' ] || exit 0

# Run only if mount-point portage is configured
[ x"$(squashmount --quiet print-tag portage 2>/dev/null)" = x'portage' ] || exit 0

squashmount remount portage

[mv]

I get a strange error on squashmount remount

squashmount has its [topic=1079548]own thread[/topic] and a github bugtracker, so we should perhaps not clutter the discussion here.
My guess is that either your kernel is too old (older than 3.18 requires $obsolete_overlayfs being set, see squasmount --man) or some options are not compiled in your kernel (cf. MOUNT_OVERLAY in squashmount --man) and therefore mounting overlay(fs) fails and something else (probably unionfs-fuse) is used as a fallback. With

Code: Select all

squashmount list

you can see what was effectively used for mounting. If you pass squashmount remount the option(s) -vvv you can see which commands/options are actually executed to understand which options overlayfs does not like. If you need further help or suspect a bug, I suggest we move discussion to either the above mentioned thread or (preferrably) to the github bug tracker.

[Khumarahn]

@mv, thank you! Running squashmount -vvv, I found that on the first attempt to mount overlayfs, the mount command does not accept the --workdir option. Probably because I am on a very old kernel with whatever custom patches by Allwinner and sunxi. By the next mount, without --workdir, it mounts and works.

I found that it is important to disable hardlinks for portage sync (`sync-allow-hardlinks = no`). Otherwise the sync is ugly, slow and takes lots of space.

I will ask more questions in the other thread or github, if I have any. So far, I am rather satisfied!

[mv]

--workdir option

This option (and the working directory) are used for the variant in newer kernels; squashmount calls this variant "overlay".
The variant in older kernels (without a working directory) is by squashmount called "overlayfs".
Since the default value of @order is qw(overlay!? overlayfs!? aufs! unionfs-fuse! unionfs! funionfs!), squashmount will first attempt 'overlay' and in case of failure 'overlayfs'. Thus, to suppress the error message, you might exchange the first two entries in your configuration file:

@order=qw(overlayfs!? overlay!? aufs! unionfs-fuse! unionfs! funionfs!);

This way, first the variant without a working directory is attempted. However, you will have to remove that line once you upgrade to >=linux-3.18.

[Khumarahn]

I already put overlayfs first... I get this that error message with

Code: Select all

@order = qw(overlayfs? overlay)

[mv]

I already put overlayfs first..

You are right; I misremembered. The working directory was introduced in linux-3.15. The correct recommendation would probably have been instead to set

$obsolete_overlayfs = 1;

[Khumarahn]

Code: Select all

$obsolete_overlayfs = 1;

That worked ))) Thanks!

[Princess Nell]

I went off the script posted here and aufs years ago and switched to https://www.brunsware.de/blog/portage-t ... -overlayfs. And finally, I got around to this (local portage dir, app-portage/squashfs-portage) as I got sick of installing it manually on a new machine every time.

Code: Select all

# Copyright 2021 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=7

DESCRIPTION="Use squashfs and overlays for portage tree"
HOMEPAGE="https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"

DEPEND="sys-fs/squashfs-tools"
RDEPEND="${DEPEND}"
BDEPEND=""

pkg_setup() {
	mkdir "${S}"
}

src_install() {
	newinitd "${FILESDIR}"/squashfs_portage-init.d squash_portage
	newconfd "${FILESDIR}"/squashfs_portage-conf.d squash_portage
}

Take the init script from HOMEPAGE and copy it to files/squashfs_portage-init.d. Remove the 4 lines starting with SQFS_CUR, OVERLAY_DIR, RO_DIR and PORT_DIR, and instead put them into files/squashfs_portage-conf.d. ebuild/repoman etc. to taste.

[szatox]

I've just had a reason to revisit my old, tiny VPS and take care of its ever growing portage tree.
Well, I guess I might just as well add bootstrap and reset buttons to the emerge sync script I wrote a few years ago and share it.


Things to watch out for during setup:
* adjust paths:
- currently uses old /usr/portage instead of more modern /var/db/repos/gentoo, make sure TARGET matches your system settings
- I put the actual data under /media because reasons. Feel free to send it anywhere else if it bothers you (or you want to use it on a machine with removable storage devices, just to avoid conflicts)
* name the script just like the target directory, or set environmental variables before running it. Multiple repositories can be supported with symlinks.
e.g. portage for /usr/portage, gentoo for /var/db/repos/gentoo, guru for /var/dv/repos/guru etc.

* Run with command bootstrap once (reset if the repo gets corrupted for whatever reason and you need to fully resync), mount during boot, update when you want to sync portage, and squash to compress the tree (will be needed for overlays if you have them).
Fall-through switch kicks off subsequent actions automatically and allows for some retries (or squashing overlays without running emerge --sync for everything again... Perhaps doing it in a loop instead of relying on script's name would be better now )
I handle this point with cron's @reboot and @daily tags.

Code: Select all

# cat /opt/scripts/squash_it/portage
#! /bin/bash
#set -x
set_vars () {
        OVFS="${OVFS:-/media/overlay}"
        INSTANCE="${INSTANCE:-$(basename $0)}"
        TARGET="${TARGET:-/usr/${INSTANCE}}"

        SQDIR="${OVFS}/squash"
        LOWER="${OVFS}/lower/${INSTANCE}"
        UPPER="${OVFS}/upper/${INSTANCE}"
        WORK="${OVFS}/work/${INSTANCE}"
        TMP="${OVFS}/tmp"

        SQNAME="${INSTANCE}"
        SQFS="${SQDIR}/${SQNAME}"
        SQCMD="/usr/bin/mksquashfs"
# excluding files does not work as reliably as I'd like, but git it already compressed, so squashing it mostly wastes CPU time and clogs RAM and disk during update. We're better off leaving it in the overlayfs.
        SQPARAM="-quiet -no-progress  -comp xz -Xdict-size 100% -wildcards -e '.git/'"

# This was for testing, not needed anymore, but I'm too lazy to remove it right now
        MODE="eval"
}
delay (){
echo "### running command: ==> $@ <=="
sleep 5
eval "$@"
}

main (){
        set_vars
        case $1 in
        "sudo: reset" )
                "${MODE}" /bin/umount -l "${TARGET}"
                "${MODE}" /bin/umount -l "${LOWER}"
                "${MODE}" rm -rf "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQFS}"
                ;&
        bootstrap )
                "${MODE}" mkdir -p "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQDIR}"
                "${MODE}" mkdir "/tmp/empty.$$"
                "${MODE}" "${SQCMD}" "/tmp/empty.$$" "${SQFS}" ${SQPARAM}
                "${MODE}" rmdir "/tmp/empty.$$"
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after bootstrap"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                main mount
                main update
                ;;
        update )
                "${MODE}" emerge --sync -q || exit 1
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after sync"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                        ;&
        squash )
                "${MODE}" "${SQCMD}" "${TARGET}" "${TMP}/${SQNAME}" ${SQPARAM} || exit 2
                        ;&
        clean )
                "${MODE}" /bin/umount -l "${TARGET}" || exit 3
                "${MODE}" /bin/umount -l "${LOWER}" || exit 4
                "${MODE}" mv "${TMP}/${SQNAME}" "${SQFS}" || exit 5
                "${MODE}" find "${UPPER}/ -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} \+" || exit 6
                        ;&
        mount )
                "${MODE}" /bin/mount -o loop "${SQFS}" "${LOWER}" || exit 7
                "${MODE}" /bin/mount -t overlay overlay -o "lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK}" "${TARGET}" || exit 8
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after remount"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                ;;
        *)
                echo "usage: $0 < sudo: reset | bootstrap | update | squash | clean | mount >"
        esac
}

main "$@"

The workdir of portage apparently contains 250MB of text files, uses 700-900MB of disk space (depending on FS) and compresses own to 35MB.
I know, I know, disk space is cheap, but my VPS only have 20GB of it.

[yoshi314]

there is one thing worth mentioning when using systemd

i had /usr/src as a symlink to a separate volume mounted elsewhere. when i specified links to files in /usr/src in squashmount.pl , the service would fail. It would work if i did sudo squashmount start later

I had to explictly list paths to actual squash files' mount points, not traversing via symlinks. I have no clue why this happens

[mv]

there is one thing worth mentioning when using systemd

i had /usr/src as a symlink to a separate volume mounted elsewhere. when i specified links to files in /usr/src in squashmount.pl , the service would fail. It would work if i did sudo squashmount start later

I had to explictly list paths to actual squash files' mount points, not traversing via symlinks. I have no clue why this happens

Wild guess: Some components of the paths to the links or of the links were not mounted when systemd started the service, that is, you might need to specify some dependency explicitly. But there are several other setups I can think of which can cause this: For instance, involving mount --bind or --rbind. There might also be permission problems to some components of the paths (to the links or of the links). Probably there are also things which can cause this I am not aware of: With containers (which systemd is heavily manipulating) a lot of strange things can happen.