Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 9, 10, 11  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Mon Sep 24, 2018 10:57 pm    Post subject: Reply with quote

I am interested in squashfs primarily for a system with little memory, namely Teres 1 arm laptop. It has 2G ram and 16G MMC. With these little resources, optimizing portage from 1G to 50M is important. Then, the sync should ideally be also efficient... May not have enough ram free to keep whole portage tree.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 20053

PostPosted: Tue Sep 25, 2018 3:54 am    Post subject: Reply with quote

tmpfs is optional. I apparently edited that out of my previous post. My squashfs images are ~59M.
_________________
Quis separabit? Quo animo?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Sep 25, 2018 4:45 am    Post subject: Reply with quote

Khumarahn wrote:
What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync?

You could use squashmount and call
Code:
squashmount remount [repos mount point]
in an /etc/portage/postsync.d hook.
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Tue Sep 25, 2018 4:04 pm    Post subject: Reply with quote

mv wrote:
You could use squashmount and call
Code:
squashmount remount [repos mount point]
in an /etc/portage/postsync.d hook.


Thanks! This worked beautifully. I get a strange error on squashmount remount:
Code:
mount: /usr/portage: wrong fs type, bad option, bad superblock on overlayfs, missing codepage or helper program, or other error.

But despite the error, it works correctly. Is there an easy way to see where the error comes from?

My config:
/etc/squashmount.pl:
#!/usr/bin/perl (this is only for editors)

@order = qw(overlay overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);

$rm_changes = $rm_workdir = $rm_readonly = 0;

my $defaults = {
   COMPRESSION => 'xz',
   COMPOPT_XZ => ['-Xbcj', 'arm'],
};

@mounts = (
   standard_mount('portage', '/usr/portage', $defaults),
);

1;  # The last executed command in this file should be a true expression


/etc/portage/repo.postsync.d/20-teres-squash-gentoo:
#!/bin/sh
# This file remounts the squashmount mount-point "gentoo" after each syncing
# of the "gentoo" repository.
set -u

repository_name=$1
sync_uri=$2
repository_path=$3

# Run only for repository gentoo:
[ x"$repository_name" = x'gentoo' ] || exit 0

# Run only if mount-point portage is configured
[ x"$(squashmount --quiet print-tag portage 2>/dev/null)" = x'portage' ] || exit 0

squashmount remount portage
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Sep 26, 2018 6:35 am    Post subject: Reply with quote

Khumarahn wrote:
I get a strange error on squashmount remount

squashmount has its own thread and a github bugtracker, so we should perhaps not clutter the discussion here.
My guess is that either your kernel is too old (older than 3.18 requires $obsolete_overlayfs being set, see squasmount --man) or some options are not compiled in your kernel (cf. MOUNT_OVERLAY in squashmount --man) and therefore mounting overlay(fs) fails and something else (probably unionfs-fuse) is used as a fallback. With
Code:
squashmount list
you can see what was effectively used for mounting. If you pass squashmount remount the option(s) -vvv you can see which commands/options are actually executed to understand which options overlayfs does not like. If you need further help or suspect a bug, I suggest we move discussion to either the above mentioned thread or (preferrably) to the github bug tracker.
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Wed Sep 26, 2018 6:44 pm    Post subject: Reply with quote

@mv, thank you! Running squashmount -vvv, I found that on the first attempt to mount overlayfs, the mount command does not accept the --workdir option. Probably because I am on a very old kernel with whatever custom patches by Allwinner and sunxi. By the next mount, without --workdir, it mounts and works.

I found that it is important to disable hardlinks for portage sync (`sync-allow-hardlinks = no`). Otherwise the sync is ugly, slow and takes lots of space.

I will ask more questions in the other thread or github, if I have any. So far, I am rather satisfied!
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Sep 26, 2018 7:54 pm    Post subject: Reply with quote

Khumarahn wrote:
--workdir option

This option (and the working directory) are used for the variant in newer kernels; squashmount calls this variant "overlay".
The variant in older kernels (without a working directory) is by squashmount called "overlayfs".
Since the default value of @order is qw(overlay!? overlayfs!? aufs! unionfs-fuse! unionfs! funionfs!), squashmount will first attempt 'overlay' and in case of failure 'overlayfs'. Thus, to suppress the error message, you might exchange the first two entries in your configuration file:
/etc/squashmount.pl wrote:
@order=qw(overlayfs!? overlay!? aufs! unionfs-fuse! unionfs! funionfs!);

This way, first the variant without a working directory is attempted. However, you will have to remove that line once you upgrade to >=linux-3.18.
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Wed Sep 26, 2018 9:10 pm    Post subject: Reply with quote

I already put overlayfs first... I get this that error message with
Code:
@order = qw(overlayfs? overlay)
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Sep 26, 2018 9:20 pm    Post subject: Reply with quote

Khumarahn wrote:
I already put overlayfs first..

You are right; I misremembered. The working directory was introduced in linux-3.15. The correct recommendation would probably have been instead to set
/etc/squashmount.pl wrote:
$obsolete_overlayfs = 1;
Back to top
View user's profile Send private message
Khumarahn
Apprentice
Apprentice


Joined: 17 Apr 2009
Posts: 199

PostPosted: Wed Sep 26, 2018 9:22 pm    Post subject: Reply with quote

/etc/squashmount.pl:
$obsolete_overlayfs = 1;

That worked :-)))) Thanks!
Back to top
View user's profile Send private message
Princess Nell
l33t
l33t


Joined: 15 Apr 2005
Posts: 916

PostPosted: Tue Feb 23, 2021 9:21 pm    Post subject: Reply with quote

I went off the script posted here and aufs years ago and switched to https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs. And finally, I got around to this (local portage dir, app-portage/squashfs-portage) as I got sick of installing it manually on a new machine every time.
Code:

# Copyright 2021 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

EAPI=7

DESCRIPTION="Use squashfs and overlays for portage tree"
HOMEPAGE="https://www.brunsware.de/blog/portage-tree-squashfs-overlayfs"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~amd64 ~x86"

DEPEND="sys-fs/squashfs-tools"
RDEPEND="${DEPEND}"
BDEPEND=""

pkg_setup() {
   mkdir "${S}"
}

src_install() {
   newinitd "${FILESDIR}"/squashfs_portage-init.d squash_portage
   newconfd "${FILESDIR}"/squashfs_portage-conf.d squash_portage
}


Take the init script from HOMEPAGE and copy it to files/squashfs_portage-init.d. Remove the 4 lines starting with SQFS_CUR, OVERLAY_DIR, RO_DIR and PORT_DIR, and instead put them into files/squashfs_portage-conf.d. ebuild/repoman etc. to taste.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3104

PostPosted: Tue Aug 09, 2022 7:08 pm    Post subject: Reply with quote

I've just had a reason to revisit my old, tiny VPS and take care of its ever growing portage tree.
Well, I guess I might just as well add bootstrap and reset buttons to the emerge sync script I wrote a few years ago and share it.


Things to watch out for during setup:
* adjust paths:
- currently uses old /usr/portage instead of more modern /var/db/repos/gentoo, make sure TARGET matches your system settings
- I put the actual data under /media because reasons. Feel free to send it anywhere else if it bothers you (or you want to use it on a machine with removable storage devices, just to avoid conflicts)
* name the script just like the target directory, or set environmental variables before running it. Multiple repositories can be supported with symlinks.
e.g. portage for /usr/portage, gentoo for /var/db/repos/gentoo, guru for /var/dv/repos/guru etc.

* Run with command bootstrap once (reset if the repo gets corrupted for whatever reason and you need to fully resync), mount during boot, update when you want to sync portage, and squash to compress the tree (will be needed for overlays if you have them).
Fall-through switch kicks off subsequent actions automatically and allows for some retries (or squashing overlays without running emerge --sync for everything again... Perhaps doing it in a loop instead of relying on script's name would be better now )
I handle this point with cron's @reboot and @daily tags.

Code:
# cat /opt/scripts/squash_it/portage
#! /bin/bash
#set -x
set_vars () {
        OVFS="${OVFS:-/media/overlay}"
        INSTANCE="${INSTANCE:-$(basename $0)}"
        TARGET="${TARGET:-/usr/${INSTANCE}}"

        SQDIR="${OVFS}/squash"
        LOWER="${OVFS}/lower/${INSTANCE}"
        UPPER="${OVFS}/upper/${INSTANCE}"
        WORK="${OVFS}/work/${INSTANCE}"
        TMP="${OVFS}/tmp"

        SQNAME="${INSTANCE}"
        SQFS="${SQDIR}/${SQNAME}"
        SQCMD="/usr/bin/mksquashfs"
# excluding files does not work as reliably as I'd like, but git it already compressed, so squashing it mostly wastes CPU time and clogs RAM and disk during update. We're better off leaving it in the overlayfs.
        SQPARAM="-quiet -no-progress  -comp xz -Xdict-size 100% -wildcards -e '.git/'"

# This was for testing, not needed anymore, but I'm too lazy to remove it right now
        MODE="eval"
}
delay (){
echo "### running command: ==> $@ <=="
sleep 5
eval "$@"
}

main (){
        set_vars
        case $1 in
        "sudo: reset" )
                "${MODE}" /bin/umount -l "${TARGET}"
                "${MODE}" /bin/umount -l "${LOWER}"
                "${MODE}" rm -rf "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQFS}"
                ;&
        bootstrap )
                "${MODE}" mkdir -p "${LOWER}" "${UPPER}" "${WORK}" "${TMP}" "${SQDIR}"
                "${MODE}" mkdir "/tmp/empty.$$"
                "${MODE}" "${SQCMD}" "/tmp/empty.$$" "${SQFS}" ${SQPARAM}
                "${MODE}" rmdir "/tmp/empty.$$"
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after bootstrap"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                main mount
                main update
                ;;
        update )
                "${MODE}" emerge --sync -q || exit 1
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after sync"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                        ;&
        squash )
                "${MODE}" "${SQCMD}" "${TARGET}" "${TMP}/${SQNAME}" ${SQPARAM} || exit 2
                        ;&
        clean )
                "${MODE}" /bin/umount -l "${TARGET}" || exit 3
                "${MODE}" /bin/umount -l "${LOWER}" || exit 4
                "${MODE}" mv "${TMP}/${SQNAME}" "${SQFS}" || exit 5
                "${MODE}" find "${UPPER}/ -mindepth 1 -maxdepth 1 ! -name .git -exec rm -rf {} \+" || exit 6
                        ;&
        mount )
                "${MODE}" /bin/mount -o loop "${SQFS}" "${LOWER}" || exit 7
                "${MODE}" /bin/mount -t overlay overlay -o "lowerdir=${LOWER},upperdir=${UPPER},workdir=${WORK}" "${TARGET}" || exit 8
                echo "disk usage (squash + upper)  and uncompressed data size (lower) after remount"
                du -hs "${SQFS}" "${LOWER}" "${UPPER}"
                ;;
        *)
                echo "usage: $0 < sudo: reset | bootstrap | update | squash | clean | mount >"
        esac
}

main "$@"


The workdir of portage apparently contains 250MB of text files, uses 700-900MB of disk space (depending on FS) and compresses own to 35MB.
I know, I know, disk space is cheap, but my VPS only have 20GB of it.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 9, 10, 11
Page 11 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum