Forums

Skip to content

Advanced search
  • Quick links
    • Unanswered topics
    • Active topics
    • Search
  • FAQ
  • Login
  • Register
  • Board index Discussion & Documentation Documentation, Tips & Tricks
  • Search

TIP: Compressing portage using squashfs: initscript method

Unofficial documentation for various parts of Gentoo Linux. Note: This is not a support forum.
Post Reply
Advanced search
264 posts
  • Page 1 of 11
    • Jump to page:
  • 1
  • 2
  • 3
  • 4
  • 5
  • …
  • 11
  • Next
Author
Message
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

TIP: Compressing portage using squashfs: initscript method

  • Quote

Post by synss » Thu May 25, 2006 2:24 pm

Admin Edit:
squashmount related information has been split from this topic. It is now here:
[topic=1079548]Compressing filesystems with squashfs and squashmount[/topic]
--pjp


Tue Nov 28 12:10:10 JST 2006, v. 0.1.5(4)

I have left gentoo and have not updated this script for a while, though it probably still works, mv has taken over and proposed new features and improvements, aufs is one good thing among others. And his script can be used on any directory (I would recommend /usr/src/linux and the docs) You should probably have a look at the thread and check mv's version, now. Last version is available here --- I was sure I had left a notice like this one here but only realize now I must have forgotten...

Abstract The portage tree takes typically 600MB on /usr/portage, some people have chosen to have it on a separate partition with a small block size to
  • decrease its global size
    decrease fragmentation
    increase the speed during searches and update
However that also means that the free space on that partition is lost. Another, better, solution is to use a stack file containing the filesystem with small block size to avoid the shortcomings of the separate partition. Following the discussion in this post, the best solution found is to compress the portage tree using the compressed, read-only squashfs filesystem and mount a read-write partition on top of it using unionfs. Second, for convenience, the portage tree should be mountable on boot-up and updated on shutdown, hence I have written an initscript to automate this process. I will publish and update this program here.

Assumptions I will assume that you have
  • read the original idea as it details what you need to get started and that you are familiar with the bootup/shutdown sequence of your computer,
    read the Gentoo doc on initscripting.
    DISTDIR should be out of the tree, etc.
Please just read the original post as everything is well documented there and come back once you have done that, unless you do understand what you are doing!

Introduction In brief, you will need to load the relevant kernel modules and emerge the userland utilities

Code: Select all

emerge -avt sys-fs/unionfs sys-fs/squashfs-tools
, unmask if needed (I am now successfully using sys-fs/unionfs-1.2 on sys-kernel/suspend2-sources-2.6.16-r8 and sys-fs/squashfs-tools-3.0)

Code: Select all

modprobe loop squashfs unionfs
add the modules to /etc/modules.autoload.d/kernel-2.6.
the script

Code: Select all

# /etc/conf.d/squash_portage

# SQFS_DIRNAME points to the directory that will contain the sqfs
# images, recommended value is /var/tmp
SQFS_DIRNAME="/var/tmp"

# Leave PORTAGE_RW empty for use with tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
PORTAGE_RW=""
and

Code: Select all

#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2 
# $Header: $ 
#
# /etc/init.d/squash_portage allows efficient compression of 
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/unionfs are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin @ users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)

source /etc/make.conf
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"

depend() {
	need localmount
}

start() {
	ebegin "Mounting read-only squashfs image"
	mount -rt squashfs -o loop,nodev,noexec $SQFS_CUR $PORTDIR
	retval=$?
	eend $retval
	[ $retval -ne 0 ] && return $retval

	ebegin "Mounting read-write with unionfs"
	if [ ! $PORTAGE_RW ]
	then
		einfo "  mounted in tmpfs (RAM)"
		PORTAGE_RW="${DEF_RW}"
	fi
	[ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
	chmod 0750 $PORTAGE_RW
	chown portage:portage $PORTAGE_RW
	mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
	eend $?
}

stop() {
	ebegin "Updating portage tree"
	[ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
	if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
	then
		einfo "  Syncing the tree"
		mv -f $SQFS_NEW $SQFS_OLD
		mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
		retval=$?
		ln -sf $SQFS_NEW $SQFS_CUR
	else
		einfo "  Nothing to do"
		retval=0
	fi
	eend $retval

	ebegin "Unmounting the tree"
	umount -t unionfs  $PORTDIR
	umount -t squashfs $PORTDIR
	rm -rf $PORTAGE_RW
	eend 0
}

That obviously is bash as any initscript. Make it executable and

Code: Select all

/etc/init.d/squash_portage start
rc-update add squash_portage default
The script assumes that you already have an sqfs image on your disk ready to be mounted and called $SQFS_DIRNAME/portage.sqfs it will save the new image if you sync'ed the tree the rest is pretty obvious and the rw partition is mounted on /dev/shm by default for speed and simplicity. But beware that it means you will lose your update in case of power failure or if you are not caring. I would recommend to restart the script upon hibernation if you use it and do not always reboot into your ram image.
Last edited by synss on Fri Sep 07, 2007 6:33 pm, edited 19 times in total.
Compress portage tree
Elog viewer
Autodetect swap
Top
yoshi314
l33t
l33t
User avatar
Posts: 852
Joined: Thu Dec 30, 2004 9:33 pm
Location: PL
Contact:
Contact yoshi314
Website

  • Quote

Post by yoshi314 » Thu May 25, 2006 11:40 pm

th...this is great! i would never think of implementing it via initscript :D

nice job.

although there is one drawback with this overlay - when you upgrade your kernel you might get stuck with no portage tree at all, since not every kernel might have squashfs and unionfs modules integrated. there could be a solution in form of an mini-overlay that would contain only ebuilds for squashfs and unionfs (and required eclasses) for forgetful folks. (would it be difficult to make such a backup overlay from an existing portage tree via an bash/sh script?)

sorry for writing that here, but i thought is fits this topic more - it's a more elegant solution.
~amd64
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

  • Quote

Post by synss » Fri May 26, 2006 2:40 am

Thank you!!! :D

No problem for writing here either...

I do not think there should be any problem:
  • 1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" :) can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
    2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.
I have had a problem with union that was acting weird during unmount, but that was a mistake from my side (a non-existing locale when compiling glibc, I think) So you might be unable to sync, but you can then compile and load the modules and go on. In any case, it is always possible to download an image of the tree from a nearby ftp server :wink: yes, that is exactly for avoiding that that I implemented the incremental timestamped backups while debugging. (Then I leave it cause it is a good idea and you can easily clean the backups via cron if you want to.) So the only risk you are taking, AFAIK, is that you might have to mount the squashed image by hand after recompiling the modules.

Thank you again!
Compress portage tree
Elog viewer
Autodetect swap
Top
lazy_bum
l33t
l33t
User avatar
Posts: 691
Joined: Wed Feb 16, 2005 8:55 am

  • Quote

Post by lazy_bum » Thu Jun 29, 2006 6:48 am

Thanks man, this is great! Fast, small and working! (-:

I'm using sys-fs/squashfs-tools-2.2_p2, sys-fs/unionfs-1.1.4-r2 on a sys-kernel/gentoo-sources-2.6.14-r5
roslin uberlay | grubelek
Top
MxxCon
n00b
n00b
Posts: 31
Joined: Fri Apr 01, 2005 8:40 am

  • Quote

Post by MxxCon » Thu Jun 29, 2006 6:35 pm

have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.
Top
Gergan Penkov
Veteran
Veteran
User avatar
Posts: 1464
Joined: Sat Jul 17, 2004 9:42 pm
Location: das kleinste Kuhdorf Deutschlands :)

  • Quote

Post by Gergan Penkov » Thu Jun 29, 2006 6:45 pm

MxxCon wrote:have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.
with squashfs you don't need the free space to unpack it, and these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Top
MxxCon
n00b
n00b
Posts: 31
Joined: Fri Apr 01, 2005 8:40 am

  • Quote

Post by MxxCon » Thu Jun 29, 2006 6:55 pm

Gergan Penkov wrote:these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.
hmm i must have misread something then. i assumed they would create a single compressed file from a dir tree.
Top
Gergan Penkov
Veteran
Veteran
User avatar
Posts: 1464
Joined: Sat Jul 17, 2004 9:42 pm
Location: das kleinste Kuhdorf Deutschlands :)

  • Quote

Post by Gergan Penkov » Thu Jun 29, 2006 7:02 pm

Well I though the same in the beginning, even with such implementation the things would have been just ok if there was some sort of file-container ala tar around the files, of course the compression ratio would not have been very good but there would have been virtually not that much fragmentation and loss of free space :)
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Top
lazy_bum
l33t
l33t
User avatar
Posts: 691
Joined: Wed Feb 16, 2005 8:55 am

  • Quote

Post by lazy_bum » Sat Jul 01, 2006 4:25 pm

Something is wrong with the new version:

Code: Select all

cat /etc/conf.d/squash_portage
source /etc/make.conf

PORTAGE_BASENAME="/usr/portage"
PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs"
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"
PORTAGE_RO=$PORTDIR
PORTAGE_RW="/dev/shm/portage"
And, when running the script:

Code: Select all

/etc/init.d/squash_portage start
 * Mounting squashfs'ed portage tree ...
Usage: mount -V                 : print version
       mount -h                 : print this help
       mount                    : list mounted filesystems
*snip*
A device can be given by name, say /dev/hda1 or /dev/cdrom,
or by label, using  -L label  or by uuid, using  -U uuid .
Other options: [-nfFrsvw] [-o options] [-p passwdfd].
For many more details, say  man 8 mount .                                 [ ok ]
... and /usr/portage/ is empty...

I have squashed portage in /usr/portage.sqfs
And i wonder what "date" is doing in here:

Code: Select all

PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"
Is it possible that a symlink is created to nothing? When i stop the script day after sync'ing the tree, and the date is different?

Hope it's usefull.
roslin uberlay | grubelek
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

  • Quote

Post by synss » Mon Jul 03, 2006 3:44 pm

"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...

I am in a cybercafe on XP with a French kbd so debugging is not easy... But I do not see what else but the first mount can be a problem.

you can also try to mount the tree by hand, sync and copy-paste

Code: Select all

source /etc/make.conf              

PORTAGE_BASENAME="/usr/portage" 
PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs" 
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs" 
PORTAGE_RO=$PORTDIR 
PORTAGE_RW="/dev/shm/portage" 

if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then 
                [ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW 
                mksquashfs $PORTDIR $PORTAGE_NEW -check_data 
                [ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS 
                ln -sf $PORTAGE_NEW $PORTAGE_SQFS 
        fi 
        umount $PORTDIR 
        umount $PORTAGE_RO 
        rm -rf $PORTAGE_RW 
that should make sth clean for starting the initscript and use it and forget it.
Compress portage tree
Elog viewer
Autodetect swap
Top
lazy_bum
l33t
l33t
User avatar
Posts: 691
Joined: Wed Feb 16, 2005 8:55 am

  • Quote

Post by lazy_bum » Tue Jul 11, 2006 6:47 am

synss wrote:"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...
Well, i don't know if it wasn't my computer problem. Had few day of "fun" with kernel panic and stuff...
But maybe it would be better to move previous portage to something like "portage-old.sqfs" and the new one would be "portage-current.sqfs". It would help for the "forgetful folks" to autodelete and keep only one portage (that you use) and one (as a backup). Otherwise few days and you have 10 squashed files... or i'm wrong? But the scprit deletes only the PORTAGE_SQFS and it's a symlink.

Oh, and I had a problem when didn't sync (IIRC ;-), the script during a reboot created "portage" (4096 bytes) and a symlink to it...

synss wrote:I do not think there should be any problem:
  • 1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" :) can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
    2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.
What about kernel change? You have to rebuild the unionfs module, but how to do it without a portage tree? When you boot the new kernel, you don't have a unionfs module. Or i'm missing something in here?...
Again IIRC when i was having kernel panic, kernel change and then "brand new kernel" the unionfs module at boot was

Code: Select all

Loading module unionfs           [!!]
No module called `unionfs`       [!!]
Then the startup script printed "no module called unionfs" and i had no portage.

PS. Sorry for all those "IIRC" but this was a long weekend with "fight with hardware". /-:
roslin uberlay | grubelek
Top
gAzo0o
n00b
n00b
Posts: 22
Joined: Wed Feb 22, 2006 12:51 pm

  • Quote

Post by gAzo0o » Mon Jul 17, 2006 4:46 pm

I had the same problem. Fixed it by setting the PORTDIR (/usr/portage by default) varible in /etc/make.conf
Top
lazy_bum
l33t
l33t
User avatar
Posts: 691
Joined: Wed Feb 16, 2005 8:55 am

  • Quote

Post by lazy_bum » Sat Jul 22, 2006 5:07 pm

Looks like it was my "kernel panic" problem.
The only problem left is 5-6 portage-$(date +%F).sqfs after 5-6 days. (-;

::edit::
And i will try again version 0.14 of the script.
roslin uberlay | grubelek
Top
Mousee
Apprentice
Apprentice
User avatar
Posts: 291
Joined: Mon Mar 29, 2004 7:14 am
Location: Illinois, USA

  • Quote

Post by Mousee » Tue Aug 01, 2006 11:27 pm

Not working at all here...

Code: Select all

HyperonPS ~ # /etc/init.d/squash_portage start
 * Mounting Squashfs'ed Portage Tree... ...
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
                                                                          [ ok ]
Going back to the Wiki one, it works 10x as good as this one.
Thanks tho :)
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

  • Quote

Post by synss » Wed Aug 02, 2006 3:14 pm

Mousee wrote:Not working at all here...

Code: Select all

HyperonPS ~ # /etc/init.d/squash_portage start
 * Mounting Squashfs'ed Portage Tree... ...
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
                                                                          [ ok ]
Going back to the Wiki one, it works 10x as good as this one.
Thanks tho :)
If you do not understand what you are doing, better not do it, you are right. BTW, if you want to mount sth with squashfs and unionfs it helps having them installed.
Compress portage tree
Elog viewer
Autodetect swap
Top
adsmith
Veteran
Veteran
Posts: 1386
Joined: Sun Sep 26, 2004 6:59 pm
Location: NC, USA

  • Quote

Post by adsmith » Wed Aug 02, 2006 8:13 pm

I'm glad to see you've found a way to make unionfs work with it reliably. I'll give it a try shortly.
(My current solution is still to have my main server make the squahsed image for me on my weekly emerge --sync)

By the way, it is good to combine the new metadata system with squahsfs, so you don't just duplicate the data yet again.

To do this, add "-metadata-transfer" to FEATURES and add the line "portdbapi.auxdbmodule = cache.metadata_overlay.database" to /etc/portage/modules.

This avoids the /var/cache/edb metadata cache from being created, as that data already lives in /usr/portage/metadata in the squashed image.
Top
dfy
Tux's lil' helper
Tux's lil' helper
Posts: 79
Joined: Mon Apr 05, 2004 12:12 pm

  • Quote

Post by dfy » Thu Aug 03, 2006 12:43 am

I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?
Top
gerardo
Apprentice
Apprentice
User avatar
Posts: 228
Joined: Thu Feb 05, 2004 8:30 pm
Location: Belgium

  • Quote

Post by gerardo » Thu Aug 10, 2006 8:13 pm

I've created a BASH script to cleanup the portage squash files older than one month:

Code: Select all

#!/bin/bash
SQFILENAME=/var/tmp/portage-$(date -d "1 month ago" +"%Y-%m-%-d").sqfs
echo "Removing obsolete portage squash files:"
echo " all files before $SQFILENAME will be deleted."
for x in `ls /var/tmp/portage-20*.sqfs` ; do 
    if [[  $SQFILENAME > $x ]]  ; then
        rm -f $x
        echo "deleting $x..."
    fi
done
Windoze : Plug and Pay...
Top
jpmayer87
n00b
n00b
Posts: 51
Joined: Sun Mar 19, 2006 7:53 pm
Location: Troy, NY

Remove older squashfs images

  • Quote

Post by jpmayer87 » Mon Aug 14, 2006 7:13 pm

This modification to the init script removes the squashfs file that was in use before the unionfs was modified. It also handles the case where the new file is the same name as the old one.

Code: Select all

#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

depend() {
        need localmount
        #after vartmp
	}

start() {
        ebegin "Mounting squashfs'ed portage tree"
	        [ -d $PORTAGE_RO ] || mkdir -p $PORTAGE_RO
		einfo "mounting squashfs RO..."
	        mount -t squashfs -o loop,ro $PORTAGE_SQFS $PORTAGE_RO
		einfo "Done"
	        [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
	        mount -t unionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR         
	        eend 0
	}

stop() {                                                       
        ebegin "Updating portage tree"
	if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then
	einfo "Syncing the tree"
	PORTAGE_OLD=`/bin/readlink $PORTAGE_SQFS`
	einfo $PORTAGE_OLD
	[ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW
	mksquashfs $PORTDIR $PORTAGE_NEW -check_data 
	einfo "Checking for files to remove..."
	
	if [ $PORTAGE_OLD != $PORTAGE_NEW ]; then
	einfo "Different Days!";
	einfo "Removing old SquashFS..."; 
	echo "will remove $PORTAGE_OLD";
	rm -f $PORTAGE_OLD
	else
	einfo "Same Day!"
	fi

	[ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS
	ln -sf $PORTAGE_NEW $PORTAGE_SQFS
	else
	einfo "Nothing to do"
	fi
	eend 0
	ebegin "Unmounting the tree"
	umount $PORTDIR
	umount $PORTAGE_RO
	rm -rf $PORTAGE_RW
	eend 0
	}

questions, comments welcome. This was my first real attempt at modifying a bash script

JP
Top
B.marc
n00b
n00b
Posts: 40
Joined: Sun Oct 16, 2005 5:26 pm
Location: Braunschweig / Germany

  • Quote

Post by B.marc » Sat Oct 07, 2006 3:55 pm

Just wanted to say, that this works for me very good. Now I'm wondering why Gentoo does not provide portage also as squashfs-files. Downloading 40MB to 50 MB and mounting it is much faster than a standard sync!

gerardo wrote:I've created a BASH script to cleanup the portage squash files older than one month:

Code: Select all

#!/bin/bash
SQFILENAME=/var/tmp/portage-$(date -d "1 month ago" +"%Y-%m-%-d").sqfs
echo "Removing obsolete portage squash files:"
echo " all files before $SQFILENAME will be deleted."
for x in `ls /var/tmp/portage-20*.sqfs` ; do 
    if [[  $SQFILENAME > $x ]]  ; then
        rm -f $x
        echo "deleting $x..."
    fi
done
Have a look at "man find". find is really good for tasks like this. And much more elegant:

Code: Select all

find /var/tmp -maxdepth 1 -name *.sqfs -mtime 30 -exec rm -f {} \;
Cheers
Marc
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

  • Quote

Post by synss » Tue Oct 10, 2006 5:13 pm

dfy wrote:I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?
Only the first time is time consuming, thereafter, everything is cached and faster, that is another advantage of having the tree compressed, the tree is in RAM. See the other thread for discussion.
Compress portage tree
Elog viewer
Autodetect swap
Top
mv
Watchman
Watchman
User avatar
Posts: 6795
Joined: Wed Apr 20, 2005 12:12 pm

Re: TIP: Compressing portage using squashfs: initscript meth

  • Quote

Post by mv » Fri Oct 20, 2006 12:08 am

I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:
  • synss wrote:

    Code: Select all

    PORTAGE_RW="/dev/shm/portage"
    Using a fixed name in a world-writable directory is a serious security issue. In this case, it might be an acceptable risk, because the initscript is hopefully started before any local attack might have a chance to get started, but even then: Don't better even think about calling the script later on with a restart option.
  • There are two other disadvantages of keeping PORTAGE_RW in a RAM disk: First, your previous changes to the portage tree are lost when your system crashes and the script cannot shutdown properly. The second disadvantage is that - surprise - it costs valuable RAM; the point is that you get practically no speed improvement from this ramdisk here (at least in normal cases), because the newly read data from a portage tree will remain in your disk ram anyway. So, it seems better not to use a ramdisk at all for portage_rw. In contrast, during shutdown, if you do not want to keep a backup of your .sqfs file or are short of disk space, it might be a good idea to build the new .sqfs file in a ramdisk.
  • It was already mentioned in this thread that keeping backups of older .sqfs files without limiting their number automatically is not necessarily the best idea. Morever, it seems better to use .tar.bz2 as a backup anyway for two reasons: First, these files need less space than .sqfs and, more importantly, if you are forced to boot from a kernel without squashfs support (e.g. from some rescue disk), you can still unpack the .tar.bz2 file if necessary. On the other hand, the disadvantage of using .tar.bz2 files as backup is of course the double packing time.
  • If you build a new kernel, you can simply compile in squashfs and loop support (or build the modules). However, to build in the unionfs support, you need access to the portage tree (unless you made a copy of the ebuild in advance). It was already mentioned that in this case you still can mount the portage tree at least readonly. However, the current script does not mount the tree in this case to the "usual" place readonly.
In the initscript of http://www.mathematik.uni-wuerzburg.de/ ... index.html there is a different attempt of a corresponding squashfs_portage with the above fixes/enhancements. Concerning the backups, you may choose in the configuration between keeping .sqfs or .tar.bz2, and you may pass options to mv to define the method and number of backups.

Edit: Fix some typos.
Last edited by mv on Fri Oct 20, 2006 10:07 am, edited 1 time in total.
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

Re: TIP: Compressing portage using squashfs: initscript meth

  • Quote

Post by synss » Fri Oct 20, 2006 4:23 am

mv wrote:I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:
etc.
Interesting comments. I will see what you have done. I also have a small bash script, which takes care of making the first image, I wanted to incorporate it in my initscript. Having only one backup, following what has been posted here by sbdy else, is probably a good idea. I also realized recently that my overlay directory is getting bigger. It does not make much sense to have a compressed portage and a large overlay, that is another issue, I (or you) may take into account.
Compress portage tree
Elog viewer
Autodetect swap
Top
synss
Apprentice
Apprentice
User avatar
Posts: 282
Joined: Wed Mar 08, 2006 5:42 pm
Location: Dijon > Berlin > Tokyo > Nürnberg > München

  • Quote

Post by synss » Fri Oct 20, 2006 3:30 pm

Interesting, yes, but you could mention me in the header for example...

if you want to make the script more complex, you definitely should have something like

Code: Select all

#!/bin/sh

archive="${1}"
sqfs_new="${2}"
tmpdir="./portage-tmp"

mkdir $tmpdir
tar -xvjf "${archive}" -C $tmpdir
mksquashfs ./$tmpdir/portage/ $sqfs_new -check_data
rm -ri ./$tmpdir
ln -sf $sqfs_new portage.sqfs

/etc/init.d/squash_portage restart
in your functions, adapted slightly to fit in. I have used that when I did not have a direct access to the Internet, had to downlad the tree, then unpack/squash the tree. It is very possible that the output of the

Code: Select all

tar -x
shall be piped into

Code: Select all

mksquashfs
to save some time, but I wrote that one in 2 min and let it-left it. I may update my script when I have time. Take it if you want.

Honestly, I find things like

Code: Select all

my_sqfsd () {
	local MY_TMP
	MY_TMP="$(mktemp "${TMPDIR}/squash_portage.XXXXXX")"
	chmod 644 "${MY_TMP}"
	mksquashfs "${PORTDIR}" "${MY_TMP}" \
		-check_data -noappend >/dev/null \
		&& my_mv_sqfs_old && mv -f "${MY_TMP}" "${PORTAGE_SQFS}"
}
difficult to read... but maybe it is just me...

Anyway thank you for pointing at the security issues and for updating the script. But you may mention me...
Compress portage tree
Elog viewer
Autodetect swap
Top
mv
Watchman
Watchman
User avatar
Posts: 6795
Joined: Wed Apr 20, 2005 12:12 pm

  • Quote

Post by mv » Sun Oct 22, 2006 5:03 pm

synss wrote:you could mention me in the header for example...
You are right, of course. I had done so in the beginning, but at some point, I decided to do a rewrite from scratch and copied the header just from somewhere else. This bug is repaired now.
if you want to make the script more complex, you definitely should have something like [...]
I don't get the point of this function: As I guess from the last command, you mean that you already have mounted some portage-tree, but it is just not up-to-date, and you want to update it from a .tar.bz2 file? Then why don't you just execute

Code: Select all

cd /usr/portage && rm -rf *
tar xjvf /path_to_file_with_tree.tar.bz2
/etc/init.d/squash_portage restart
(actually, the last command is not really needed, as it will be handled automatically on shutdown, but it might be useful if you are short of diskspace).
Honestly, I find things like

Code: Select all

my_sqfsd () {
	local MY_TMP
	MY_TMP="$(mktemp "${TMPDIR}/squash_portage.XXXXXX")"
	chmod 644 "${MY_TMP}"
	mksquashfs "${PORTDIR}" "${MY_TMP}" \
		-check_data -noappend >/dev/null \
		&& my_mv_sqfs_old && mv -f "${MY_TMP}" "${PORTAGE_SQFS}"
}
difficult to read...
They are difficult to read, but this is the way required by shell-programming- there is not much one can do to make it better readable: The quotation signs are all necessary to make sure that also file names with spaces work correctly. The "&&" syntax can hardly be rewritten with "if" if the exitcode should be correct. OK, one might split the last line into two and/or change its indentation. Not sure whether this is better readable...

Since you mentioned overlays in an earlier posting: If you keep your overlays in /usr/portage/local (which is e.g. the default of layman), then you need no special treatment for them... [only the rm-command in my above suggestion should then be modified, of course, although it is not really a mess if you executed it by accident, as long as you didn't call "/etc/init.d/squash_portage restart"].
Top
Post Reply

264 posts
  • Page 1 of 11
    • Jump to page:
  • 1
  • 2
  • 3
  • 4
  • 5
  • …
  • 11
  • Next

Return to “Documentation, Tips & Tricks”

Jump to
  • Assistance
  • ↳   News & Announcements
  • ↳   Frequently Asked Questions
  • ↳   Installing Gentoo
  • ↳   Multimedia
  • ↳   Desktop Environments
  • ↳   Networking & Security
  • ↳   Kernel & Hardware
  • ↳   Portage & Programming
  • ↳   Gamers & Players
  • ↳   Other Things Gentoo
  • ↳   Unsupported Software
  • Discussion & Documentation
  • ↳   Documentation, Tips & Tricks
  • ↳   Gentoo Chat
  • ↳   Gentoo Forums Feedback
  • ↳   Duplicate Threads
  • International Gentoo Users
  • ↳   中文 (Chinese)
  • ↳   Dutch
  • ↳   Finnish
  • ↳   French
  • ↳   Deutsches Forum (German)
  • ↳   Diskussionsforum
  • ↳   Deutsche Dokumentation
  • ↳   Greek
  • ↳   Forum italiano (Italian)
  • ↳   Forum di discussione italiano
  • ↳   Risorse italiane (documentazione e tools)
  • ↳   Polskie forum (Polish)
  • ↳   Instalacja i sprzęt
  • ↳   Polish OTW
  • ↳   Portuguese
  • ↳   Documentação, Ferramentas e Dicas
  • ↳   Russian
  • ↳   Scandinavian
  • ↳   Spanish
  • ↳   Other Languages
  • Architectures & Platforms
  • ↳   Gentoo on ARM
  • ↳   Gentoo on PPC
  • ↳   Gentoo on Sparc
  • ↳   Gentoo on Alternative Architectures
  • ↳   Gentoo on AMD64
  • ↳   Gentoo for Mac OS X (Portage for Mac OS X)
  • Board index
  • All times are UTC
  • Delete cookies

© 2001–2026 Gentoo Foundation, Inc.

Powered by phpBB® Forum Software © phpBB Limited

Privacy Policy