Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3 ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Thu May 25, 2006 2:24 pm    Post subject: TIP: Compressing portage using squashfs: initscript method Reply with quote

Admin Edit:
squashmount related information has been split from this topic. It is now here:
Compressing filesystems with squashfs and squashmount
--pjp


Tue Nov 28 12:10:10 JST 2006, v. 0.1.5(4)

I have left gentoo and have not updated this script for a while, though it probably still works, mv has taken over and proposed new features and improvements, aufs is one good thing among others. And his script can be used on any directory (I would recommend /usr/src/linux and the docs) You should probably have a look at the thread and check mv's version, now. Last version is available here --- I was sure I had left a notice like this one here but only realize now I must have forgotten...

Abstract The portage tree takes typically 600MB on /usr/portage, some people have chosen to have it on a separate partition with a small block size to
    decrease its global size
    decrease fragmentation
    increase the speed during searches and update
However that also means that the free space on that partition is lost. Another, better, solution is to use a stack file containing the filesystem with small block size to avoid the shortcomings of the separate partition. Following the discussion in this post, the best solution found is to compress the portage tree using the compressed, read-only squashfs filesystem and mount a read-write partition on top of it using unionfs. Second, for convenience, the portage tree should be mountable on boot-up and updated on shutdown, hence I have written an initscript to automate this process. I will publish and update this program here.

Assumptions I will assume that you have
    read the original idea as it details what you need to get started and that you are familiar with the bootup/shutdown sequence of your computer,
    read the Gentoo doc on initscripting.
    DISTDIR should be out of the tree, etc.
Please just read the original post as everything is well documented there and come back once you have done that, unless you do understand what you are doing!

Introduction In brief, you will need to load the relevant kernel modules and emerge the userland utilities
Code:
emerge -avt sys-fs/unionfs sys-fs/squashfs-tools
, unmask if needed (I am now successfully using sys-fs/unionfs-1.2 on sys-kernel/suspend2-sources-2.6.16-r8 and sys-fs/squashfs-tools-3.0)
Code:
modprobe loop squashfs unionfs
add the modules to /etc/modules.autoload.d/kernel-2.6.
the script
/etc/conf.d/squash_portage:
# /etc/conf.d/squash_portage

# SQFS_DIRNAME points to the directory that will contain the sqfs
# images, recommended value is /var/tmp
SQFS_DIRNAME="/var/tmp"

# Leave PORTAGE_RW empty for use with tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
PORTAGE_RW=""

and
/etc/init.d/squash_portage:
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# /etc/init.d/squash_portage allows efficient compression of
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/unionfs are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin @ users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)

source /etc/make.conf
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"

depend() {
   need localmount
}

start() {
   ebegin "Mounting read-only squashfs image"
   mount -rt squashfs -o loop,nodev,noexec $SQFS_CUR $PORTDIR
   retval=$?
   eend $retval
   [ $retval -ne 0 ] && return $retval

   ebegin "Mounting read-write with unionfs"
   if [ ! $PORTAGE_RW ]
   then
      einfo "  mounted in tmpfs (RAM)"
      PORTAGE_RW="${DEF_RW}"
   fi
   [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
   chmod 0750 $PORTAGE_RW
   chown portage:portage $PORTAGE_RW
   mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
   eend $?
}

stop() {
   ebegin "Updating portage tree"
   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
   then
      einfo "  Syncing the tree"
      mv -f $SQFS_NEW $SQFS_OLD
      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
      retval=$?
      ln -sf $SQFS_NEW $SQFS_CUR
   else
      einfo "  Nothing to do"
      retval=0
   fi
   eend $retval

   ebegin "Unmounting the tree"
   umount -t unionfs  $PORTDIR
   umount -t squashfs $PORTDIR
   rm -rf $PORTAGE_RW
   eend 0
}



That obviously is bash as any initscript. Make it executable and
Code:
/etc/init.d/squash_portage start
rc-update add squash_portage default


The script assumes that you already have an sqfs image on your disk ready to be mounted and called $SQFS_DIRNAME/portage.sqfs it will save the new image if you sync'ed the tree the rest is pretty obvious and the rw partition is mounted on /dev/shm by default for speed and simplicity. But beware that it means you will lose your update in case of power failure or if you are not caring. I would recommend to restart the script upon hibernation if you use it and do not always reboot into your ram image.
_________________
Compress portage tree
Elog viewer
Autodetect swap


Last edited by synss on Fri Sep 07, 2007 6:33 pm; edited 19 times in total
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 850
Location: PL

PostPosted: Thu May 25, 2006 11:40 pm    Post subject: Reply with quote

th...this is great! i would never think of implementing it via initscript :D

nice job.

although there is one drawback with this overlay - when you upgrade your kernel you might get stuck with no portage tree at all, since not every kernel might have squashfs and unionfs modules integrated. there could be a solution in form of an mini-overlay that would contain only ebuilds for squashfs and unionfs (and required eclasses) for forgetful folks. (would it be difficult to make such a backup overlay from an existing portage tree via an bash/sh script?)

sorry for writing that here, but i thought is fits this topic more - it's a more elegant solution.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Fri May 26, 2006 2:40 am    Post subject: Reply with quote

Thank you!!! :D

No problem for writing here either...

I do not think there should be any problem:
    1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" :) can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
    2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.

I have had a problem with union that was acting weird during unmount, but that was a mistake from my side (a non-existing locale when compiling glibc, I think) So you might be unable to sync, but you can then compile and load the modules and go on. In any case, it is always possible to download an image of the tree from a nearby ftp server :wink: yes, that is exactly for avoiding that that I implemented the incremental timestamped backups while debugging. (Then I leave it cause it is a good idea and you can easily clean the backups via cron if you want to.) So the only risk you are taking, AFAIK, is that you might have to mount the squashed image by hand after recompiling the modules.

Thank you again!
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 691

PostPosted: Thu Jun 29, 2006 6:48 am    Post subject: Reply with quote

Thanks man, this is great! Fast, small and working! (-:

I'm using sys-fs/squashfs-tools-2.2_p2, sys-fs/unionfs-1.1.4-r2 on a sys-kernel/gentoo-sources-2.6.14-r5
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
MxxCon
n00b
n00b


Joined: 01 Apr 2005
Posts: 31

PostPosted: Thu Jun 29, 2006 6:35 pm    Post subject: Reply with quote

have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Thu Jun 29, 2006 6:45 pm    Post subject: Reply with quote

MxxCon wrote:
have you guys considered using http://www.miio.net/fusecompress/ or http://parallel.vub.ac.be/~johan/compFUSEd/ instead of squashfs?
from what i understand for squashfs you still need to have enough free space to unpack the whole thing, while with these 2 it's like working with compressed .tar file.

with squashfs you don't need the free space to unpack it, and these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
MxxCon
n00b
n00b


Joined: 01 Apr 2005
Posts: 31

PostPosted: Thu Jun 29, 2006 6:55 pm    Post subject: Reply with quote

Gergan Penkov wrote:
these do not work as one compressed tar file they work as many small compressed files (or have been working in this way some two months ago), which IMHO makes the whole idea to compress the tree futile.
hmm i must have misread something then. i assumed they would create a single compressed file from a dir tree.
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Thu Jun 29, 2006 7:02 pm    Post subject: Reply with quote

Well I though the same in the beginning, even with such implementation the things would have been just ok if there was some sort of file-container ala tar around the files, of course the compression ratio would not have been very good but there would have been virtually not that much fragmentation and loss of free space :)
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 691

PostPosted: Sat Jul 01, 2006 4:25 pm    Post subject: Reply with quote

Something is wrong with the new version:

Code:
cat /etc/conf.d/squash_portage
source /etc/make.conf

PORTAGE_BASENAME="/usr/portage"
PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs"
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"
PORTAGE_RO=$PORTDIR
PORTAGE_RW="/dev/shm/portage"


And, when running the script:

Code:
/etc/init.d/squash_portage start
 * Mounting squashfs'ed portage tree ...
Usage: mount -V                 : print version
       mount -h                 : print this help
       mount                    : list mounted filesystems
*snip*
A device can be given by name, say /dev/hda1 or /dev/cdrom,
or by label, using  -L label  or by uuid, using  -U uuid .
Other options: [-nfFrsvw] [-o options] [-p passwdfd].
For many more details, say  man 8 mount .                                 [ ok ]

... and /usr/portage/ is empty...

I have squashed portage in /usr/portage.sqfs
And i wonder what "date" is doing in here:
Code:
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"

Is it possible that a symlink is created to nothing? When i stop the script day after sync'ing the tree, and the date is different?

Hope it's usefull.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Mon Jul 03, 2006 3:44 pm    Post subject: Reply with quote

"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...

I am in a cybercafe on XP with a French kbd so debugging is not easy... But I do not see what else but the first mount can be a problem.

you can also try to mount the tree by hand, sync and copy-paste
Code:
source /etc/make.conf             

PORTAGE_BASENAME="/usr/portage"
PORTAGE_SQFS="$PORTAGE_BASENAME.sqfs"
PORTAGE_NEW="$PORTAGE_BASENAME-$(date +%F).sqfs"
PORTAGE_RO=$PORTDIR
PORTAGE_RW="/dev/shm/portage"

if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then
                [ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW
                mksquashfs $PORTDIR $PORTAGE_NEW -check_data
                [ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS
                ln -sf $PORTAGE_NEW $PORTAGE_SQFS
        fi
        umount $PORTDIR
        umount $PORTAGE_RO
        rm -rf $PORTAGE_RW


that should make sth clean for starting the initscript and use it and forget it.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 691

PostPosted: Tue Jul 11, 2006 6:47 am    Post subject: Reply with quote

synss wrote:
"date" creates incremental backups

What about moving the squashed tree to, ie /usr/portage_tree.sqfs and creating a symlink to it, ln -s /usr/portage_tree.sqfs /usr/portage.sqfs the second one should only be a symlink as it is overwritten on updates to point to the actual tree.

I do not know whether that helps... Could you make sure it is the first mount that is broken? and check that something is mounted by echoing the variables for example. I am sorry for the inconvenience, it works at home...

Well, i don't know if it wasn't my computer problem. Had few day of "fun" with kernel panic and stuff...
But maybe it would be better to move previous portage to something like "portage-old.sqfs" and the new one would be "portage-current.sqfs". It would help for the "forgetful folks" to autodelete and keep only one portage (that you use) and one (as a backup). Otherwise few days and you have 10 squashed files... or i'm wrong? But the scprit deletes only the PORTAGE_SQFS and it's a symlink.

Oh, and I had a problem when didn't sync (IIRC ;-), the script during a reboot created "portage" (4096 bytes) and a symlink to it...


synss wrote:
I do not think there should be any problem:
    1. squashfs is a standard kernel module, which does not require the userland utilities, if you have a recent kernel, you just enable squashfs and loop and that is enough for mounting the squashed image. Even "forgetful folks" :) can compile the necessary modules and modprobe them (no need to reboot into the new kernel).
    2. unionfs and sys-fs/squashfs-tools are only needed for the updates because the squashed image is read only. Hence, it is not required at all as long as you do not try to sync your tree.

What about kernel change? You have to rebuild the unionfs module, but how to do it without a portage tree? When you boot the new kernel, you don't have a unionfs module. Or i'm missing something in here?...
Again IIRC when i was having kernel panic, kernel change and then "brand new kernel" the unionfs module at boot was
Code:
Loading module unionfs           [!!]
No module called `unionfs`       [!!]

Then the startup script printed "no module called unionfs" and i had no portage.

PS. Sorry for all those "IIRC" but this was a long weekend with "fight with hardware". /-:
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
gAzo0o
n00b
n00b


Joined: 22 Feb 2006
Posts: 22

PostPosted: Mon Jul 17, 2006 4:46 pm    Post subject: Reply with quote

I had the same problem. Fixed it by setting the PORTDIR (/usr/portage by default) varible in /etc/make.conf
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 691

PostPosted: Sat Jul 22, 2006 5:07 pm    Post subject: Reply with quote

Looks like it was my "kernel panic" problem.
The only problem left is 5-6 portage-$(date +%F).sqfs after 5-6 days. (-;

::edit::
And i will try again version 0.14 of the script.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
Mousee
Apprentice
Apprentice


Joined: 29 Mar 2004
Posts: 291
Location: Illinois, USA

PostPosted: Tue Aug 01, 2006 11:27 pm    Post subject: Reply with quote

Not working at all here...
Code:

HyperonPS ~ # /etc/init.d/squash_portage start
 * Mounting Squashfs'ed Portage Tree... ...
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
                                                                          [ ok ]

Going back to the Wiki one, it works 10x as good as this one.
Thanks tho :)
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Wed Aug 02, 2006 3:14 pm    Post subject: Reply with quote

Mousee wrote:
Not working at all here...
Code:

HyperonPS ~ # /etc/init.d/squash_portage start
 * Mounting Squashfs'ed Portage Tree... ...
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
                                                                          [ ok ]

Going back to the Wiki one, it works 10x as good as this one.
Thanks tho :)


If you do not understand what you are doing, better not do it, you are right. BTW, if you want to mount sth with squashfs and unionfs it helps having them installed.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Wed Aug 02, 2006 8:13 pm    Post subject: Reply with quote

I'm glad to see you've found a way to make unionfs work with it reliably. I'll give it a try shortly.
(My current solution is still to have my main server make the squahsed image for me on my weekly emerge --sync)

By the way, it is good to combine the new metadata system with squahsfs, so you don't just duplicate the data yet again.

To do this, add "-metadata-transfer" to FEATURES and add the line "portdbapi.auxdbmodule = cache.metadata_overlay.database" to /etc/portage/modules.

This avoids the /var/cache/edb metadata cache from being created, as that data already lives in /usr/portage/metadata in the squashed image.
Back to top
View user's profile Send private message
dfy
Tux's lil' helper
Tux's lil' helper


Joined: 05 Apr 2004
Posts: 79

PostPosted: Thu Aug 03, 2006 12:43 am    Post subject: Reply with quote

I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?
Back to top
View user's profile Send private message
gerardo
Apprentice
Apprentice


Joined: 05 Feb 2004
Posts: 228
Location: Belgium

PostPosted: Thu Aug 10, 2006 8:13 pm    Post subject: Reply with quote

I've created a BASH script to cleanup the portage squash files older than one month:
Code:
#!/bin/bash
SQFILENAME=/var/tmp/portage-$(date -d "1 month ago" +"%Y-%m-%-d").sqfs
echo "Removing obsolete portage squash files:"
echo " all files before $SQFILENAME will be deleted."
for x in `ls /var/tmp/portage-20*.sqfs` ; do
    if [[  $SQFILENAME > $x ]]  ; then
        rm -f $x
        echo "deleting $x..."
    fi
done

_________________
Windoze : Plug and Pay...
Back to top
View user's profile Send private message
jpmayer87
n00b
n00b


Joined: 19 Mar 2006
Posts: 51
Location: Troy, NY

PostPosted: Mon Aug 14, 2006 7:13 pm    Post subject: Remove older squashfs images Reply with quote

This modification to the init script removes the squashfs file that was in use before the unionfs was modified. It also handles the case where the new file is the same name as the old one.

Code:

#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

depend() {
        need localmount
        #after vartmp
   }

start() {
        ebegin "Mounting squashfs'ed portage tree"
           [ -d $PORTAGE_RO ] || mkdir -p $PORTAGE_RO
      einfo "mounting squashfs RO..."
           mount -t squashfs -o loop,ro $PORTAGE_SQFS $PORTAGE_RO
      einfo "Done"
           [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
           mount -t unionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR         
           eend 0
   }

stop() {                                                       
        ebegin "Updating portage tree"
   if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then
   einfo "Syncing the tree"
   PORTAGE_OLD=`/bin/readlink $PORTAGE_SQFS`
   einfo $PORTAGE_OLD
   [ -f $PORTAGE_NEW ] && rm -f $PORTAGE_NEW
   mksquashfs $PORTDIR $PORTAGE_NEW -check_data
   einfo "Checking for files to remove..."
   
   if [ $PORTAGE_OLD != $PORTAGE_NEW ]; then
   einfo "Different Days!";
   einfo "Removing old SquashFS...";
   echo "will remove $PORTAGE_OLD";
   rm -f $PORTAGE_OLD
   else
   einfo "Same Day!"
   fi

   [ -L $PORTAGE_SQFS ] && rm -f $PORTAGE_SQFS
   ln -sf $PORTAGE_NEW $PORTAGE_SQFS
   else
   einfo "Nothing to do"
   fi
   eend 0
   ebegin "Unmounting the tree"
   umount $PORTDIR
   umount $PORTAGE_RO
   rm -rf $PORTAGE_RW
   eend 0
   }



questions, comments welcome. This was my first real attempt at modifying a bash script

JP
Back to top
View user's profile Send private message
B.marc
n00b
n00b


Joined: 16 Oct 2005
Posts: 40
Location: Braunschweig / Germany

PostPosted: Sat Oct 07, 2006 3:55 pm    Post subject: Reply with quote

Just wanted to say, that this works for me very good. Now I'm wondering why Gentoo does not provide portage also as squashfs-files. Downloading 40MB to 50 MB and mounting it is much faster than a standard sync!


gerardo wrote:
I've created a BASH script to cleanup the portage squash files older than one month:
Code:
#!/bin/bash
SQFILENAME=/var/tmp/portage-$(date -d "1 month ago" +"%Y-%m-%-d").sqfs
echo "Removing obsolete portage squash files:"
echo " all files before $SQFILENAME will be deleted."
for x in `ls /var/tmp/portage-20*.sqfs` ; do
    if [[  $SQFILENAME > $x ]]  ; then
        rm -f $x
        echo "deleting $x..."
    fi
done


Have a look at "man find". find is really good for tasks like this. And much more elegant:
Code:

find /var/tmp -maxdepth 1 -name *.sqfs -mtime 30 -exec rm -f {} \;


Cheers
Marc
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Tue Oct 10, 2006 5:13 pm    Post subject: Reply with quote

dfy wrote:
I am considering this at the moment, but even creating the initial portage.sqfs images takes more than 10 minutes. Am I right to think that every time I shutdown my system and the initscript resyncs the squashfs, it will take just as long?


Only the first time is time consuming, thereafter, everything is cached and faster, that is another advantage of having the tree compressed, the tree is in RAM. See the other thread for discussion.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Fri Oct 20, 2006 12:08 am    Post subject: Re: TIP: Compressing portage using squashfs: initscript meth Reply with quote

I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:
  • synss wrote:
    Code:
    PORTAGE_RW="/dev/shm/portage"

    Using a fixed name in a world-writable directory is a serious security issue. In this case, it might be an acceptable risk, because the initscript is hopefully started before any local attack might have a chance to get started, but even then: Don't better even think about calling the script later on with a restart option.

  • There are two other disadvantages of keeping PORTAGE_RW in a RAM disk: First, your previous changes to the portage tree are lost when your system crashes and the script cannot shutdown properly. The second disadvantage is that - surprise - it costs valuable RAM; the point is that you get practically no speed improvement from this ramdisk here (at least in normal cases), because the newly read data from a portage tree will remain in your disk ram anyway. So, it seems better not to use a ramdisk at all for portage_rw. In contrast, during shutdown, if you do not want to keep a backup of your .sqfs file or are short of disk space, it might be a good idea to build the new .sqfs file in a ramdisk.

  • It was already mentioned in this thread that keeping backups of older .sqfs files without limiting their number automatically is not necessarily the best idea. Morever, it seems better to use .tar.bz2 as a backup anyway for two reasons: First, these files need less space than .sqfs and, more importantly, if you are forced to boot from a kernel without squashfs support (e.g. from some rescue disk), you can still unpack the .tar.bz2 file if necessary. On the other hand, the disadvantage of using .tar.bz2 files as backup is of course the double packing time.

  • If you build a new kernel, you can simply compile in squashfs and loop support (or build the modules). However, to build in the unionfs support, you need access to the portage tree (unless you made a copy of the ebuild in advance). It was already mentioned that in this case you still can mount the portage tree at least readonly. However, the current script does not mount the tree in this case to the "usual" place readonly.

In the initscript of http://www.mathematik.uni-wuerzburg.de/~vaeth/gentoo/index.html there is a different attempt of a corresponding squashfs_portage with the above fixes/enhancements. Concerning the backups, you may choose in the configuration between keeping .sqfs or .tar.bz2, and you may pass options to mv to define the method and number of backups.

Edit: Fix some typos.


Last edited by mv on Fri Oct 20, 2006 10:07 am; edited 1 time in total
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Fri Oct 20, 2006 4:23 am    Post subject: Re: TIP: Compressing portage using squashfs: initscript meth Reply with quote

mv wrote:
I like the idea to realize the task as an init-script very much. However, the current script has an security issue, and several enhancements are possible:
etc.


Interesting comments. I will see what you have done. I also have a small bash script, which takes care of making the first image, I wanted to incorporate it in my initscript. Having only one backup, following what has been posted here by sbdy else, is probably a good idea. I also realized recently that my overlay directory is getting bigger. It does not make much sense to have a compressed portage and a large overlay, that is another issue, I (or you) may take into account.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Fri Oct 20, 2006 3:30 pm    Post subject: Reply with quote

Interesting, yes, but you could mention me in the header for example...

if you want to make the script more complex, you definitely should have something like
new_sqfs.sh:
#!/bin/sh

archive="${1}"
sqfs_new="${2}"
tmpdir="./portage-tmp"

mkdir $tmpdir
tar -xvjf "${archive}" -C $tmpdir
mksquashfs ./$tmpdir/portage/ $sqfs_new -check_data
rm -ri ./$tmpdir
ln -sf $sqfs_new portage.sqfs

/etc/init.d/squash_portage restart


in your functions, adapted slightly to fit in. I have used that when I did not have a direct access to the Internet, had to downlad the tree, then unpack/squash the tree. It is very possible that the output of the
Code:
tar -x
shall be piped into
Code:
mksquashfs
to save some time, but I wrote that one in 2 min and let it-left it. I may update my script when I have time. Take it if you want.

Honestly, I find things like
Code:
my_sqfsd () {
   local MY_TMP
   MY_TMP="$(mktemp "${TMPDIR}/squash_portage.XXXXXX")"
   chmod 644 "${MY_TMP}"
   mksquashfs "${PORTDIR}" "${MY_TMP}" \
      -check_data -noappend >/dev/null \
      && my_mv_sqfs_old && mv -f "${MY_TMP}" "${PORTAGE_SQFS}"
}
difficult to read... but maybe it is just me...

Anyway thank you for pointing at the security issues and for updating the script. But you may mention me...
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Oct 22, 2006 5:03 pm    Post subject: Reply with quote

synss wrote:
you could mention me in the header for example...

You are right, of course. I had done so in the beginning, but at some point, I decided to do a rewrite from scratch and copied the header just from somewhere else. This bug is repaired now.
Quote:
if you want to make the script more complex, you definitely should have something like [...]

I don't get the point of this function: As I guess from the last command, you mean that you already have mounted some portage-tree, but it is just not up-to-date, and you want to update it from a .tar.bz2 file? Then why don't you just execute
Code:
cd /usr/portage && rm -rf *
tar xjvf /path_to_file_with_tree.tar.bz2
/etc/init.d/squash_portage restart

(actually, the last command is not really needed, as it will be handled automatically on shutdown, but it might be useful if you are short of diskspace).
Quote:
Honestly, I find things like
Code:
my_sqfsd () {
   local MY_TMP
   MY_TMP="$(mktemp "${TMPDIR}/squash_portage.XXXXXX")"
   chmod 644 "${MY_TMP}"
   mksquashfs "${PORTDIR}" "${MY_TMP}" \
      -check_data -noappend >/dev/null \
      && my_mv_sqfs_old && mv -f "${MY_TMP}" "${PORTAGE_SQFS}"
}
difficult to read...

They are difficult to read, but this is the way required by shell-programming- there is not much one can do to make it better readable: The quotation signs are all necessary to make sure that also file names with spaces work correctly. The "&&" syntax can hardly be rewritten with "if" if the exitcode should be correct. OK, one might split the last line into two and/or change its indentation. Not sure whether this is better readable...

Since you mentioned overlays in an earlier posting: If you keep your overlays in /usr/portage/local (which is e.g. the default of layman), then you need no special treatment for them... [only the rm-command in my above suggestion should then be modified, of course, although it is not really a mess if you executed it by accident, as long as you didn't call "/etc/init.d/squash_portage restart"].
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3 ... 9, 10, 11  Next
Page 1 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum