TIP: Compressing portage using squashfs: initscript method

[mv]

New POSIX compatible release 5.2

(For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).

baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms.

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

[steveL]

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

Woot! Well done mv :D

[synss]

New POSIX compatible release 5.2

(For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).

baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms.

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

That was fast, thanks!

EDIT: just to say I have just tried it and it works well with dash. (But my system does not really boot faster.)

[mv]

There is a new version of the initscript available at the initscripts. (The most important bugfix is that an unmount error is now not mistakenly ignored.)

Moreover, also a simple script can be found there which treats several of the corresponding initscripts simultaneously, if you want to recompress/start/stop/view the status of them manually.

Finally, there is also a patch for squashfs-tools-3.2_p2 which redirects the progress bar of mksquashfs to stderr. The advantage is that the init-script will then display the progress bar without printing additional confusing information (but you can also have the latter without the patch with the new version of the script).

If you use the script note that squashfs-tools-3.3 is meanwhile in the portage tree but will create a format which seems to be incompatible with the current gentoo kernel patchset, so you should not upgrade to squashfs-tools-3.3 in the moment...

[mv]

There is again a new version of the script available. Now it is possible to ignore certain files/dirs resp. ignore "touches" of these files/dirs (in the main directory). This is important if you squash a tex directory and want to avoid unnecessary re-squashing just because one of the system tools updated "ls-R" without actually making any changes to that file.

[Darknight]

Thanks for this nice script.

[188562]

synss Thank you very much for the idea and its embodiment!
My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines

#mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

After that, everything worked! The truth is now at the time of each file portage.sqfs re-rebooting again this takes some time.

[synss]

synss Thank you very much for the idea and its embodiment!
My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines

#mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

After that, everything worked! The truth is now at the time of each file portage.sqfs re-rebooting again this takes some time.

Thank you! now I am not sure I understand your last sentence correctly: you do not need to reboot to update the image, you only need to restart the initscript, which can be done in a monthly cron job, too, BTW.

[188562]

Every time I do a "reboot" or "halt" or "poweroff" script re-creates file portage.sqfs so if I update portage "emerge --sync"
Most likely this feature aufs but I'm not sure.

[IvanMajhen]

Add this

Code: Select all

stop() {
   ebegin "Updating portage tree"
   rm -rf /dev/shm/.portage-rw/.wh..wh.plink
   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
   then

[188562]

So here is the corrected version of the script for squashfs & aufs. All except out in Section stop working as it should work.

Code: Select all

#!/sbin/runscript

source /etc/make.globals
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"

depend() {
    need localmount
}

start() {
    ebegin "Mounting read-only squashfs image"
    mount -rt squashfs -o loop $SQFS_CUR $PORTDIR
    retval=$?
    eend $retval
    [ $retval -ne 0 ] && return $retval

    ebegin "Mounting read-write with aufs"
    if [ ! $PORTAGE_RW ]
	then einfo " mounted in tmpfs (RAM)"
	PORTAGE_RW="${DEF_RW}"
    fi
    [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
    chmod 0750 $PORTAGE_RW
    chown portage:portage $PORTAGE_RW
    mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR
    eend $?
}

stop() {
#   ebegin "Updating portage tree"
#   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
#   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
#   then
#      einfo "  Syncing the tree"
#      mv -f $SQFS_NEW $SQFS_OLD
#      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
#      retval=$?
#      ln -sf $SQFS_NEW $SQFS_CUR
#   else
#      einfo "  Nothing to do"
#      retval=0
#   fi
#   eend $retval

    ebegin "Unmounting the tree"
    umount -t aufs $PORTDIR
    umount -t squashfs $PORTDIR
    rm -rf $PORTAGE_RW	
    eend 0
}

Here is this condition (if) why it is always true

# ebegin "Updating portage tree"
# [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
# if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]

[IvanMajhen]

This is wrong. Sqfs wont be updated when synced.

This is how my stop function looks:

Code: Select all

stop() {
   ebegin "Updating portage tree"
   rm -rf /dev/shm/.portage-rw/.wh..wh.plink
   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
   then
      einfo "  Syncing the tree"
      mv -f $SQFS_NEW $SQFS_OLD
      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
      retval=$?
      ln -sf $SQFS_NEW $SQFS_CUR
   else
      einfo "  Nothing to do"
      retval=0
   fi
   eend $retval

   ebegin "Unmounting the tree"
   umount -t aufs  $PORTDIR
   umount -t squashfs $PORTDIR
   rm -rf $PORTAGE_RW
   eend 0
}

[NaiL]

I have using the MV's script for some time, (currently the last version).

I found a problem stopping the init script.

/etc/init.d/squash_portage start
* Mounting /var/tmp/portage-current.sqfs as /usr/portage ... [ ok ]

mount
/dev/sda3 on / type reiserfs (rw,noatime,notail)
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,gid=5,mode=620)
shm on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
securityfs on /sys/kernel/security type securityfs (rw,noexec,nosuid,nodev)
/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)
aufs on /usr/portage type aufs (rw,xino=/dev/shm/portage/.aufs.xino,br:/dev/shm/portage=rw:/tmp/squash_dir.readonly.4TQ2X8=ro)

# /etc/init.d/squash_portage stop
* Unmounting /usr/portage ...
umount: /var/tmp/portage-current.sqfs: not mounted
* umount -d -- /var/tmp/portage-current.sqfs failed [exit with 1] [ !! ]

mount
/dev/sda3 on / type reiserfs (rw,noatime,notail)
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,gid=5,mode=620)
shm on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
securityfs on /sys/kernel/security type securityfs (rw,noexec,nosuid,nodev)
/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)

# cat /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

DIRECTORY="/usr/portage"
DIR_CHANGE="/dev/shm/portage"
FILE_SQFS="/var/tmp/portage-current.sqfs"
FILE_SQFS_OLD="/var/tmp/portage-old.sqfs"
DIR_TMP="/dev/shm"

Maybe is it a problem of configuration?

[Darknight]

I have the same problem. As far as I know it used to work perfectly until some days ago but I may have missed that error for some time. Unfortunately I can't pinpoint any update, nor change the script to make it work with the new stuff...

EDIT: It has to do with temporary filenames. Set DIR_SQUASH as a workaround.

[mv]

I have the same problem

Sorry, somehow I had missed NaiL's posting.

Anyway, I cannot reproduce the problem on my machine if I use exactly NaiL's config (only with a different DIRECTORY for testing purposes).
Do you use the most current version (7.4) of the script? (The version is in a comment near the beginning).

However, as remarked in the documentation, using the temporary filename feature for DIR_SQUASH is always hackish: It depends on the undocumented /etc/mtab format and thus it might depend on the mount command (i.e. on the util-linux version) whether this will work. During my test just now I had util-linux-2.14.1 installed: The critical part is whether in line 686-688 the correct filename can be read from /etc/mtab by the sed-expression.

[Darknight]

I have the same util-linux version but somehow sed fails (I had nailed it, but I have not bash skillz to modify the script).
Anyway re-reading through the documentation gave me a solution that is good enough for me.

[NaiL]

Hi,

I'm using the 7.4 version of the script and:

sys-apps/util-linux-2.14.1
sys-fs/squashfs-tools-3.3
sys-fs/aufs-20081208-r1
sys-kernel/gentoo-sources-2.6.27-r8

Do you need any other information?

[mv]

NaiL, apparently, the problem is your strange output of

/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)

This shouldn't be so. It should be instead something like

/var/tmp/portage-current.sqfs on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro,loop=/dev/loop0)

I cannot understand why your output is so. What happens after you do manually

Code: Select all

mkdir /some_directory
mount -t squashfs -o loop,ro /var/tmp/portage-current.sqfs /some_directory

If this works as expected, the output of mount should then contain

/var/tmp/portage-current.sqfs on /some_directory type squashfs (ro,loop=/dev/loop*)

and afterwards you should be able to umount by

Code: Select all

umount /var/tmp/portage-current.sqfs

(of course, afterwards, you can remove /some_directory).

Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?

[NaiL]

I cannot understand why your output is so. What happens after you do manually

Code: Select all

mkdir /some_directory
mount -t squashfs -o loop,ro /var/tmp/portage-current.sqfs /some_directory

If this works as expected, the output of mount should then contain

/var/tmp/portage-current.sqfs on /some_directory type squashfs (ro,loop=/dev/loop*)

and afterwards you should be able to umount by

Code: Select all

umount /var/tmp/portage-current.sqfs

(of course, afterwards, you can remove /some_directory).

The result was:

/dev/loop1 on /some_directory type squashfs (ro)

Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?

Affter the upgrade i get the same output.

[mv]

NaiL, I have no idea what is responsible on your system that the same mount-command (with same versions of mount and squashfs than here) works differently on your system. Do you have perhaps some of these terrible hal things activated or some special udev rules which could cause this behavior? Here are my useflags for util-linux:

Code: Select all

crypt loop-aes nls slang unicode -old-linux -selinux -uclibc

[NaiL]

Tnx! I finally found the problem.

I was some how related to "loop-aes" USE flag.

I had this USE flag disabled. By some reason it has to be activated, may be it will be good to document that some where.

I also noticed that.

* The loop-aes code has been split out of USE=crypt and into USE=loop-aes.
* If you need support for it, make sure to update your USE accordingly.

[mv]

In the new release of the script the temporary file name feature was removed for DIR_SQUASH. Instead, now a sane default is chosen for DIR_SQUASH in /var/run. Indeed, since even with the temporary file name feature the thing could only work by using a non-temporary file (in this case /etc/mtab), it is probably more reasonable to force directly to use a non-temporary directory. In this way the implementation is independent of undocumented behavior of the mount command.

[jowr]

Where can I get a squashfs utils that makes a SquashFS 4 file system? Apparently 2.6.29 now only has v4, and there are no [known to me..] userland utils for making v4 file systems.

[Dont Panic]

I've just upgraded to a 2.6.29 kernel, and I am now getting errors like this when trying to mount the squashfs:

Code: Select all

SQUASHFS error: Major/Minor mismatch, older Squashfs 3.1 filesystems are unsupported

I may not be understanding correctly, but the information I'm reading on the Internet indicates that backward compatibility has been broken for squashfs as part of the process for bringing squashfs into the mainline kernel.

Here's a post from the author of squashfs:

http://lkml.indiana.edu/hypermail/linux ... 02601.html

I'm not finding much information about this on squashfs' homepage on sourceforge

And I also see the entire set of sys-fs/squashfs-tools ebuilds have been hard-masked (as of the time of this post).

This looks like a bumpy ride for my squashfs portage directory.

[Darknight]

When I tried aufs didn't compile against 2.6.29, therefore I'm back to 2.6.27 for now.