Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TRICKS] Speed up system with aufs + squashfs
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
kokoko3k
n00b
n00b


Joined: 06 May 2005
Posts: 72

PostPosted: Wed Mar 19, 2008 4:44 pm    Post subject: Reply with quote

First of all, Thanks!

Quote:
And yes, the OS would be caching the compressed sectors (not the uncompressed data) from the squash file every time you try to read something from /usr.

Are you sure about this?

Also, i wonder if it is possible to append rw files to the previously created squashfs file, instead of create a new one from zero (it's a time consuming task!), or to manage multiple squashed files (it should be possible, but i'm not really in aufs)
Back to top
View user's profile Send private message
stanbogdan
n00b
n00b


Joined: 14 Jan 2008
Posts: 12

PostPosted: Wed Mar 19, 2008 7:44 pm    Post subject: Reply with quote

kokoko3k wrote:
First of all, Thanks!

Quote:
And yes, the OS would be caching the compressed sectors (not the uncompressed data) from the squash file every time you try to read something from /usr.

Are you sure about this?


Hmmm, I will investigate further, I don't know for sure, but I feel that is what is happening, because of the following symptom: when doing the random access pattern, the squashed file system test was more than 3 times faster than the normal test (a LOT more), even though the compression factor was about 3x. I have no other way to explain this, other than the fact that the entire squashed image of my /usr folder (which is appx 1gb) could be loaded entirely into my RAM (1gb), one sector at a time, and therefore a lot of the random disk reads were actually directly from the OS disk cache, as opposed to the normal case (no squashed image), where the 3gb of information would not fit and therefore a lot of sectors would be loaded into the disk cache, then unloaded, then loaded again, etc.


kokoko3k wrote:

Also, i wonder if it is possible to append rw files to the previously created squashfs file, instead of create a new one from zero (it's a time consuming task!), or to manage multiple squashed files (it should be possible, but i'm not really in aufs)


You can append data to a squashed image, but you can't delete anything from it, so if you modify a file (that is already contained in the image) and try to re-add it to the squashed image, you would have a filename conflict that squashfs would solve in a non-convenient way (by renaming the second file). You don't have to re-make the squashed image too often, and you can also leave it over night, if it takes too much. You could manage to use multiple squash files with aufs / unionfs, and that would eliminate the file name conflict problem, but you would end up using too much disk space, as some files would be included in multiple squashed images (in different versions, of course) and it will be a bit more complicated to manage them. If you want to go for this last option, I could give you some pointers on how to use aufs for this purpose, just give me a message.

Thanks for trying my trick and for giving me feedback, maybe I will actually make some time to make this easier to use...
Back to top
View user's profile Send private message
medius
n00b
n00b


Joined: 04 Jul 2002
Posts: 7
Location: Vancouver

PostPosted: Tue Apr 01, 2008 6:08 pm    Post subject: Reply with quote

I've been playing with this trick for a few days now. I've found it pretty effective and wanted to contribute my squashmount init script. It is more of an outline, and could definitely be rewritten to take advantage of a conf file in /etc/conf.d but it works for me.

First some background: I have a fast 8GB usb flash drive that hosts the squashed images of my /usr and /opt directories. The flash drive is normally mounted on /mnt/patriot (it is a patriot drive) and is mounted based on the device id. However if the drive is not available for some reason, then I want the initscript to mount the usr.sfs and opt.sfs that I store on my local disk. This uses more drive space, but I'm doing this mostly for performance reasons so I prefer to have the security of a local copy of the squashed images.

I've modified my /etc/init.d/localmount so that it does not mount devices of type aufs or squashfs. Then my squashmount script mounts my opt and usr directories from flash if available, or from disk if not available.

I also modified any other init scripts that need localmount to need squashmount. This ensures that any init script that needs /usr will not run until after /usr is available.

Since the initscript also has a stop section I let it handle unmounting /usr and /opt and the flash drive.

Here is the snippet from my /etc/fstab that supports my /etc/init.d/squashmount

Code:

/dev/disk/by-id/usb-_Patriot_Memory_078304AD0A3F-0:0-part1  /mnt/patriot  reiserfs  rw,user,noatime,sync  0 0

/squashed/usr/usr.sfs  /squashed/usr/ro  squashfs  loop,ro,noauto  0 0
/mnt/patriot/squashed/usr.sfs  /squashed/usr/ro  squashfs  loop,ro,noauto  0 0
usr  /usr  aufs  udba=reval,br:/squashed/usr/rw:/squashed/usr/ro  0 0

/squashed/opt/opt.sfs  /squashed/opt/ro  squashfs  loop,ro,noauto  0 0
/mnt/patriot/squashed/opt.sfs  /squashed/opt/ro  squashfs  loop,ro,noauto  0 0
opt  /opt  aufs  udba=reval,br:/squashed/opt/rw:/squashed/opt/ro  0 0



Note the noauto options on the squashfs entries. This lets my script determine if the flashdrive is mounted before attempting to mount the wrong image.

Below is the content of my prototype /etc/init.d/squashmount

Code:

#!/sbin/runscript
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

depend() {
        need localmount
}

start() {

        # Mount squashfs filesystems. Use flash based images if they exist.

        ebegin "Mounting squashfs and aufs filesystems"

        if [[ -e /mnt/patriot/squashed/usr.sfs ]] ; then
                einfo "Mounting usr.sfs from flash"
                mount /mnt/patriot/squashed/usr.sfs > /dev/null
        elif [[ -e /squashed/usr/usr.sfs ]] ; then
                einfo "Mounting usr.sfs from disk"
                mount /squashed/usr/usr.sfs > /dev/null
        else
                eerror "usr.sfs not found"
        fi

        if [[ -e /mnt/patriot/squashed/opt.sfs ]] ; then
                einfo "Mounting opt.sfs from flash"
                mount /mnt/patriot/squashed/opt.sfs > /dev/null
        elif [[ -e /squashed/opt/opt.sfs ]] ; then
                einfo "Mounting opt.sfs from disk"
                mount /squashed/opt/opt.sfs > /dev/null
        else
                eerror "opt.sfs not found"
        fi

        eend $?

        # Mount aufs filesystems in /etc/fstab.
        ebegin "Mounting aufs filesystems"
        mount -at aufs \
                -O no_netdev >/dev/null
        eend $? "Some aufs filesystems failed to mount"

}

stop() {

        ebegin "Unmounting aufs filesystems"
        umount -alt aufs \
                -O no_netdev >/dev/null
        eend $? "Some aufs filesystems failed to unmount"

        ebegin "Unmounting squashfs filesystems"
        umount -alt squashfs \
                -O no_netdev >/dev/null
        eend $? "Some aufs filesystems failed to unmount"

        if [[ -d /mnt/patriot/squashed ]] ; then
                einfo "Unmounting flash"
                sync
                umount -l /mnt/patriot/ > /dev/null
        fi

        sleep 1

}

# vim:ts=4


And here is my modifed /etc/init.d/localmount

Code:

#!/sbin/runscript
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

depend() {
        need checkfs
}

start() {
        # Mount local filesystems in /etc/fstab.
        ebegin "Mounting local filesystems"
        mount -at noaufs,nosquashfs,noproc,noshm${NET_FS_LIST:+,no}${NET_FS_LIST// /,no} \
                -O no_netdev >/dev/null
        eend $? "Some local filesystem failed to mount"

        # Make sure we insert usbcore if its a module
        if [[ -f /proc/modules && ! -d /proc/bus/usb ]] ; then
                # >/dev/null to hide errors from non-USB users
                modprobe usbcore &> /dev/null
        fi

        # Check what USB fs the kernel support.  Currently
        # 2.5+ kernels, and later 2.4 kernels have 'usbfs',
        # while older kernels have 'usbdevfs'.
        if [[ -d /proc/bus/usb && ! -e /proc/bus/usb/devices ]] ; then
                local usbfs=$(grep -Fow usbfs /proc/filesystems ||
                        grep -Fow usbdevfs /proc/filesystems)
                if [[ -n ${usbfs} ]] ; then
                        ebegin "Mounting USB device filesystem (${usbfs})"
                        usbgid=$(echo $(getent group usb) | awk -F: '{print $3}')
                        mount -t ${usbfs} usbfs /proc/bus/usb \
                                -o ${usbgid:+devmode=0664,devgid=${usbgid},}noexec,nosuid
                        eend $?
                fi
        fi

        # Setup Kernel Support for miscellaneous Binary Formats
        if [[ -d /proc/sys/fs/binfmt_misc ]] ; then
                local binfmt=$(grep -Fow binfmt_misc /proc/filesystems)
                if [[ -n ${binfmt} ]] ; then
                        ebegin $"Mounting misc binary format filesystem"
                        mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc \
                                -o nodev,noexec,nosuid
                        eend $?
                fi
        fi

        # Setup Kernel Support for the security filesystem
        if [[ -d /sys/kernel/security ]] ; then
                local secfs=$(grep -Fow securityfs /proc/filesystems)
                if [[ -n ${secfs} ]] ; then
                        ebegin $"Mounting security filesystem"
                        mount -t securityfs securityfs /sys/kernel/security \
                                -o nodev,noexec,nosuid
                        eend $?
                fi
        fi

        # We do our swapping here instead of rc so we can get urandom started
        # before us for people that like an encrypted swap.
        ebegin "Activating (possible) swap"
        /sbin/swapon -a
        eend $?

        # Start dm-crypt mappings, if any
        start_addon dm-crypt
}

# vim:ts=4


I hope this is useful for someone.
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 280
Location: Dijon (F) > Berlin > Tokyo > Nürnberg (D)

PostPosted: Fri May 02, 2008 5:10 am    Post subject: Reply with quote

Nice, I did not think about it myself but it is a great idea! I have done it on my slow but ultra lightweight laptop running ubuntu [also posted my how to on the ubuntu forums] and starting openoffice is a new experience. It will probably save battery life, too.

Thanks!
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
DingoDom
n00b
n00b


Joined: 22 Nov 2004
Posts: 2
Location: france

PostPosted: Mon Aug 04, 2008 5:57 pm    Post subject: Reply with quote

Nice Tip..
I've been installing this tricks for few days... It's works and my gentoo is faster than ever
but it break my gcc
gcc-config error: Could not run/locate "x86_64-pc-linux-gnu-gcc"

Do you have any ideas ?

Thanks
Back to top
View user's profile Send private message
Palatis
n00b
n00b


Joined: 07 Oct 2006
Posts: 23
Location: Taipei/Taiwan

PostPosted: Mon Aug 11, 2008 2:22 pm    Post subject: Reply with quote

a little script to do the updates automatically via cron.
put this script into /etc/cron.weekly OR /etc/cron.monthly.
please modify the three VARIABLES to suite your needs!
i named this script `squashed-rebuild' :)
Code:
#!/bin/bash

BLOCK_SIZE=1048576

MOUNTPOINTS="usr opt"
SQFS_LOCATION="/mnt/sqfs/"

for MP in ${MOUNTPOINTS}
do
   /usr/bin/mksquashfs /${MP} ${SQFS_LOCATION}/${MP}_tmp.sqfs -b ${BLOCK_SIZE}
   /bin/umount -l /${MP}
   /bin/umount -l ${SQFS_LOCATION}/${MP}/ro
   /bin/rm -rf ${SQFS_LOCATION}/${MP}/rw &
   /bin/rm ${SQFS_LOCATION}/${MP}.sqfs
   /bin/mv ${SQFS_LOCATION}/${MP}_tmp.sqfs ${SQFS_LOCATION}
   /bin/mount ${SQFS_LOCATION}/${MP}/ro
   /bin/mount /${MP}
done

ohh, my setup is as below (/mnt/sqfs/ on a seperate lvm LV):

  • /mnt/sqfs/usr/ro
  • /mnt/sqfs/usr/rw
  • /mnt/sqfs/usr.sqfs
  • /mnt/sqfs/opt/ro
  • /mnt/sqfs/opt/rw
  • /mnt/sqfs/opt.sqfs


fstab:
Code:
/mnt/sqfs/usr.sqfs   /mnt/sqfs/usr/ro   squashfs   noatime,nodev,loop,ro   0 0
none         /usr         aufs      udba=reval,br:/mnt/sqfs/usr/rw:/mnt/sqfs/usr/ro  0 0
/home/sqfs/opt.sqfs   /home/sqfs/opt/ro   squashfs   noatime,nodev,loop,ro   0 0
none         /opt         aufs      udba=reval,br:/mnt/sqfs/opt/rw:/mnt/sqfs/opt/ro  0 0
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 280
Location: Dijon (F) > Berlin > Tokyo > Nürnberg (D)

PostPosted: Mon Sep 22, 2008 8:53 am    Post subject: Reply with quote

Hi, I'll go with my little script too. I use it on my laptop, which is running Debian Lenny, but it should not matter. Also note that I am using LVM2, that /home/synss/local is mounted on /usr/local (and therefore excluded from the sqfs image) and that /usr/opt is mounted on /opt (and therefore /opt is included in the sqfs image).
The script
    compresses /usr
    creates a new LV with appropriate size
    copies the sqfs filesystem into the LV


Two more files are created,
    squash.log, where all the files being squashed are reported, in case mkquashfs gets stuck (it has happened on my system)
    dd.log, output of dd, if any


Since I do that on my laptop, I do not want it in a cron nor initscript, so it is fairly simple and I run it by hand when /var/squashed/ gets too big. Using LVM2 makes everything easy.

Code:
#!/bin/sh

VG="/dev/vg"
DIR_OUT="lenny_usr_$(date +%Y%m%d)"
SQFS_OUT="$DIR_OUT.sqfs"
SQFS_LV="$VG/$SQFS_OUT"

die()
{
   exit 1
}
cout()
{
   echo " ** $@"
}

#if [ $UID != 0 ]; then
#   cout "Needs elevated privileges"
#   die
#fi

cout "Create $SQFS_OUT"
mksquashfs /usr $SQFS_OUT \
   -info \
   -no-exports -no-progress \
   -e /usr/local/* \
   > squash.log || die
# DEBUG try -lzma compression, now breaks v 3.3

mkdir -p /var/squashed/$DIR_OUT/ro &>/dev/null
mkdir -p /var/squashed/$DIR_OUT/rw &>/dev/null

SIZE=$(ls -l $SQFS_OUT | cut -d" " -f5)
SIZE="$(($SIZE/1024))k"

if [ -e $SQFS_LV ]
then
   cout "Remove $SQFS_LV"
   lvremove $SQFS_LV
fi
cout "Create $SQFS_LV"
lvcreate -n "$SQFS_OUT" -L $SIZE $VG || die

cout "Copy $SQFS_OUT to $SQFS_LV"
dd if=$SQFS_OUT of=$SQFS_LV > dd.log || die

#rm squash.log dd.log $SQFS_OUT

cout "Please edit /etc/fstab now"



PS: Don't be scared by the lvremove command, it will actually ask you if you want to do that.
_________________
Compress portage tree
Elog viewer
Autodetect swap


Last edited by synss on Mon Sep 22, 2008 9:06 am; edited 1 time in total
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 280
Location: Dijon (F) > Berlin > Tokyo > Nürnberg (D)

PostPosted: Mon Sep 22, 2008 9:03 am    Post subject: Reply with quote

Has anyone tried to compress the root? excluding, i.e.,
Code:
/dev/*
/home/*
/media/*
/srv/*
/tmp/*
/var/*
/boot/*
/lost+found/*
/mnt/*
/proc/*
/sys/*

(check the -ef option in mksquashfs), I am guessing that, as long as the squashfs and aufs modules are compiled in the kernel or that you have an initramfs, it should work. If somebody means to actually try, she should check live CD how-tos, because some files in /etc will not like that (I am thinking of /etc/mtab). I may give it a try myself later, as that should save some more battery life. EDIT: this might help to get the initramfs script right.

Last question: has anyone tried the user space solutions (i.e., fuse) and how does it perform?
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
shgadwa
Guru
Guru


Joined: 12 Mar 2009
Posts: 326

PostPosted: Wed Apr 08, 2009 10:29 am    Post subject: Reply with quote

OK... I have a problem here.

I did everything as described but now my system will not boot. It locks up at 'loading local filesystems'... And I'm unable to mount the filesystem in a Chrooted environment because it says "Unknown filesystem type 'squashfs'. "

Because I was having problems loadng the modules, I just set squashfs and aufs as enabled in the kernel, rather an setting it as a module.

Any ideas?

EDIT: I WAS able to load modules until I did a reboot (after creating the squashfs filesystem and stuff), now I am no longer able to load modules.

I get this:

Microknoppix / # modprobe squashfs
FATAL: Could not load /lib/modules/2.6.28/modules.dep: No such file or directory

Also, I've been thinking... I first followed this guide: http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree

And squashed the portage tree... then without making any changes, I decided to squash the entire /usr folder... something tells me I did it wrong. That is, I should not of squashed the portage tree if I was planning to squash the /usr folder (?).
Back to top
View user's profile Send private message
eruvaer_ohta
n00b
n00b


Joined: 18 Jul 2007
Posts: 22

PostPosted: Wed Apr 08, 2009 6:45 pm    Post subject: Reply with quote

belikeyeshua, have you deleted the original /usr folder? The kernel needs the files in /usr/src/linux-? to load the module 'aufs' (or any other module), which is then needed to uncompress the squashed filesystem. So if you deleted the uncompressed files, the kernel can't load the files it needs. To circumvent this, you can move the /usr/src/ folder to another location and soft-link it back.
Back to top
View user's profile Send private message
shgadwa
Guru
Guru


Joined: 12 Mar 2009
Posts: 326

PostPosted: Fri Apr 10, 2009 10:10 am    Post subject: Reply with quote

Well, it does work if I comment out the new entries in fstab, and in the other file, forget what that file was called...

Then the squashfs does not work, but at least it works. I've done everything I can think of.... I conclude that this guide is broken. It does not work for the latest system. I'd recommend that it was updated, or deleted.

Thank you for your help.
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 280
Location: Dijon (F) > Berlin > Tokyo > Nürnberg (D)

PostPosted: Fri May 01, 2009 11:13 pm    Post subject: Reply with quote

belikeyeshua wrote:
Well, it does work if I comment out the new entries in fstab, and in the other file, forget what that file was called...

Then the squashfs does not work, but at least it works. I've done everything I can think of.... I conclude that this guide is broken. It does not work for the latest system. I'd recommend that it was updated, or deleted.

Thank you for your help.


I do not want to sound rude but the thing is that if you do not understand what you are doing, do not follow how to blindly. Please swallow a little bit at a time and it will work. This how to works now better than ever since squashfs has been accepted in the latest kernels (.29). But it is still highly unsupported and you have to be able to fix it yourself.

This how to needs not being deleted or amended, it is you who need to understand more about your system.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
justinkb
Apprentice
Apprentice


Joined: 23 Dec 2008
Posts: 160

PostPosted: Sat May 02, 2009 12:34 am    Post subject: Reply with quote

very nice idea, but i compile new stuff too often for this to be useful for me :( since write performance gets worse with this.

@belikeyeshua, clearly, you didn't follow the howto properly... :/
_________________
~amd64 | gcc 4.4.2 | linux 2.6.33-git w/ alsa 1.0.22.1 and drm-radeon-testing | gnome(-light) 2.28.2 | xorg-x11 and xf86-video-ati from git
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 690

PostPosted: Wed May 06, 2009 7:06 am    Post subject: Reply with quote

waynesherman wrote:
1) The aufs mount in /etc/fstab depends on mount.aufs
2) mount.aufs is a shell script which happens to depend on 4 files in /usr/bin:
Code:
/usr/bin/comm
/usr/bin/diff
/usr/bin/paste
/usr/bin/tee

3) Since the /usr directory was empty at that point in the boot process, mount.aufs failed.

Why this is a problem? Looking at the example fstab it should first mount sqashfs (with all the /usr content) and then aufs. So it shouldn't be a problem, right? Anyway, you can make some dirty hack. SlashBeast suggested something similar but with /usr/src.
1) copy those files to ie. /usr.needed/bin
2) remove everything from /usr (except the bin dir)
3) make symlinks from /usr/bin/comm -> /usr.needed/bin/comm etc.
4) remember to update those files in /usr.needed when coreutils and/or diffutils are updated.

waynesherman wrote:
If no, why does it happen on my system and not others?

For me it looks like you tried to mount some aufs *before* squashfs /usr.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
ocbMaurice
Tux's lil' helper
Tux's lil' helper


Joined: 14 Feb 2003
Posts: 82
Location: Switzerland

PostPosted: Thu May 07, 2009 12:40 am    Post subject: Reply with quote

I really love this solution for my eeepc installation.

I did adjust the script quite a bit and wanted to share this:

/etc/init.d/squashed.usr.portage :
Code:
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/aufs are necessary for read-write support.
#
# Author: Marcel Greter <marcel.greter@crevis.ch>
# Inspired by http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree

opts="init sync info commit rollback"

# prepend root and normalize
DIRECTORY="/${DIRECTORY}"
DIRECTORY=`echo ${DIRECTORY} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# get mount points and squash fs file
if [ ! ${DEV_SHM} ]; then DEV_SHM="/dev/shm"; fi
if [ ! ${MOUNT_RO} ]; then MOUNT_RO="${SQUASH_ROOT}${DIRECTORY}.ro"; fi
if [ ! ${MOUNT_RW} ]; then MOUNT_RW="${SQUASH_ROOT}${DIRECTORY}.rw"; fi
if [ ! ${MOUNT_SFS} ]; then MOUNT_SFS="${SQUASH_ROOT}${DIRECTORY}.sfs"; fi
if [ ! ${MOUNT_SFS_NEW} ]; then MOUNT_SFS_NEW="${SQUASH_ROOT}${DIRECTORY}.sfs.new"; fi
if [ ! ${MOUNT_SFS_OLD} ]; then MOUNT_SFS_OLD="${SQUASH_ROOT}${DIRECTORY}.sfs.old"; fi

if [ ! ${MV_OPTS} ]; then MV_OPTS="-i"; fi
if [ ! ${RM_OPTS} ]; then RM_OPTS="-rI"; fi
if [ ! ${MKDIR_OPTS} ]; then MKDIR_OPTS="-p"; fi
if [ ! ${RMDIR_OPTS} ]; then RMDIR_OPTS="-p"; fi

if [ ! ${SYNC_THRESHOLD} ]; then SYNC_THRESHOLD="0"; fi
if [ ! ${KEEP_SFS_BACKUP} ]; then KEEP_SFS_BACKUP="0"; fi

# maybe use memory for overlay
if [ ${IN_MEMORY} ]; then MOUNT_RW="${DEV_SHM}${DIRECTORY}"; fi

depend() {
   need localmount
   after localmount
   before hostname
}

start() {

  _start

}

_start() {

  ebegin "Start squashed ${DIRECTORY}"
  mark_service_starting
 
  if [ ! -d ${DIRECTORY} ]; then

    eend 1 "${DIRECTORY} does not exist" || return $?

  fi

  if [ ! -e ${MOUNT_SFS} ]; then

    ewarn "you probably need to init me first?"
    eend 1 "squashed fs file does not exist" || return 1

  fi

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"

    /bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
    /bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"

    einfo "  bind ${EXCL_DIR} to temp location"
    /bin/mount -o bind "${EXCL_DIR}" "${EXCL_BIND}"
    eend $? "  could not bind ${EXCL_DIR} ${EXCL_BIND}" || return $?
 
  done

  einfo "  mount ${DIRECTORY} readonly squasfs ${SQUASHFS_OPTS}"
  /bin/mount -rt squashfs -o loop,ro,nodev \
    "${MOUNT_SFS}" "${MOUNT_RO}" ${SQUASHFS_OPTS}
  eend $? "  could not mount ${MOUNT_RO} ${SQUASHFS_OPTS}" || return $?

  # maybe use memory for overlay
  if [ ${IN_MEMORY} ]; then
    ewarn "  use writeable overlay in memory"
    ewarn "  changes WILL best lost on unexpected shutdown"
    if [ ! ${SYNC_ON_SHUTDOWN} ]; then
      ewarn "  changes WILL be lost on restart"
    else
      if [ 15 -lt ${SYNC_THRESHOLD} ]; then
        ewarn "  changes MAY be lost on restart"
      fi
    fi
  fi

  # create directory for write overlay
  [ -d ${MOUNT_RW} ] || mkdir ${MKDIR_OPTS} ${MOUNT_RW}

  # get info from directory (XXX - replace perl by sed or something like that)
  SFS_MOD=`/usr/bin/stat ${DIRECTORY} -L | /usr/bin/perl -e 'while(<>){ if(m/Access:\s*\(([0-9]{4})\//) { print $1; } }'`
  SFS_USR=`/usr/bin/stat ${DIRECTORY} -L | /usr/bin/perl -e 'while(<>){ if(m/Uid:\s*\(\s*[0-9]+\s*\/\s*([^\)]+)\)\s*Gid:\s*\(\s*[0-9]+\s*\/\s*([^\)]+)\)/) { print join(":", $1, $2); } }'`

  einfo "  chmod ${SFS_MOD} ${MOUNT_RW}"
  /bin/chmod ${SFS_MOD} ${MOUNT_RW}
  eend $? "  could not change file mode"

  einfo "  chown ${SFS_USR} ${MOUNT_RW}"
  /bin/chown ${SFS_USR} ${MOUNT_RW}
  eend $? "  could not change file owner"

  einfo "  mount ${DIRECTORY} writable overlay ${AUFS_OPTS}"
  /bin/mount -t aufs -o nodev,udba=reval,dirs=${MOUNT_RW}=rw:${MOUNT_RO}=ro \
    aufs ${AUFS_OPTS} "${DIRECTORY}"
  eend $? " could not mount ${AUFS_OPTS} ${DIRECTORY}" || return $?

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"

    einfo "  bind temp location to ${EXCL_DIR}"
    /bin/mount -o bind "${EXCL_BIND}" "${EXCL_DIR}"
    eend $? "  could not bind ${EXCL_BIND} ${EXCL_DIR}" || return $?

  done
 
  mark_service_started
  eend 0

}

stop() {

  _stop

}

_stop() {

  ebegin "Stop squashed ${DIRECTORY}"
  mark_service_stopping
 
  if [ ${SYNC_ON_SHUTDOWN} != 0 ]; then
    if [ $(du -s ${MOUNT_RW} | cut -f 1) -gt ${SYNC_THRESHOLD} ]; then
      einfo "  changes detected, updating image"
      commit; eend $? "  could not commit changes"
      return $?
    else
      einfo "  no changes detected, skipping update"; eend 0
    fi
  fi

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"

    einfo "  unbind ${EXCL_DIR}"
    /bin/umount "${EXCL_DIR}"
    eend $? "  could not umount ${EXCL_DIR}" || return $?

  done

  einfo "  umount ${DIRECTORY} writable overlay"
  /bin/umount ${DIRECTORY}
  eend $? "  could not umount ${DIRECTORY}" || return $?

  einfo "  umount ${DIRECTORY} readonly overlay"
  /bin/umount "${MOUNT_RO}"
  eend $? "  could not umount ${MOUNT_RO}" || return $?

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"

    einfo "  unbind ${EXCL_BIND}"
    /bin/umount "${EXCL_BIND}"
    eend $? "  could not umount ${EXCL_BIND}" || return $?

  done

  mark_service_stopped
  eend 0

}

restart() {

  svc_stop
  svc_start

}

init() {

  if service_started ${myservice}; then svc_stop; fi

  if [ ! -d ${DIRECTORY} ]; then

    eend 1 "${DIRECTORY} does not exist" || return $?

  fi

  if [ -e ${MOUNT_SFS} ]; then

    eerror "squashed fs file already exists"; eend 1
    ewarn "remove ${MOUNT_SFS} to force init"; return 1

  fi

  einfo "  initializing directory, please wait"

  /bin/mkdir ${MKDIR_OPTS} ${MOUNT_RO}
  /bin/mkdir ${MKDIR_OPTS} ${MOUNT_RW}

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
    EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"

    /bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
    /bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"
    /bin/mkdir ${MKDIR_OPTS} "${EXCL_SKEL}"

    einfo "  bind empty temp location to ${EXCL_DIR}"
    /bin/mount -o bind "${EXCL_SKEL}" "${EXCL_DIR}"
    eend $? "  could not bind ${EXCL_SKEL} ${EXCL_DIR}" || return $?
 
  done
 
  einfo "  /usr/bin/mksquashfs ${DIRECTORY} ${MOUNT_SFS} ${MKSQUASHFS_OPTS}"
  /usr/bin/mksquashfs ${DIRECTORY} ${MOUNT_SFS} ${MKSQUASHFS_OPTS}
  eend $? "  could not make squashfs ${DIRECTORY} ${MOUNT_SFS}" || return $?

  # einfo mount -rt squashfs -o loop,ro,nodev,noexec "${MOUNT_SFS}" "${MOUNT_RO}"
  # einfo mount -t aufs -o "nodev,noexec,udba=reval,dirs=${MOUNT_RW}=rw:${DIRECTORY}=ro" aufs "${DIRECTORY}"

  einfo "  cleaning ${DIRECTORY}"
  /bin/rm ${RM_OPTS} "${DIRECTORY}"
  eend "  could not remove ${DIRECTORY}"

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"

    einfo "  unbind empty temp location on ${EXCL_DIR}"
    /bin/umount "${EXCL_DIR}"
    eend $? "  could not umount ${EXCL_DIR}" || return $?
 
  done

  eend 0

  if service_stopped ${myservice}; then  svc_start; fi

}

sync() {

  commit

}

commit() {

  if [ ! -d ${DIRECTORY} ]; then

    eend 1 "${DIRECTORY} does not exist" || return $?

  fi

  if [ ! -e ${MOUNT_SFS} ]; then

    ewarn "you probably need to init me first?"
    eend 1 "squashed fs file does not exist" || return 1

  fi

  if [ -d "${MOUNT_SFS_NEW}" ]; then

    einfo "  /bin/mv ${MOV_OPTS} ${MOUNT_SFS_NEW} ${MOUNT_SFS_OLD}"
    /bin/mv ${MOV_OPTS} "${MOUNT_SFS_NEW}" "${MOUNT_SFS_OLD}"
    eend $? "  could not move ${MOUNT_SFS_NEW} ${MOUNT_SFS_OLD}"
 
  fi

  ebegin "Recreate squashed fs file and commit changes"

  if service_started ${myservice}; then RESTART=1; fi
  # if service_stopping ${myservice}; then RESTART=1; fi
  if service_stopped ${myservice}; then _start; fi
 
  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
    EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"

    /bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
    /bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"
    /bin/mkdir ${MKDIR_OPTS} "${EXCL_SKEL}"

    einfo "  bind empty temp location to ${EXCL_DIR}"
    /bin/mount -o bind "${EXCL_SKEL}" "${EXCL_DIR}"
    eend $? "  could not mount ${EXCL_SKEL} ${EXCL_DIR}" || return $?

  done

  einfo /usr/bin/mksquashfs "${DIRECTORY}" "${MOUNT_SFS_NEW}" ${MKSQUASHFS_OPTS}
  /usr/bin/mksquashfs "${DIRECTORY}" "${MOUNT_SFS_NEW}" ${MKSQUASHFS_OPTS}
  eend $? "  could not make squashfs ${DIRECTORY} ${MOUNT_SFS_NEW}" || return $?

  for EXCLUDE in ${EXCLUDE_DIRS}; do

    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"

    einfo "  unbind empty temp location on ${EXCL_DIR}"
    /bin/umount "${EXCL_DIR}"
    eend $? "  could not umount ${EXCL_DIR}" || return $?
 
  done

  # disable sync on shutdown
  SYNC_ON_SHUTDOWN="0" _stop

  einfo "  move '${MOUNT_SFS}' '${MOUNT_SFS_OLD}'"
  /bin/mv ${MV_OPTS} "${MOUNT_SFS}" "${MOUNT_SFS_OLD}"
  eend $? "  could not move ${MOUNT_SFS} ${MOUNT_SFS_OLD}" || return $?

  rollback

  einfo "  move '${MOUNT_SFS_NEW}' '${MOUNT_SFS}'"
  /bin/mv ${MV_OPTS} "${MOUNT_SFS_NEW}" "${MOUNT_SFS}"
  eend $? "  could not move ${MOUNT_SFS_NEW} ${MOUNT_SFS}" || return $?
 
  if [ ${KEEP_SFS_BACKUP} == 0 ]; then
    einfo "  delete ${MOUNT_SFS_OLD}"
    /bin/rm ${RM_OPTS} "${MOUNT_SFS_OLD}"
    eend $? "  could not delete ${MOUNT_SFS_OLD}"
  fi

  # restart if service was started before
  if [ ${RESTART} ] ; then _start; fi

  eend $?

}

info() {

  einfo "DIRECTORY        : ${DIRECTORY}";
  if [ -d ${DIRECTORY} ]; then eend 0; else eend 1; fi
  einfo "MOUNT_RO         : ${MOUNT_RO}";
  if [ -d ${MOUNT_RO} ]; then eend 0; else eend 1; fi
  einfo "MOUNT_RW         : ${MOUNT_RW}";
  if [ -d ${MOUNT_RW} ]; then eend 0; else eend 1; fi
  einfo "MOUNT_SFS        : ${MOUNT_SFS}";
  if [ -f ${MOUNT_SFS} ]; then eend 0; else eend 1; fi
  einfo "MOUNT_SFS_NEW    : ${MOUNT_SFS_NEW}";
  if [ ! -e ${MOUNT_SFS_NEW} ]; then eend 0; else eend 1; fi
  einfo "MOUNT_SFS_OLD    : ${MOUNT_SFS_OLD}";
  # this is only a backup file
  # if [ ! -e ${MOUNT_SFS_OLD} ]; then eend 0; else eend 1; fi

  einfo "AUFS_OPTS        : ${AUFS_OPTS}";
  einfo "SQUASHFS_OPTS    : ${SQUASHFS_OPTS}";
  einfo "MKSQUASHFS_OPTS  : ${MKSQUASHFS_OPTS}";

  einfo "MV_OPTS          : ${MV_OPTS}";
  einfo "RM_OPTS          : ${RM_OPTS}";
  einfo "MKDIR_OPTS       : ${MKDIR_OPTS}";
  einfo "RMDIR_OPTS       : ${RMDIR_OPTS}";

  einfo "DEV_SHM          : ${DEV_SHM}";
  einfo "IN_MEMORY        : ${IN_MEMORY}";
  einfo "SYNC_ON_SHUTDOWN : ${SYNC_ON_SHUTDOWN}";
  einfo "SYNC_THRESHOLD   : ${SYNC_THRESHOLD}";
  einfo "  changed so far : $(du -s ${MOUNT_RW} | cut -f 1)";

  for EXCLUDE in ${EXCLUDE_DIRS}; do
    # prepend root and normalize
    EXCLUDE="/${EXCLUDE}"
    EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
    # absolute paths
    EXCL_DIR="${DIRECTORY}${EXCLUDE}"
    EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
    EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"
    einfo " exclude dir     : ${EXCL_DIR}";
    einfo " exclude bind    : ${EXCL_BIND}";
    einfo " exclude skel    : ${EXCL_SKEL}";
  done
 
  eend 0

}

rollback() {

  einfo "  ${MOUNT_RW} delete writable overlay"
  /bin/rm ${RM_OPTS} ${MOUNT_RW}
  eend $? "  could not delete ${MOUNT_RW}" || return $?

  einfo "  ${MOUNT_RW} re-create writable overlay"
  /bin/mkdir ${MKDIR_OPTS} ${MOUNT_RW}
  eend $? "  could not create ${MOUNT_RW}" || return $?

}


/etc/conf.d/squashed.usr.portage :
Code:

# the actual directory you want to squash
DIRECTORY="/usr/portage"

# generic data storage directory
# will create sub-directories
SQUASH_ROOT="/mnt/squashed"

# additional options for mksquashfs
MKSQUASHFS_OPTS="-b 65536" # -no-duplicates

# additional squashfs mount options
SQUASHFS_OPTS="-o noexec"

# additional squashfs mount options
AUFS_OPTS="-o noexec"

# redefine auto configuration
# MV_OPTS="-i"
# RM_OPTS="-rI"
# MKDIR_OPTS="-p"
# RMDIR_OPTS="-p"

# directories will be created on demand
# MOUNT_RO="${SQUASH_ROOT}${DIRECTORY}.ro"
# MOUNT_RW="${SQUASH_ROOT}${DIRECTORY}.rw"
# MOUNT_SFS="${SQUASH_ROOT}${DIRECTORY}.sfs"
# MOUNT_SFS_NEW="${SQUASH_ROOT}${DIRECTORY}.sfs.new"
# MOUNT_SFS_OLD="${SQUASH_ROOT}${DIRECTORY}.sfs.old"

# mount writable overlay in memory
# IN_MEMORY=1; # DEV_SHM="/dev/shm";

# do not include in squash image
EXCLUDE_DIRS="distfiles"

# update squash image on shutdown
# only if there have been changes
SYNC_ON_SHUTDOWN=1

# minimum size reported by du to
# consider that something has changed
SYNC_THRESHOLD=14

# keep old sfs image
KEEP_SFS_BACKUP=0


Configuration is obviousely done in /etc/conf.d/squashed.usr.portage. If you want to squash other directories, you can make a symlink of the init script and copy the configuration (as with net.* scripts).

Once it's configured, you only have to do :

Code:
# /etc/init.d/squashed.usr.portage init


Other commands :

Code:
# /etc/init.d/squashed.usr.portage start
# /etc/init.d/squashed.usr.portage stop
# /etc/init.d/squashed.usr.portage restart
  -> should be self explained

# /etc/init.d/squashed.usr.portage sync
# /etc/init.d/squashed.usr.portage commit
  -> save changes to squashed image and clear rw store

# /etc/init.d/squashed.usr.portage rollback
  -> purge rw store without touching old squased image

# /etc/init.d/squashed.usr.portage info
 * DIRECTORY        : /usr/portage
 * MOUNT_RO         : /mnt/squashed/usr/portage.ro
 * MOUNT_RW         : /mnt/squashed/usr/portage.rw
 * MOUNT_SFS        : /mnt/squashed/usr/portage.sfs
 * MOUNT_SFS_NEW    : /mnt/squashed/usr/portage.sfs.new
 * MOUNT_SFS_OLD    : /mnt/squashed/usr/portage.sfs.old
 * AUFS_OPTS        : -o noexec
 * SQUASHFS_OPTS    : -o noexec
 * MKSQUASHFS_OPTS  : -b 65536
 * MV_OPTS          : -i
 * RM_OPTS          : -rI
 * MKDIR_OPTS       : -p
 * RMDIR_OPTS       : -p
 * DEV_SHM          : /dev/shm
 * IN_MEMORY        :
 * SYNC_ON_SHUTDOWN : 1
 * SYNC_THRESHOLD   : 1500
 *   changed so far : 12
 *  exclude dir     : /usr/portage/distfiles
 *  exclude bind    : /mnt/squashed/usr/portage.bind/distfiles
 *  exclude skel    : /mnt/squashed/usr/portage.skel/distfiles


Features :

- Define sub directories to be excluded from the squash image
- Configurable threshold for auto syncing the image on shutdown
- So far you will be asked for confirmation before overwriting stuff


I needed to add this to the init script

Code:
depend() {
   need localmount
   after localmount
   before hostname
}


so I could compress stuff like

- /usr/share
- /var/cache
- /var/db

I'm not sure if this is the best way, for me it seems to work.

My current setup is like this (all in boot runlevel) :

Code:
dir /etc/init.d/squashed.*
/etc/init.d/squashed.usr.portage
/etc/init.d/squashed.usr.share -> squashed.usr.portage
/etc/init.d/squashed.usr.src -> squashed.usr.portage
/etc/init.d/squashed.var.cache -> squashed.usr.portage
/etc/init.d/squashed.var.db -> squashed.usr.portage


I tested the script quite well but I'm not responsible if anything goes wrong!
Anyway, feedback is very welcome!

Hope you like it,
Maurice
Back to top
View user's profile Send private message
iamboredr
n00b
n00b


Joined: 09 May 2009
Posts: 10

PostPosted: Sat May 09, 2009 7:08 pm    Post subject: Reply with quote

really good idea.. but still not very much dynamic
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Sat May 09, 2009 7:37 pm    Post subject: Reply with quote

I didn't look through the whole thread, but it seems to me like a dupe of Compressing portage using squashfs: initscript method thread (at least at the newer parts of that thread). In particular, ocbMaurice's script is practically completely covered with the corresponding (meanwhile rather mature, although there were revisions recently) initscript from this page which was tested with several directories.

The only difference seems to me the threshhold test which however could be easily configured in /etc/conf.d/squash_... by setting there the "MAGIC_FILE" variable depending on some "ds" command. However, I think the original magic file feature of the script on the webpage is more powerful, since you can manually decide (without changing any config files) whether you want to save or not, just by creating a magic ".no-save" file in the corresponding directory.

Or am I missing something and the scripts in this thread here can really do something crucial which you cannot do easily with the script on the webpage?
Back to top
View user's profile Send private message
ocbMaurice
Tux's lil' helper
Tux's lil' helper


Joined: 14 Feb 2003
Posts: 82
Location: Switzerland

PostPosted: Mon May 11, 2009 10:02 pm    Post subject: Reply with quote

mv wrote:
I didn't look through the whole thread, but it seems to me like a dupe of Compressing portage using squashfs: initscript method thread (at least at the newer parts of that thread). In particular, ocbMaurice's script is practically completely covered with the corresponding (meanwhile rather mature, although there were revisions recently) initscript from this page which was tested with several directories.

Thanks for pointing me to this, I really missed that one. Well I saw the page, but didn't really investigate the script as I did now. And it really looks like the thing I was initially looking for.

I think there are only two functions missing in this script:
- The threshold as in my script (which is IMO quite usefull)
- Set certain Directories to be excluded from the squash image
- [edit] my script sets the permissions for the new mounted dir to the same as the old (chmod/chown)

As you said, the first one might be even possible with the current solution (the other script even seems to have a better function to check if anything has changed).
There also could be some more options for the init script (like my init, sync or info function). I also think a "uninit" method would be nice to undo the init step (not done in my script yet either).
Well, the script from M. Väth seems to be more mature than mine (specially when it comes to housekeeping).

IMO, you find quite a lot of info about this on the web or on the forum. But it's difficult to really find the best solution (that's why I coded my own script). There are even two threads here about the same issue.
It seems that M. Väth still updates his script, so he probably can integrate those features in his script. It then would be wise to put that in a prominent Position here (IMO this could even get into portage tree).

FYI: To exclude some dirs from squashing I bind/mount them to some temp location, then mount squash ro and aufs rw, then bind/mount the temp location back to the original location.
When creating the squash image, the location is bound to an empty temp location (so the directory will be empty in the squash image). IMO this is a must, since it is quite easy to do and works like a charm!

Maurice

p.s. I still don't know if it's possible to query if anything has changed. IMO it's not possible to know that anything has changed if you only delte files ... I also checked out the proc stuff ...
p.p.s. This approach does not only save harddisk space, it also can free up quite some inodes ;)


Last edited by ocbMaurice on Mon May 11, 2009 10:56 pm; edited 4 times in total
Back to top
View user's profile Send private message
ocbMaurice
Tux's lil' helper
Tux's lil' helper


Joined: 14 Feb 2003
Posts: 82
Location: Switzerland

PostPosted: Mon May 11, 2009 10:04 pm    Post subject: Reply with quote

iamboredr wrote:
really good idea.. but still not very much dynamic

To be honest, I didn't get the point about dynamic?
How should/could it be more dynamic?
Maybe you can enlighten me!
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Tue May 12, 2009 7:23 pm    Post subject: Reply with quote

ocbMaurice wrote:
- Set certain Directories to be excluded from the squash image

I think the proper way to exclude mounted subdirectories is to specify it in the mksquashfs-options with -ef (which can be done with the other script).
If you want to mount/umount them automatically, I suggest you do this in another init-script which "need"s the squash_dir-script: If all would be done in the same script, then in case of errors, it would be unreasonable hard to track down the problems; e.g. what to do if mounting to a temporary directory succeeds, but the aufs-mount fails or something similar. Not to speak about the problems which happen if you are in a --make-shared/--make-slave directory or something similar.
Quote:
my script sets the permissions for the new mounted dir to the same as the old (chmod/chown)

I do not understand this: Why should it be unreasonable to use the permissions stored in the squash-file? The content also has the permissions stored in that file, so why should it be different for the directory itself?
Quote:
p.s. I still don't know if it's possible to query if anything has changed. IMO it's not possible to know that anything has changed if you only delte files

Not sure what you mean. The other script does detect this: If you delete files, aufs will store a corresponding file in the writable branch. Of course, this would not go well with the threshhold option. Or do you mean deletion of flies which were only created during the last boot? These do not have to be detected, since if all of them are deleted, there is no reason to resquash the image.
Back to top
View user's profile Send private message
x-demon
n00b
n00b


Joined: 27 Oct 2009
Posts: 1

PostPosted: Tue Oct 27, 2009 11:39 am    Post subject: Reply with quote

Is it possible to recreate squash img based on old img? Because building image from scratch takes about 30-40 mins on my netbook...
Back to top
View user's profile Send private message
^marcs
Apprentice
Apprentice


Joined: 09 Mar 2005
Posts: 169

PostPosted: Wed Mar 23, 2011 3:58 pm    Post subject: Reply with quote

Sorry for digging such an old topic, but it's best I can find.

Anyone managed to install aufs2 patch in 2.6.38?
Way those patches are distributed is a mystery to me
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 690

PostPosted: Sat Mar 26, 2011 11:48 am    Post subject: Reply with quote

You can try this one → aufs2-2.6.38.patch. I didn't test it yet, but it should work.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
tclover
Guru
Guru


Joined: 10 Apr 2011
Posts: 322

PostPosted: Wed Apr 13, 2011 2:16 am    Post subject: issue with umount /some/file.sfs Reply with quote

First off, thanks for this tricks and tips. I cannot possibly run my system on now without usr mounted on a squashsf/aufs combo. Once you get used to the responsiveness of the thing, you cannot come back.

Here are two modified scripts grabbed from this thread, I'll just post 'em in case, well if ever, somebody could find some use. No waranty whatsover of course nor for some usefulness:
Quote:
#!/bin/sh
#~/sqfsd-build.sh
BSIZE=131072
COMP=xz
EXCL="/lib64/rc/init.d/* /lib64/splash/cache/*"
sqfsd="lib64"
sqfsdir="/sqfsd"
FSTAB=n
[ -n "$EXCL" ] && e="-e $EXCL"
die() {
echo "$1"
exit 1
}
for s in ${sqfsd}
do
/bin/echo "#"
/bin/echo "# [re]building squashed-${s} #"
/bin/echo "#"
/bin/mkdir -p ${sqfsdir}/${s}/{ro,rw} || die "failed to create $s/{ro,rw} dirs."
/usr/bin/mksquashfs /${s} ${sqfsdir}/${s}_tmp.sfs -b ${BSIZE} -comp ${COMP} $e || die "failed to build $s.sfs img."
[ -n "`mount -t aufs | grep ${s}`" ] && { /bin/umount -l /${s} || die "failed to umount $s aufs branch."; }
[ -n "`mount -t squashfs | grep ${sqfsdir}/${s}/ro`" ] && { /bin/umount -l ${sqfsdir}/${s}/ro || die "failed to umount sfs img."; }
/bin/rm -rf ${sqfsdir}/${s}/rw/* || die "failed to clean up $s/rw."
[ -e ${sqfsdir}/${s}.sfs ] && /bin/rm -f ${sqfsdir}/${s}.sfs
/bin/mv ${sqfsdir}/${s}_tmp.sfs ${sqfsdir}/${s}.sfs || die "failed to move tmp.sfs img."
#
# set FSTAB=y if you want localmount to take care of mounting the squashed stuff.
# For disk squashed based stuff, you don't realy need to add sqfsmount init service,
# localmount will mount the necessary stuff with the following two lines.
#
[ "${FSTAB}" = "y" ] && {
/bin/echo "${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro squashfs noatime,nodev,loop,ro 0 0" >> /etc/fstab || die "fstab er.1"
/bin/echo "${s} /${s} aufs udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro 0 0" >> /etc/fstab || die "fstab er.2"
}
/bin/mount -t squashfs ${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro -o noatime,nodev,loop,ro || die "failed to mount $s.sfs img."
/bin/mount -t aufs ${s} /${s} -o udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro || die "failed to mount $s aufs branch."
#
# start /etc/init.d/sqfsmount and add it in the boot runlevel if you added the init script
# and don't forget to add /etc/conf.d/sqfsmount configuration file... else it wont do anything.
/bin/echo "#"
/bin/echo "# squashed-${s} sucessfully [re]build #"
/bin/echo "#"
done

Now fstab is optional, I keep only usr lines related as a safe net, as the contains of usr disapeared, so localmount can mount it if need be:
Quote:
/sqfsd/usr.sfs /sqfs/usr/ro squashfs noatime,nodev,loop,ro 0 0
usr /usr aufs udba=reval,br:/sqfs/usr/rw:/sqfs/usr/ro 0 0


And here my /etc/init.d/sqfsmount:
Quote:
#!/sbin/runscript
#!/sbin/runscript
# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
#
#/etc/init.d/sqsfdmount
#
# Adding need dependancies is required, after removing usr, to avoid bootup errors, else you can add usr[squashed] in fstab.
# so localmount will take care of mounting usr before acpid, bootmisc, consolefont and keymaps use some binary in usr.
# for exemple, usr is mounted from a sqfsd image. You can always add usr required 2 lines in fastab, the init script will check anyway,
# if usr is mounted, it will not mount it. Comment out/in the before line if need be.

description="Mounts squashed directory [according to... /etc/fstab if you want to]."


depend()
{
need fsck localmount
use lvm modules mtab
after lvm modules
#before consolefont bootmisc acpid keymaps
keyword -jail -openvz -prefix -vserver -lxc
}

start() {

[ "${USQFS}" = "y" ] && {
echo "mounting flash-squashed dirs"
sqfsd_mount "${USQFSDIR}" "${USQFSD}";
}
echo "mounting disk-squashed dirs"
sqfsd_mount "${SQFSDIR}" "${SQFSD}"

}

stop() {

[ "${USQFS}" = "y" ] && {
echo "umounting flash-squashed dirs"
sqfsd_mount "${USQFSDIR}" "${USQFSD}";
}
echo "umounting disk-squashed dirs"
sqfsd_mount "${SQFSDIR}" "${SQFSD}"
sleep 1

}

sqfsd_mount() {

sqfsdir="${1}" sqfsd="${2}"
for s in ${sqfsd} ; do
if [ -n "`mount -t squashfs | grep "${sqfsdir}/${s}"/ro`" ]; then
echo "${sqfsdir}/${s}/ro already mounted."
else
ebegin "mounting squashed ${s}image..."
mount -t squashfs ${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro -o noatime,nodev,loop,ro
eend $? "mount squashed ${s} image failed."
fi
if [ -n "`mount -t aufs | grep /${s}`" ]; then
echo "${s} aufs branch already mounted"
else
ebegin "mounting ${s} aufs branch..."
mount -t aufs ${s} /${s} -o udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro
eend $? "mount squashed ${s} aufs branch failed"
fi
done

}

sqfsd_umount() {

sqfsdir="${1}" sqfsd="${2}"
for s in ${sqfsd} ; do
ebegin "umounting squashed ${s} aufs branch..."
umount -lt aufs /${s} -O no_netdev 2>/dev/null
eend $? "umount squashed ${s} aufs branch failed."
ebegin "umounting squashed ${s} image..."
umount -lt squashfs ${sqfsdir}/${s}/ro -O no_netdev 2>/dev/null
eend $? "umount squashed ${s} image failed."
done

}

# vim:ts=4

Quote:
#/etc/conf.d/sqfsdmount
SQFSDIR="/sqfsd"
SQFSD="usr sbin opt var/db var/lib/layman"
USQFS="n"
USQFSDIR="/mnt/usqfsd"
USQFSD="lib32"

An yeah, the idea of getting /lib32 and /lib64 squashed seems appealing but I have an issue when booting. I have baselayout-2.0 openrc-0.8 and it seems /sbin/init write in /lib64 in the bootup process. When the bootup process end some services 'die' somewhere. I guess some services infos get masked after the squashed files got mounted. I don't have any guess on what is happening in the bootup process. I didn't have the time to dig up about that.

EDIT: EDITED Mai 09, 2011, new update version of sqfsd-build.sh. And now I have almost the same script for fcron. I modified the sqfsd="some dirs" line to sqfsd="`echo ${1}|tr ':' ' '`" and named the script sqfsd-rebuild.sh and put it to /root. So now you can call the rebuild script using a line like this in crontab:
* * 1,15 * * root $HOME/sqfsd-rebuild.sh usr:bin:sbin:lib32
or/and
* * 1,10,20 * * root $HOME/sqfsd-rebuild.sh var/db:var/cache/edb:var/lib/layman:var/portage:var/local/portage
say you can put different cron job depending on your needs.

EDIT2: WHAT'S SO GOOD ABOUT SQUASHING STUFF YOU ASK!?
I happen to have wiped out the whole [squashed] usr, after having wiped out the real usr, directory. Well, with a squashed usr... you have to recover only your squashed image. A go in debugfs </dev/sdxx> and then a "cd /sqfsd && ls -d" you note the inode number for usr.sfs and everything is back in no time! And, of course a wiped out squashed file is easy to recover... just rm the .wh* related file in */rw and...
No good you say!? Doesn't worth the trouble?
I could never go back to the much slower usr again. Just the pain to wait emerge to... calculate dependencies and...


Last edited by tclover on Tue May 10, 2011 11:29 am; edited 13 times in total
Back to top
View user's profile Send private message
tclover
Guru
Guru


Joined: 10 Apr 2011
Posts: 322

PostPosted: Wed Apr 13, 2011 10:56 pm    Post subject: Reply with quote

just adding a side note of my previous post for those who would want to grab sqfsmount init script, you still had to add a 'need dependency' to the following init services, otherwise you will get errors in the bootup process, but your system could just work with 'em, those errors disappear after sqfsmount is runned: acpid, bootmisc, consolefont, setfont.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum