View previous topic :: View next topic |
Author |
Message |
kokoko3k n00b
Joined: 06 May 2005 Posts: 72
|
Posted: Wed Mar 19, 2008 4:44 pm Post subject: |
|
|
First of all, Thanks!
Quote: | And yes, the OS would be caching the compressed sectors (not the uncompressed data) from the squash file every time you try to read something from /usr. |
Are you sure about this?
Also, i wonder if it is possible to append rw files to the previously created squashfs file, instead of create a new one from zero (it's a time consuming task!), or to manage multiple squashed files (it should be possible, but i'm not really in aufs) |
|
Back to top |
|
|
stanbogdan n00b
Joined: 14 Jan 2008 Posts: 12
|
Posted: Wed Mar 19, 2008 7:44 pm Post subject: |
|
|
kokoko3k wrote: | First of all, Thanks!
Quote: | And yes, the OS would be caching the compressed sectors (not the uncompressed data) from the squash file every time you try to read something from /usr. |
Are you sure about this?
|
Hmmm, I will investigate further, I don't know for sure, but I feel that is what is happening, because of the following symptom: when doing the random access pattern, the squashed file system test was more than 3 times faster than the normal test (a LOT more), even though the compression factor was about 3x. I have no other way to explain this, other than the fact that the entire squashed image of my /usr folder (which is appx 1gb) could be loaded entirely into my RAM (1gb), one sector at a time, and therefore a lot of the random disk reads were actually directly from the OS disk cache, as opposed to the normal case (no squashed image), where the 3gb of information would not fit and therefore a lot of sectors would be loaded into the disk cache, then unloaded, then loaded again, etc.
kokoko3k wrote: |
Also, i wonder if it is possible to append rw files to the previously created squashfs file, instead of create a new one from zero (it's a time consuming task!), or to manage multiple squashed files (it should be possible, but i'm not really in aufs) |
You can append data to a squashed image, but you can't delete anything from it, so if you modify a file (that is already contained in the image) and try to re-add it to the squashed image, you would have a filename conflict that squashfs would solve in a non-convenient way (by renaming the second file). You don't have to re-make the squashed image too often, and you can also leave it over night, if it takes too much. You could manage to use multiple squash files with aufs / unionfs, and that would eliminate the file name conflict problem, but you would end up using too much disk space, as some files would be included in multiple squashed images (in different versions, of course) and it will be a bit more complicated to manage them. If you want to go for this last option, I could give you some pointers on how to use aufs for this purpose, just give me a message.
Thanks for trying my trick and for giving me feedback, maybe I will actually make some time to make this easier to use... |
|
Back to top |
|
|
medius n00b
Joined: 04 Jul 2002 Posts: 7 Location: Vancouver
|
Posted: Tue Apr 01, 2008 6:08 pm Post subject: |
|
|
I've been playing with this trick for a few days now. I've found it pretty effective and wanted to contribute my squashmount init script. It is more of an outline, and could definitely be rewritten to take advantage of a conf file in /etc/conf.d but it works for me.
First some background: I have a fast 8GB usb flash drive that hosts the squashed images of my /usr and /opt directories. The flash drive is normally mounted on /mnt/patriot (it is a patriot drive) and is mounted based on the device id. However if the drive is not available for some reason, then I want the initscript to mount the usr.sfs and opt.sfs that I store on my local disk. This uses more drive space, but I'm doing this mostly for performance reasons so I prefer to have the security of a local copy of the squashed images.
I've modified my /etc/init.d/localmount so that it does not mount devices of type aufs or squashfs. Then my squashmount script mounts my opt and usr directories from flash if available, or from disk if not available.
I also modified any other init scripts that need localmount to need squashmount. This ensures that any init script that needs /usr will not run until after /usr is available.
Since the initscript also has a stop section I let it handle unmounting /usr and /opt and the flash drive.
Here is the snippet from my /etc/fstab that supports my /etc/init.d/squashmount
Code: |
/dev/disk/by-id/usb-_Patriot_Memory_078304AD0A3F-0:0-part1 /mnt/patriot reiserfs rw,user,noatime,sync 0 0
/squashed/usr/usr.sfs /squashed/usr/ro squashfs loop,ro,noauto 0 0
/mnt/patriot/squashed/usr.sfs /squashed/usr/ro squashfs loop,ro,noauto 0 0
usr /usr aufs udba=reval,br:/squashed/usr/rw:/squashed/usr/ro 0 0
/squashed/opt/opt.sfs /squashed/opt/ro squashfs loop,ro,noauto 0 0
/mnt/patriot/squashed/opt.sfs /squashed/opt/ro squashfs loop,ro,noauto 0 0
opt /opt aufs udba=reval,br:/squashed/opt/rw:/squashed/opt/ro 0 0
|
Note the noauto options on the squashfs entries. This lets my script determine if the flashdrive is mounted before attempting to mount the wrong image.
Below is the content of my prototype /etc/init.d/squashmount
Code: |
#!/sbin/runscript
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
depend() {
need localmount
}
start() {
# Mount squashfs filesystems. Use flash based images if they exist.
ebegin "Mounting squashfs and aufs filesystems"
if [[ -e /mnt/patriot/squashed/usr.sfs ]] ; then
einfo "Mounting usr.sfs from flash"
mount /mnt/patriot/squashed/usr.sfs > /dev/null
elif [[ -e /squashed/usr/usr.sfs ]] ; then
einfo "Mounting usr.sfs from disk"
mount /squashed/usr/usr.sfs > /dev/null
else
eerror "usr.sfs not found"
fi
if [[ -e /mnt/patriot/squashed/opt.sfs ]] ; then
einfo "Mounting opt.sfs from flash"
mount /mnt/patriot/squashed/opt.sfs > /dev/null
elif [[ -e /squashed/opt/opt.sfs ]] ; then
einfo "Mounting opt.sfs from disk"
mount /squashed/opt/opt.sfs > /dev/null
else
eerror "opt.sfs not found"
fi
eend $?
# Mount aufs filesystems in /etc/fstab.
ebegin "Mounting aufs filesystems"
mount -at aufs \
-O no_netdev >/dev/null
eend $? "Some aufs filesystems failed to mount"
}
stop() {
ebegin "Unmounting aufs filesystems"
umount -alt aufs \
-O no_netdev >/dev/null
eend $? "Some aufs filesystems failed to unmount"
ebegin "Unmounting squashfs filesystems"
umount -alt squashfs \
-O no_netdev >/dev/null
eend $? "Some aufs filesystems failed to unmount"
if [[ -d /mnt/patriot/squashed ]] ; then
einfo "Unmounting flash"
sync
umount -l /mnt/patriot/ > /dev/null
fi
sleep 1
}
# vim:ts=4
|
And here is my modifed /etc/init.d/localmount
Code: |
#!/sbin/runscript
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
depend() {
need checkfs
}
start() {
# Mount local filesystems in /etc/fstab.
ebegin "Mounting local filesystems"
mount -at noaufs,nosquashfs,noproc,noshm${NET_FS_LIST:+,no}${NET_FS_LIST// /,no} \
-O no_netdev >/dev/null
eend $? "Some local filesystem failed to mount"
# Make sure we insert usbcore if its a module
if [[ -f /proc/modules && ! -d /proc/bus/usb ]] ; then
# >/dev/null to hide errors from non-USB users
modprobe usbcore &> /dev/null
fi
# Check what USB fs the kernel support. Currently
# 2.5+ kernels, and later 2.4 kernels have 'usbfs',
# while older kernels have 'usbdevfs'.
if [[ -d /proc/bus/usb && ! -e /proc/bus/usb/devices ]] ; then
local usbfs=$(grep -Fow usbfs /proc/filesystems ||
grep -Fow usbdevfs /proc/filesystems)
if [[ -n ${usbfs} ]] ; then
ebegin "Mounting USB device filesystem (${usbfs})"
usbgid=$(echo $(getent group usb) | awk -F: '{print $3}')
mount -t ${usbfs} usbfs /proc/bus/usb \
-o ${usbgid:+devmode=0664,devgid=${usbgid},}noexec,nosuid
eend $?
fi
fi
# Setup Kernel Support for miscellaneous Binary Formats
if [[ -d /proc/sys/fs/binfmt_misc ]] ; then
local binfmt=$(grep -Fow binfmt_misc /proc/filesystems)
if [[ -n ${binfmt} ]] ; then
ebegin $"Mounting misc binary format filesystem"
mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc \
-o nodev,noexec,nosuid
eend $?
fi
fi
# Setup Kernel Support for the security filesystem
if [[ -d /sys/kernel/security ]] ; then
local secfs=$(grep -Fow securityfs /proc/filesystems)
if [[ -n ${secfs} ]] ; then
ebegin $"Mounting security filesystem"
mount -t securityfs securityfs /sys/kernel/security \
-o nodev,noexec,nosuid
eend $?
fi
fi
# We do our swapping here instead of rc so we can get urandom started
# before us for people that like an encrypted swap.
ebegin "Activating (possible) swap"
/sbin/swapon -a
eend $?
# Start dm-crypt mappings, if any
start_addon dm-crypt
}
# vim:ts=4
|
I hope this is useful for someone. |
|
Back to top |
|
|
synss Apprentice
Joined: 08 Mar 2006 Posts: 282 Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
Posted: Fri May 02, 2008 5:10 am Post subject: |
|
|
Nice, I did not think about it myself but it is a great idea! I have done it on my slow but ultra lightweight laptop running ubuntu [also posted my how to on the ubuntu forums] and starting openoffice is a new experience. It will probably save battery life, too.
Thanks! _________________ Compress portage tree
Elog viewer
Autodetect swap |
|
Back to top |
|
|
DingoDom n00b
Joined: 22 Nov 2004 Posts: 2 Location: france
|
Posted: Mon Aug 04, 2008 5:57 pm Post subject: |
|
|
Nice Tip..
I've been installing this tricks for few days... It's works and my gentoo is faster than ever
but it break my gcc
gcc-config error: Could not run/locate "x86_64-pc-linux-gnu-gcc"
Do you have any ideas ?
Thanks |
|
Back to top |
|
|
Palatis n00b
Joined: 07 Oct 2006 Posts: 23 Location: Taipei/Taiwan
|
Posted: Mon Aug 11, 2008 2:22 pm Post subject: |
|
|
a little script to do the updates automatically via cron.
put this script into /etc/cron.weekly OR /etc/cron.monthly.
please modify the three VARIABLES to suite your needs!
i named this script `squashed-rebuild'
Code: | #!/bin/bash
BLOCK_SIZE=1048576
MOUNTPOINTS="usr opt"
SQFS_LOCATION="/mnt/sqfs/"
for MP in ${MOUNTPOINTS}
do
/usr/bin/mksquashfs /${MP} ${SQFS_LOCATION}/${MP}_tmp.sqfs -b ${BLOCK_SIZE}
/bin/umount -l /${MP}
/bin/umount -l ${SQFS_LOCATION}/${MP}/ro
/bin/rm -rf ${SQFS_LOCATION}/${MP}/rw &
/bin/rm ${SQFS_LOCATION}/${MP}.sqfs
/bin/mv ${SQFS_LOCATION}/${MP}_tmp.sqfs ${SQFS_LOCATION}
/bin/mount ${SQFS_LOCATION}/${MP}/ro
/bin/mount /${MP}
done |
ohh, my setup is as below (/mnt/sqfs/ on a seperate lvm LV):
- /mnt/sqfs/usr/ro
- /mnt/sqfs/usr/rw
- /mnt/sqfs/usr.sqfs
- /mnt/sqfs/opt/ro
- /mnt/sqfs/opt/rw
- /mnt/sqfs/opt.sqfs
fstab:
Code: | /mnt/sqfs/usr.sqfs /mnt/sqfs/usr/ro squashfs noatime,nodev,loop,ro 0 0
none /usr aufs udba=reval,br:/mnt/sqfs/usr/rw:/mnt/sqfs/usr/ro 0 0
/home/sqfs/opt.sqfs /home/sqfs/opt/ro squashfs noatime,nodev,loop,ro 0 0
none /opt aufs udba=reval,br:/mnt/sqfs/opt/rw:/mnt/sqfs/opt/ro 0 0 |
|
|
Back to top |
|
|
synss Apprentice
Joined: 08 Mar 2006 Posts: 282 Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
Posted: Mon Sep 22, 2008 8:53 am Post subject: |
|
|
Hi, I'll go with my little script too. I use it on my laptop, which is running Debian Lenny, but it should not matter. Also note that I am using LVM2, that /home/synss/local is mounted on /usr/local (and therefore excluded from the sqfs image) and that /usr/opt is mounted on /opt (and therefore /opt is included in the sqfs image).
The script
compresses /usr
creates a new LV with appropriate size
copies the sqfs filesystem into the LV
Two more files are created,
squash.log, where all the files being squashed are reported, in case mkquashfs gets stuck (it has happened on my system)
dd.log, output of dd, if any
Since I do that on my laptop, I do not want it in a cron nor initscript, so it is fairly simple and I run it by hand when /var/squashed/ gets too big. Using LVM2 makes everything easy.
Code: | #!/bin/sh
VG="/dev/vg"
DIR_OUT="lenny_usr_$(date +%Y%m%d)"
SQFS_OUT="$DIR_OUT.sqfs"
SQFS_LV="$VG/$SQFS_OUT"
die()
{
exit 1
}
cout()
{
echo " ** $@"
}
#if [ $UID != 0 ]; then
# cout "Needs elevated privileges"
# die
#fi
cout "Create $SQFS_OUT"
mksquashfs /usr $SQFS_OUT \
-info \
-no-exports -no-progress \
-e /usr/local/* \
> squash.log || die
# DEBUG try -lzma compression, now breaks v 3.3
mkdir -p /var/squashed/$DIR_OUT/ro &>/dev/null
mkdir -p /var/squashed/$DIR_OUT/rw &>/dev/null
SIZE=$(ls -l $SQFS_OUT | cut -d" " -f5)
SIZE="$(($SIZE/1024))k"
if [ -e $SQFS_LV ]
then
cout "Remove $SQFS_LV"
lvremove $SQFS_LV
fi
cout "Create $SQFS_LV"
lvcreate -n "$SQFS_OUT" -L $SIZE $VG || die
cout "Copy $SQFS_OUT to $SQFS_LV"
dd if=$SQFS_OUT of=$SQFS_LV > dd.log || die
#rm squash.log dd.log $SQFS_OUT
cout "Please edit /etc/fstab now"
|
PS: Don't be scared by the lvremove command, it will actually ask you if you want to do that. _________________ Compress portage tree
Elog viewer
Autodetect swap
Last edited by synss on Mon Sep 22, 2008 9:06 am; edited 1 time in total |
|
Back to top |
|
|
synss Apprentice
Joined: 08 Mar 2006 Posts: 282 Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
Posted: Mon Sep 22, 2008 9:03 am Post subject: |
|
|
Has anyone tried to compress the root? excluding, i.e.,
Code: | /dev/*
/home/*
/media/*
/srv/*
/tmp/*
/var/*
/boot/*
/lost+found/*
/mnt/*
/proc/*
/sys/*
|
(check the -ef option in mksquashfs), I am guessing that, as long as the squashfs and aufs modules are compiled in the kernel or that you have an initramfs, it should work. If somebody means to actually try, she should check live CD how-tos, because some files in /etc will not like that (I am thinking of /etc/mtab). I may give it a try myself later, as that should save some more battery life. EDIT: this might help to get the initramfs script right.
Last question: has anyone tried the user space solutions (i.e., fuse) and how does it perform? _________________ Compress portage tree
Elog viewer
Autodetect swap |
|
Back to top |
|
|
shgadwa Guru
Joined: 12 Mar 2009 Posts: 327
|
Posted: Wed Apr 08, 2009 10:29 am Post subject: |
|
|
OK... I have a problem here.
I did everything as described but now my system will not boot. It locks up at 'loading local filesystems'... And I'm unable to mount the filesystem in a Chrooted environment because it says "Unknown filesystem type 'squashfs'. "
Because I was having problems loadng the modules, I just set squashfs and aufs as enabled in the kernel, rather an setting it as a module.
Any ideas?
EDIT: I WAS able to load modules until I did a reboot (after creating the squashfs filesystem and stuff), now I am no longer able to load modules.
I get this:
Microknoppix / # modprobe squashfs
FATAL: Could not load /lib/modules/2.6.28/modules.dep: No such file or directory
Also, I've been thinking... I first followed this guide: http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree
And squashed the portage tree... then without making any changes, I decided to squash the entire /usr folder... something tells me I did it wrong. That is, I should not of squashed the portage tree if I was planning to squash the /usr folder (?). |
|
Back to top |
|
|
eruvaer_ohta n00b
Joined: 18 Jul 2007 Posts: 23
|
Posted: Wed Apr 08, 2009 6:45 pm Post subject: |
|
|
belikeyeshua, have you deleted the original /usr folder? The kernel needs the files in /usr/src/linux-? to load the module 'aufs' (or any other module), which is then needed to uncompress the squashed filesystem. So if you deleted the uncompressed files, the kernel can't load the files it needs. To circumvent this, you can move the /usr/src/ folder to another location and soft-link it back. |
|
Back to top |
|
|
shgadwa Guru
Joined: 12 Mar 2009 Posts: 327
|
Posted: Fri Apr 10, 2009 10:10 am Post subject: |
|
|
Well, it does work if I comment out the new entries in fstab, and in the other file, forget what that file was called...
Then the squashfs does not work, but at least it works. I've done everything I can think of.... I conclude that this guide is broken. It does not work for the latest system. I'd recommend that it was updated, or deleted.
Thank you for your help. |
|
Back to top |
|
|
synss Apprentice
Joined: 08 Mar 2006 Posts: 282 Location: Dijon > Berlin > Tokyo > Nürnberg > München
|
Posted: Fri May 01, 2009 11:13 pm Post subject: |
|
|
belikeyeshua wrote: | Well, it does work if I comment out the new entries in fstab, and in the other file, forget what that file was called...
Then the squashfs does not work, but at least it works. I've done everything I can think of.... I conclude that this guide is broken. It does not work for the latest system. I'd recommend that it was updated, or deleted.
Thank you for your help. |
I do not want to sound rude but the thing is that if you do not understand what you are doing, do not follow how to blindly. Please swallow a little bit at a time and it will work. This how to works now better than ever since squashfs has been accepted in the latest kernels (.29). But it is still highly unsupported and you have to be able to fix it yourself.
This how to needs not being deleted or amended, it is you who need to understand more about your system. _________________ Compress portage tree
Elog viewer
Autodetect swap |
|
Back to top |
|
|
justinkb Apprentice
Joined: 23 Dec 2008 Posts: 161
|
Posted: Sat May 02, 2009 12:34 am Post subject: |
|
|
very nice idea, but i compile new stuff too often for this to be useful for me since write performance gets worse with this.
@belikeyeshua, clearly, you didn't follow the howto properly... :/ |
|
Back to top |
|
|
lazy_bum l33t
Joined: 16 Feb 2005 Posts: 691
|
Posted: Wed May 06, 2009 7:06 am Post subject: |
|
|
waynesherman wrote: | 1) The aufs mount in /etc/fstab depends on mount.aufs
2) mount.aufs is a shell script which happens to depend on 4 files in /usr/bin:
Code: | /usr/bin/comm
/usr/bin/diff
/usr/bin/paste
/usr/bin/tee
|
3) Since the /usr directory was empty at that point in the boot process, mount.aufs failed.
|
Why this is a problem? Looking at the example fstab it should first mount sqashfs (with all the /usr content) and then aufs. So it shouldn't be a problem, right? Anyway, you can make some dirty hack. SlashBeast suggested something similar but with /usr/src.
1) copy those files to ie. /usr.needed/bin
2) remove everything from /usr (except the bin dir)
3) make symlinks from /usr/bin/comm -> /usr.needed/bin/comm etc.
4) remember to update those files in /usr.needed when coreutils and/or diffutils are updated.
waynesherman wrote: | If no, why does it happen on my system and not others? |
For me it looks like you tried to mount some aufs *before* squashfs /usr. _________________ roslin uberlay | grubelek |
|
Back to top |
|
|
ocbMaurice Tux's lil' helper
Joined: 14 Feb 2003 Posts: 90 Location: Switzerland
|
Posted: Thu May 07, 2009 12:40 am Post subject: |
|
|
I really love this solution for my eeepc installation.
I did adjust the script quite a bit and wanted to share this:
/etc/init.d/squashed.usr.portage :
Code: | #!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/aufs are necessary for read-write support.
#
# Author: Marcel Greter <marcel.greter@crevis.ch>
# Inspired by http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree
opts="init sync info commit rollback"
# prepend root and normalize
DIRECTORY="/${DIRECTORY}"
DIRECTORY=`echo ${DIRECTORY} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# get mount points and squash fs file
if [ ! ${DEV_SHM} ]; then DEV_SHM="/dev/shm"; fi
if [ ! ${MOUNT_RO} ]; then MOUNT_RO="${SQUASH_ROOT}${DIRECTORY}.ro"; fi
if [ ! ${MOUNT_RW} ]; then MOUNT_RW="${SQUASH_ROOT}${DIRECTORY}.rw"; fi
if [ ! ${MOUNT_SFS} ]; then MOUNT_SFS="${SQUASH_ROOT}${DIRECTORY}.sfs"; fi
if [ ! ${MOUNT_SFS_NEW} ]; then MOUNT_SFS_NEW="${SQUASH_ROOT}${DIRECTORY}.sfs.new"; fi
if [ ! ${MOUNT_SFS_OLD} ]; then MOUNT_SFS_OLD="${SQUASH_ROOT}${DIRECTORY}.sfs.old"; fi
if [ ! ${MV_OPTS} ]; then MV_OPTS="-i"; fi
if [ ! ${RM_OPTS} ]; then RM_OPTS="-rI"; fi
if [ ! ${MKDIR_OPTS} ]; then MKDIR_OPTS="-p"; fi
if [ ! ${RMDIR_OPTS} ]; then RMDIR_OPTS="-p"; fi
if [ ! ${SYNC_THRESHOLD} ]; then SYNC_THRESHOLD="0"; fi
if [ ! ${KEEP_SFS_BACKUP} ]; then KEEP_SFS_BACKUP="0"; fi
# maybe use memory for overlay
if [ ${IN_MEMORY} ]; then MOUNT_RW="${DEV_SHM}${DIRECTORY}"; fi
depend() {
need localmount
after localmount
before hostname
}
start() {
_start
}
_start() {
ebegin "Start squashed ${DIRECTORY}"
mark_service_starting
if [ ! -d ${DIRECTORY} ]; then
eend 1 "${DIRECTORY} does not exist" || return $?
fi
if [ ! -e ${MOUNT_SFS} ]; then
ewarn "you probably need to init me first?"
eend 1 "squashed fs file does not exist" || return 1
fi
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"
einfo " bind ${EXCL_DIR} to temp location"
/bin/mount -o bind "${EXCL_DIR}" "${EXCL_BIND}"
eend $? " could not bind ${EXCL_DIR} ${EXCL_BIND}" || return $?
done
einfo " mount ${DIRECTORY} readonly squasfs ${SQUASHFS_OPTS}"
/bin/mount -rt squashfs -o loop,ro,nodev \
"${MOUNT_SFS}" "${MOUNT_RO}" ${SQUASHFS_OPTS}
eend $? " could not mount ${MOUNT_RO} ${SQUASHFS_OPTS}" || return $?
# maybe use memory for overlay
if [ ${IN_MEMORY} ]; then
ewarn " use writeable overlay in memory"
ewarn " changes WILL best lost on unexpected shutdown"
if [ ! ${SYNC_ON_SHUTDOWN} ]; then
ewarn " changes WILL be lost on restart"
else
if [ 15 -lt ${SYNC_THRESHOLD} ]; then
ewarn " changes MAY be lost on restart"
fi
fi
fi
# create directory for write overlay
[ -d ${MOUNT_RW} ] || mkdir ${MKDIR_OPTS} ${MOUNT_RW}
# get info from directory (XXX - replace perl by sed or something like that)
SFS_MOD=`/usr/bin/stat ${DIRECTORY} -L | /usr/bin/perl -e 'while(<>){ if(m/Access:\s*\(([0-9]{4})\//) { print $1; } }'`
SFS_USR=`/usr/bin/stat ${DIRECTORY} -L | /usr/bin/perl -e 'while(<>){ if(m/Uid:\s*\(\s*[0-9]+\s*\/\s*([^\)]+)\)\s*Gid:\s*\(\s*[0-9]+\s*\/\s*([^\)]+)\)/) { print join(":", $1, $2); } }'`
einfo " chmod ${SFS_MOD} ${MOUNT_RW}"
/bin/chmod ${SFS_MOD} ${MOUNT_RW}
eend $? " could not change file mode"
einfo " chown ${SFS_USR} ${MOUNT_RW}"
/bin/chown ${SFS_USR} ${MOUNT_RW}
eend $? " could not change file owner"
einfo " mount ${DIRECTORY} writable overlay ${AUFS_OPTS}"
/bin/mount -t aufs -o nodev,udba=reval,dirs=${MOUNT_RW}=rw:${MOUNT_RO}=ro \
aufs ${AUFS_OPTS} "${DIRECTORY}"
eend $? " could not mount ${AUFS_OPTS} ${DIRECTORY}" || return $?
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
einfo " bind temp location to ${EXCL_DIR}"
/bin/mount -o bind "${EXCL_BIND}" "${EXCL_DIR}"
eend $? " could not bind ${EXCL_BIND} ${EXCL_DIR}" || return $?
done
mark_service_started
eend 0
}
stop() {
_stop
}
_stop() {
ebegin "Stop squashed ${DIRECTORY}"
mark_service_stopping
if [ ${SYNC_ON_SHUTDOWN} != 0 ]; then
if [ $(du -s ${MOUNT_RW} | cut -f 1) -gt ${SYNC_THRESHOLD} ]; then
einfo " changes detected, updating image"
commit; eend $? " could not commit changes"
return $?
else
einfo " no changes detected, skipping update"; eend 0
fi
fi
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
einfo " unbind ${EXCL_DIR}"
/bin/umount "${EXCL_DIR}"
eend $? " could not umount ${EXCL_DIR}" || return $?
done
einfo " umount ${DIRECTORY} writable overlay"
/bin/umount ${DIRECTORY}
eend $? " could not umount ${DIRECTORY}" || return $?
einfo " umount ${DIRECTORY} readonly overlay"
/bin/umount "${MOUNT_RO}"
eend $? " could not umount ${MOUNT_RO}" || return $?
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
einfo " unbind ${EXCL_BIND}"
/bin/umount "${EXCL_BIND}"
eend $? " could not umount ${EXCL_BIND}" || return $?
done
mark_service_stopped
eend 0
}
restart() {
svc_stop
svc_start
}
init() {
if service_started ${myservice}; then svc_stop; fi
if [ ! -d ${DIRECTORY} ]; then
eend 1 "${DIRECTORY} does not exist" || return $?
fi
if [ -e ${MOUNT_SFS} ]; then
eerror "squashed fs file already exists"; eend 1
ewarn "remove ${MOUNT_SFS} to force init"; return 1
fi
einfo " initializing directory, please wait"
/bin/mkdir ${MKDIR_OPTS} ${MOUNT_RO}
/bin/mkdir ${MKDIR_OPTS} ${MOUNT_RW}
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_SKEL}"
einfo " bind empty temp location to ${EXCL_DIR}"
/bin/mount -o bind "${EXCL_SKEL}" "${EXCL_DIR}"
eend $? " could not bind ${EXCL_SKEL} ${EXCL_DIR}" || return $?
done
einfo " /usr/bin/mksquashfs ${DIRECTORY} ${MOUNT_SFS} ${MKSQUASHFS_OPTS}"
/usr/bin/mksquashfs ${DIRECTORY} ${MOUNT_SFS} ${MKSQUASHFS_OPTS}
eend $? " could not make squashfs ${DIRECTORY} ${MOUNT_SFS}" || return $?
# einfo mount -rt squashfs -o loop,ro,nodev,noexec "${MOUNT_SFS}" "${MOUNT_RO}"
# einfo mount -t aufs -o "nodev,noexec,udba=reval,dirs=${MOUNT_RW}=rw:${DIRECTORY}=ro" aufs "${DIRECTORY}"
einfo " cleaning ${DIRECTORY}"
/bin/rm ${RM_OPTS} "${DIRECTORY}"
eend " could not remove ${DIRECTORY}"
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
einfo " unbind empty temp location on ${EXCL_DIR}"
/bin/umount "${EXCL_DIR}"
eend $? " could not umount ${EXCL_DIR}" || return $?
done
eend 0
if service_stopped ${myservice}; then svc_start; fi
}
sync() {
commit
}
commit() {
if [ ! -d ${DIRECTORY} ]; then
eend 1 "${DIRECTORY} does not exist" || return $?
fi
if [ ! -e ${MOUNT_SFS} ]; then
ewarn "you probably need to init me first?"
eend 1 "squashed fs file does not exist" || return 1
fi
if [ -d "${MOUNT_SFS_NEW}" ]; then
einfo " /bin/mv ${MOV_OPTS} ${MOUNT_SFS_NEW} ${MOUNT_SFS_OLD}"
/bin/mv ${MOV_OPTS} "${MOUNT_SFS_NEW}" "${MOUNT_SFS_OLD}"
eend $? " could not move ${MOUNT_SFS_NEW} ${MOUNT_SFS_OLD}"
fi
ebegin "Recreate squashed fs file and commit changes"
if service_started ${myservice}; then RESTART=1; fi
# if service_stopping ${myservice}; then RESTART=1; fi
if service_stopped ${myservice}; then _start; fi
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_DIR}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_BIND}"
/bin/mkdir ${MKDIR_OPTS} "${EXCL_SKEL}"
einfo " bind empty temp location to ${EXCL_DIR}"
/bin/mount -o bind "${EXCL_SKEL}" "${EXCL_DIR}"
eend $? " could not mount ${EXCL_SKEL} ${EXCL_DIR}" || return $?
done
einfo /usr/bin/mksquashfs "${DIRECTORY}" "${MOUNT_SFS_NEW}" ${MKSQUASHFS_OPTS}
/usr/bin/mksquashfs "${DIRECTORY}" "${MOUNT_SFS_NEW}" ${MKSQUASHFS_OPTS}
eend $? " could not make squashfs ${DIRECTORY} ${MOUNT_SFS_NEW}" || return $?
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
einfo " unbind empty temp location on ${EXCL_DIR}"
/bin/umount "${EXCL_DIR}"
eend $? " could not umount ${EXCL_DIR}" || return $?
done
# disable sync on shutdown
SYNC_ON_SHUTDOWN="0" _stop
einfo " move '${MOUNT_SFS}' '${MOUNT_SFS_OLD}'"
/bin/mv ${MV_OPTS} "${MOUNT_SFS}" "${MOUNT_SFS_OLD}"
eend $? " could not move ${MOUNT_SFS} ${MOUNT_SFS_OLD}" || return $?
rollback
einfo " move '${MOUNT_SFS_NEW}' '${MOUNT_SFS}'"
/bin/mv ${MV_OPTS} "${MOUNT_SFS_NEW}" "${MOUNT_SFS}"
eend $? " could not move ${MOUNT_SFS_NEW} ${MOUNT_SFS}" || return $?
if [ ${KEEP_SFS_BACKUP} == 0 ]; then
einfo " delete ${MOUNT_SFS_OLD}"
/bin/rm ${RM_OPTS} "${MOUNT_SFS_OLD}"
eend $? " could not delete ${MOUNT_SFS_OLD}"
fi
# restart if service was started before
if [ ${RESTART} ] ; then _start; fi
eend $?
}
info() {
einfo "DIRECTORY : ${DIRECTORY}";
if [ -d ${DIRECTORY} ]; then eend 0; else eend 1; fi
einfo "MOUNT_RO : ${MOUNT_RO}";
if [ -d ${MOUNT_RO} ]; then eend 0; else eend 1; fi
einfo "MOUNT_RW : ${MOUNT_RW}";
if [ -d ${MOUNT_RW} ]; then eend 0; else eend 1; fi
einfo "MOUNT_SFS : ${MOUNT_SFS}";
if [ -f ${MOUNT_SFS} ]; then eend 0; else eend 1; fi
einfo "MOUNT_SFS_NEW : ${MOUNT_SFS_NEW}";
if [ ! -e ${MOUNT_SFS_NEW} ]; then eend 0; else eend 1; fi
einfo "MOUNT_SFS_OLD : ${MOUNT_SFS_OLD}";
# this is only a backup file
# if [ ! -e ${MOUNT_SFS_OLD} ]; then eend 0; else eend 1; fi
einfo "AUFS_OPTS : ${AUFS_OPTS}";
einfo "SQUASHFS_OPTS : ${SQUASHFS_OPTS}";
einfo "MKSQUASHFS_OPTS : ${MKSQUASHFS_OPTS}";
einfo "MV_OPTS : ${MV_OPTS}";
einfo "RM_OPTS : ${RM_OPTS}";
einfo "MKDIR_OPTS : ${MKDIR_OPTS}";
einfo "RMDIR_OPTS : ${RMDIR_OPTS}";
einfo "DEV_SHM : ${DEV_SHM}";
einfo "IN_MEMORY : ${IN_MEMORY}";
einfo "SYNC_ON_SHUTDOWN : ${SYNC_ON_SHUTDOWN}";
einfo "SYNC_THRESHOLD : ${SYNC_THRESHOLD}";
einfo " changed so far : $(du -s ${MOUNT_RW} | cut -f 1)";
for EXCLUDE in ${EXCLUDE_DIRS}; do
# prepend root and normalize
EXCLUDE="/${EXCLUDE}"
EXCLUDE=`echo ${EXCLUDE} | sed 's/\/$//' | sed 's/\/\+/\//g'`
# absolute paths
EXCL_DIR="${DIRECTORY}${EXCLUDE}"
EXCL_BIND="${SQUASH_ROOT}${DIRECTORY}.bind${EXCLUDE}"
EXCL_SKEL="${SQUASH_ROOT}${DIRECTORY}.skel${EXCLUDE}"
einfo " exclude dir : ${EXCL_DIR}";
einfo " exclude bind : ${EXCL_BIND}";
einfo " exclude skel : ${EXCL_SKEL}";
done
eend 0
}
rollback() {
einfo " ${MOUNT_RW} delete writable overlay"
/bin/rm ${RM_OPTS} ${MOUNT_RW}
eend $? " could not delete ${MOUNT_RW}" || return $?
einfo " ${MOUNT_RW} re-create writable overlay"
/bin/mkdir ${MKDIR_OPTS} ${MOUNT_RW}
eend $? " could not create ${MOUNT_RW}" || return $?
}
|
/etc/conf.d/squashed.usr.portage :
Code: |
# the actual directory you want to squash
DIRECTORY="/usr/portage"
# generic data storage directory
# will create sub-directories
SQUASH_ROOT="/mnt/squashed"
# additional options for mksquashfs
MKSQUASHFS_OPTS="-b 65536" # -no-duplicates
# additional squashfs mount options
SQUASHFS_OPTS="-o noexec"
# additional squashfs mount options
AUFS_OPTS="-o noexec"
# redefine auto configuration
# MV_OPTS="-i"
# RM_OPTS="-rI"
# MKDIR_OPTS="-p"
# RMDIR_OPTS="-p"
# directories will be created on demand
# MOUNT_RO="${SQUASH_ROOT}${DIRECTORY}.ro"
# MOUNT_RW="${SQUASH_ROOT}${DIRECTORY}.rw"
# MOUNT_SFS="${SQUASH_ROOT}${DIRECTORY}.sfs"
# MOUNT_SFS_NEW="${SQUASH_ROOT}${DIRECTORY}.sfs.new"
# MOUNT_SFS_OLD="${SQUASH_ROOT}${DIRECTORY}.sfs.old"
# mount writable overlay in memory
# IN_MEMORY=1; # DEV_SHM="/dev/shm";
# do not include in squash image
EXCLUDE_DIRS="distfiles"
# update squash image on shutdown
# only if there have been changes
SYNC_ON_SHUTDOWN=1
# minimum size reported by du to
# consider that something has changed
SYNC_THRESHOLD=14
# keep old sfs image
KEEP_SFS_BACKUP=0
|
Configuration is obviousely done in /etc/conf.d/squashed.usr.portage. If you want to squash other directories, you can make a symlink of the init script and copy the configuration (as with net.* scripts).
Once it's configured, you only have to do :
Code: | # /etc/init.d/squashed.usr.portage init |
Other commands :
Code: | # /etc/init.d/squashed.usr.portage start
# /etc/init.d/squashed.usr.portage stop
# /etc/init.d/squashed.usr.portage restart
-> should be self explained
# /etc/init.d/squashed.usr.portage sync
# /etc/init.d/squashed.usr.portage commit
-> save changes to squashed image and clear rw store
# /etc/init.d/squashed.usr.portage rollback
-> purge rw store without touching old squased image
# /etc/init.d/squashed.usr.portage info
* DIRECTORY : /usr/portage
* MOUNT_RO : /mnt/squashed/usr/portage.ro
* MOUNT_RW : /mnt/squashed/usr/portage.rw
* MOUNT_SFS : /mnt/squashed/usr/portage.sfs
* MOUNT_SFS_NEW : /mnt/squashed/usr/portage.sfs.new
* MOUNT_SFS_OLD : /mnt/squashed/usr/portage.sfs.old
* AUFS_OPTS : -o noexec
* SQUASHFS_OPTS : -o noexec
* MKSQUASHFS_OPTS : -b 65536
* MV_OPTS : -i
* RM_OPTS : -rI
* MKDIR_OPTS : -p
* RMDIR_OPTS : -p
* DEV_SHM : /dev/shm
* IN_MEMORY :
* SYNC_ON_SHUTDOWN : 1
* SYNC_THRESHOLD : 1500
* changed so far : 12
* exclude dir : /usr/portage/distfiles
* exclude bind : /mnt/squashed/usr/portage.bind/distfiles
* exclude skel : /mnt/squashed/usr/portage.skel/distfiles
|
Features :
- Define sub directories to be excluded from the squash image
- Configurable threshold for auto syncing the image on shutdown
- So far you will be asked for confirmation before overwriting stuff
I needed to add this to the init script
Code: | depend() {
need localmount
after localmount
before hostname
}
|
so I could compress stuff like
- /usr/share
- /var/cache
- /var/db
I'm not sure if this is the best way, for me it seems to work.
My current setup is like this (all in boot runlevel) :
Code: | dir /etc/init.d/squashed.*
/etc/init.d/squashed.usr.portage
/etc/init.d/squashed.usr.share -> squashed.usr.portage
/etc/init.d/squashed.usr.src -> squashed.usr.portage
/etc/init.d/squashed.var.cache -> squashed.usr.portage
/etc/init.d/squashed.var.db -> squashed.usr.portage |
I tested the script quite well but I'm not responsible if anything goes wrong!
Anyway, feedback is very welcome!
Hope you like it,
Maurice |
|
Back to top |
|
|
iamboredr n00b
Joined: 09 May 2009 Posts: 10
|
Posted: Sat May 09, 2009 7:08 pm Post subject: |
|
|
really good idea.. but still not very much dynamic |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6749
|
Posted: Sat May 09, 2009 7:37 pm Post subject: |
|
|
I didn't look through the whole thread, but it seems to me like a dupe of Compressing portage using squashfs: initscript method thread (at least at the newer parts of that thread). In particular, ocbMaurice's script is practically completely covered with the corresponding (meanwhile rather mature, although there were revisions recently) initscript from this page which was tested with several directories.
The only difference seems to me the threshhold test which however could be easily configured in /etc/conf.d/squash_... by setting there the "MAGIC_FILE" variable depending on some "ds" command. However, I think the original magic file feature of the script on the webpage is more powerful, since you can manually decide (without changing any config files) whether you want to save or not, just by creating a magic ".no-save" file in the corresponding directory.
Or am I missing something and the scripts in this thread here can really do something crucial which you cannot do easily with the script on the webpage? |
|
Back to top |
|
|
ocbMaurice Tux's lil' helper
Joined: 14 Feb 2003 Posts: 90 Location: Switzerland
|
Posted: Mon May 11, 2009 10:02 pm Post subject: |
|
|
mv wrote: | I didn't look through the whole thread, but it seems to me like a dupe of Compressing portage using squashfs: initscript method thread (at least at the newer parts of that thread). In particular, ocbMaurice's script is practically completely covered with the corresponding (meanwhile rather mature, although there were revisions recently) initscript from this page which was tested with several directories. |
Thanks for pointing me to this, I really missed that one. Well I saw the page, but didn't really investigate the script as I did now. And it really looks like the thing I was initially looking for.
I think there are only two functions missing in this script:
- The threshold as in my script (which is IMO quite usefull)
- Set certain Directories to be excluded from the squash image
- [edit] my script sets the permissions for the new mounted dir to the same as the old (chmod/chown)
As you said, the first one might be even possible with the current solution (the other script even seems to have a better function to check if anything has changed).
There also could be some more options for the init script (like my init, sync or info function). I also think a "uninit" method would be nice to undo the init step (not done in my script yet either).
Well, the script from M. Väth seems to be more mature than mine (specially when it comes to housekeeping).
IMO, you find quite a lot of info about this on the web or on the forum. But it's difficult to really find the best solution (that's why I coded my own script). There are even two threads here about the same issue.
It seems that M. Väth still updates his script, so he probably can integrate those features in his script. It then would be wise to put that in a prominent Position here (IMO this could even get into portage tree).
FYI: To exclude some dirs from squashing I bind/mount them to some temp location, then mount squash ro and aufs rw, then bind/mount the temp location back to the original location.
When creating the squash image, the location is bound to an empty temp location (so the directory will be empty in the squash image). IMO this is a must, since it is quite easy to do and works like a charm!
Maurice
p.s. I still don't know if it's possible to query if anything has changed. IMO it's not possible to know that anything has changed if you only delte files ... I also checked out the proc stuff ...
p.p.s. This approach does not only save harddisk space, it also can free up quite some inodes
Last edited by ocbMaurice on Mon May 11, 2009 10:56 pm; edited 4 times in total |
|
Back to top |
|
|
ocbMaurice Tux's lil' helper
Joined: 14 Feb 2003 Posts: 90 Location: Switzerland
|
Posted: Mon May 11, 2009 10:04 pm Post subject: |
|
|
iamboredr wrote: | really good idea.. but still not very much dynamic |
To be honest, I didn't get the point about dynamic?
How should/could it be more dynamic?
Maybe you can enlighten me! |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6749
|
Posted: Tue May 12, 2009 7:23 pm Post subject: |
|
|
ocbMaurice wrote: | - Set certain Directories to be excluded from the squash image |
I think the proper way to exclude mounted subdirectories is to specify it in the mksquashfs-options with -ef (which can be done with the other script).
If you want to mount/umount them automatically, I suggest you do this in another init-script which "need"s the squash_dir-script: If all would be done in the same script, then in case of errors, it would be unreasonable hard to track down the problems; e.g. what to do if mounting to a temporary directory succeeds, but the aufs-mount fails or something similar. Not to speak about the problems which happen if you are in a --make-shared/--make-slave directory or something similar.
Quote: | my script sets the permissions for the new mounted dir to the same as the old (chmod/chown) |
I do not understand this: Why should it be unreasonable to use the permissions stored in the squash-file? The content also has the permissions stored in that file, so why should it be different for the directory itself?
Quote: | p.s. I still don't know if it's possible to query if anything has changed. IMO it's not possible to know that anything has changed if you only delte files |
Not sure what you mean. The other script does detect this: If you delete files, aufs will store a corresponding file in the writable branch. Of course, this would not go well with the threshhold option. Or do you mean deletion of flies which were only created during the last boot? These do not have to be detected, since if all of them are deleted, there is no reason to resquash the image. |
|
Back to top |
|
|
x-demon n00b
Joined: 27 Oct 2009 Posts: 1
|
Posted: Tue Oct 27, 2009 11:39 am Post subject: |
|
|
Is it possible to recreate squash img based on old img? Because building image from scratch takes about 30-40 mins on my netbook... |
|
Back to top |
|
|
^marcs Apprentice
Joined: 09 Mar 2005 Posts: 169
|
Posted: Wed Mar 23, 2011 3:58 pm Post subject: |
|
|
Sorry for digging such an old topic, but it's best I can find.
Anyone managed to install aufs2 patch in 2.6.38?
Way those patches are distributed is a mystery to me |
|
Back to top |
|
|
lazy_bum l33t
Joined: 16 Feb 2005 Posts: 691
|
|
Back to top |
|
|
tclover Guru
Joined: 10 Apr 2011 Posts: 516
|
Posted: Wed Apr 13, 2011 2:16 am Post subject: issue with umount /some/file.sfs |
|
|
First off, thanks for this tricks and tips. I cannot possibly run my system on now without usr mounted on a squashsf/aufs combo. Once you get used to the responsiveness of the thing, you cannot come back.
Here are two modified scripts grabbed from this thread, I'll just post 'em in case, well if ever, somebody could find some use. No waranty whatsover of course nor for some usefulness:
Quote: | #!/bin/sh
#~/sqfsd-build.sh
BSIZE=131072
COMP=xz
EXCL="/lib64/rc/init.d/* /lib64/splash/cache/*"
sqfsd="lib64"
sqfsdir="/sqfsd"
FSTAB=n
[ -n "$EXCL" ] && e="-e $EXCL"
die() {
echo "$1"
exit 1
}
for s in ${sqfsd}
do
/bin/echo "#"
/bin/echo "# [re]building squashed-${s} #"
/bin/echo "#"
/bin/mkdir -p ${sqfsdir}/${s}/{ro,rw} || die "failed to create $s/{ro,rw} dirs."
/usr/bin/mksquashfs /${s} ${sqfsdir}/${s}_tmp.sfs -b ${BSIZE} -comp ${COMP} $e || die "failed to build $s.sfs img."
[ -n "`mount -t aufs | grep ${s}`" ] && { /bin/umount -l /${s} || die "failed to umount $s aufs branch."; }
[ -n "`mount -t squashfs | grep ${sqfsdir}/${s}/ro`" ] && { /bin/umount -l ${sqfsdir}/${s}/ro || die "failed to umount sfs img."; }
/bin/rm -rf ${sqfsdir}/${s}/rw/* || die "failed to clean up $s/rw."
[ -e ${sqfsdir}/${s}.sfs ] && /bin/rm -f ${sqfsdir}/${s}.sfs
/bin/mv ${sqfsdir}/${s}_tmp.sfs ${sqfsdir}/${s}.sfs || die "failed to move tmp.sfs img."
#
# set FSTAB=y if you want localmount to take care of mounting the squashed stuff.
# For disk squashed based stuff, you don't realy need to add sqfsmount init service,
# localmount will mount the necessary stuff with the following two lines.
#
[ "${FSTAB}" = "y" ] && {
/bin/echo "${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro squashfs noatime,nodev,loop,ro 0 0" >> /etc/fstab || die "fstab er.1"
/bin/echo "${s} /${s} aufs udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro 0 0" >> /etc/fstab || die "fstab er.2"
}
/bin/mount -t squashfs ${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro -o noatime,nodev,loop,ro || die "failed to mount $s.sfs img."
/bin/mount -t aufs ${s} /${s} -o udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro || die "failed to mount $s aufs branch."
#
# start /etc/init.d/sqfsmount and add it in the boot runlevel if you added the init script
# and don't forget to add /etc/conf.d/sqfsmount configuration file... else it wont do anything.
/bin/echo "#"
/bin/echo "# squashed-${s} sucessfully [re]build #"
/bin/echo "#"
done |
Now fstab is optional, I keep only usr lines related as a safe net, as the contains of usr disapeared, so localmount can mount it if need be:
Quote: | /sqfsd/usr.sfs /sqfs/usr/ro squashfs noatime,nodev,loop,ro 0 0
usr /usr aufs udba=reval,br:/sqfs/usr/rw:/sqfs/usr/ro 0 0 |
And here my /etc/init.d/sqfsmount:
Quote: | #!/sbin/runscript
#!/sbin/runscript
# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
#
#/etc/init.d/sqsfdmount
#
# Adding need dependancies is required, after removing usr, to avoid bootup errors, else you can add usr[squashed] in fstab.
# so localmount will take care of mounting usr before acpid, bootmisc, consolefont and keymaps use some binary in usr.
# for exemple, usr is mounted from a sqfsd image. You can always add usr required 2 lines in fastab, the init script will check anyway,
# if usr is mounted, it will not mount it. Comment out/in the before line if need be.
description="Mounts squashed directory [according to... /etc/fstab if you want to]."
depend()
{
need fsck localmount
use lvm modules mtab
after lvm modules
#before consolefont bootmisc acpid keymaps
keyword -jail -openvz -prefix -vserver -lxc
}
start() {
[ "${USQFS}" = "y" ] && {
echo "mounting flash-squashed dirs"
sqfsd_mount "${USQFSDIR}" "${USQFSD}";
}
echo "mounting disk-squashed dirs"
sqfsd_mount "${SQFSDIR}" "${SQFSD}"
}
stop() {
[ "${USQFS}" = "y" ] && {
echo "umounting flash-squashed dirs"
sqfsd_mount "${USQFSDIR}" "${USQFSD}";
}
echo "umounting disk-squashed dirs"
sqfsd_mount "${SQFSDIR}" "${SQFSD}"
sleep 1
}
sqfsd_mount() {
sqfsdir="${1}" sqfsd="${2}"
for s in ${sqfsd} ; do
if [ -n "`mount -t squashfs | grep "${sqfsdir}/${s}"/ro`" ]; then
echo "${sqfsdir}/${s}/ro already mounted."
else
ebegin "mounting squashed ${s}image..."
mount -t squashfs ${sqfsdir}/${s}.sfs ${sqfsdir}/${s}/ro -o noatime,nodev,loop,ro
eend $? "mount squashed ${s} image failed."
fi
if [ -n "`mount -t aufs | grep /${s}`" ]; then
echo "${s} aufs branch already mounted"
else
ebegin "mounting ${s} aufs branch..."
mount -t aufs ${s} /${s} -o udba=reval,br:${sqfsdir}/${s}/rw:${sqfsdir}/${s}/ro
eend $? "mount squashed ${s} aufs branch failed"
fi
done
}
sqfsd_umount() {
sqfsdir="${1}" sqfsd="${2}"
for s in ${sqfsd} ; do
ebegin "umounting squashed ${s} aufs branch..."
umount -lt aufs /${s} -O no_netdev 2>/dev/null
eend $? "umount squashed ${s} aufs branch failed."
ebegin "umounting squashed ${s} image..."
umount -lt squashfs ${sqfsdir}/${s}/ro -O no_netdev 2>/dev/null
eend $? "umount squashed ${s} image failed."
done
}
# vim:ts=4 |
Quote: | #/etc/conf.d/sqfsdmount
SQFSDIR="/sqfsd"
SQFSD="usr sbin opt var/db var/lib/layman"
USQFS="n"
USQFSDIR="/mnt/usqfsd"
USQFSD="lib32" |
An yeah, the idea of getting /lib32 and /lib64 squashed seems appealing but I have an issue when booting. I have baselayout-2.0 openrc-0.8 and it seems /sbin/init write in /lib64 in the bootup process. When the bootup process end some services 'die' somewhere. I guess some services infos get masked after the squashed files got mounted. I don't have any guess on what is happening in the bootup process. I didn't have the time to dig up about that.
EDIT: EDITED Mai 09, 2011, new update version of sqfsd-build.sh. And now I have almost the same script for fcron. I modified the sqfsd="some dirs" line to sqfsd="`echo ${1}|tr ':' ' '`" and named the script sqfsd-rebuild.sh and put it to /root. So now you can call the rebuild script using a line like this in crontab:
* * 1,15 * * root $HOME/sqfsd-rebuild.sh usr:bin:sbin:lib32
or/and
* * 1,10,20 * * root $HOME/sqfsd-rebuild.sh var/db:var/cache/edb:var/lib/layman:var/portage:var/local/portage
say you can put different cron job depending on your needs.
EDIT2: WHAT'S SO GOOD ABOUT SQUASHING STUFF YOU ASK!?
I happen to have wiped out the whole [squashed] usr, after having wiped out the real usr, directory. Well, with a squashed usr... you have to recover only your squashed image. A go in debugfs </dev/sdxx> and then a "cd /sqfsd && ls -d" you note the inode number for usr.sfs and everything is back in no time! And, of course a wiped out squashed file is easy to recover... just rm the .wh* related file in */rw and...
No good you say!? Doesn't worth the trouble?
I could never go back to the much slower usr again. Just the pain to wait emerge to... calculate dependencies and...
Last edited by tclover on Tue May 10, 2011 11:29 am; edited 13 times in total |
|
Back to top |
|
|
tclover Guru
Joined: 10 Apr 2011 Posts: 516
|
Posted: Wed Apr 13, 2011 10:56 pm Post subject: |
|
|
just adding a side note of my previous post for those who would want to grab sqfsmount init script, you still had to add a 'need dependency' to the following init services, otherwise you will get errors in the bootup process, but your system could just work with 'em, those errors disappear after sqfsmount is runned: acpid, bootmisc, consolefont, setfont. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|