Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5 ... 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Thu Sep 13, 2007 12:13 pm    Post subject: Reply with quote

New POSIX compatible release 5.2

(For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).

synss wrote:
baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms.

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Thu Sep 13, 2007 2:12 pm    Post subject: Reply with quote

mv wrote:
I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.

Woot! Well done mv :D
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Fri Sep 14, 2007 5:09 am    Post subject: Reply with quote

mv wrote:
New POSIX compatible release 5.2

(For those who downloaded in the previous days/weeks: For some reason there was not 5.1 on the server but an older version).

synss wrote:
baselayout-2 is nearing completion and will only use POSIX shell initscripts, i.e. no bashisms.

I was in the opinion that also the previous version of the script did not use any bashisms, but I had missed some things which were not POSIX compatible. Now the thing has been tested with dash.


That was fast, thanks!

EDIT: just to say I have just tried it and it works well with dash. (But my system does not really boot faster.)
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Jan 01, 2008 9:46 pm    Post subject: Reply with quote

There is a new version of the initscript available at the initscripts. (The most important bugfix is that an unmount error is now not mistakenly ignored.)

Moreover, also a simple script can be found there which treats several of the corresponding initscripts simultaneously, if you want to recompress/start/stop/view the status of them manually.

Finally, there is also a patch for squashfs-tools-3.2_p2 which redirects the progress bar of mksquashfs to stderr. The advantage is that the init-script will then display the progress bar without printing additional confusing information (but you can also have the latter without the patch with the new version of the script).

If you use the script note that squashfs-tools-3.3 is meanwhile in the portage tree but will create a format which seems to be incompatible with the current gentoo kernel patchset, so you should not upgrade to squashfs-tools-3.3 in the moment...
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jan 06, 2008 4:13 pm    Post subject: Reply with quote

There is again a new version of the script available. Now it is possible to ignore certain files/dirs resp. ignore "touches" of these files/dirs (in the main directory). This is important if you squash a tex directory and want to avoid unnecessary re-squashing just because one of the system tools updated "ls-R" without actually making any changes to that file.
Back to top
View user's profile Send private message
Darknight
Guru
Guru


Joined: 26 Jan 2004
Posts: 483
Location: Italy

PostPosted: Mon Mar 24, 2008 11:32 pm    Post subject: Reply with quote

Thanks for this nice script.
Back to top
View user's profile Send private message
188562
Apprentice
Apprentice


Joined: 22 Jun 2008
Posts: 186

PostPosted: Mon Jun 23, 2008 3:54 pm    Post subject: Thanks Reply with quote

synss Thank you very much for the idea and its embodiment!
My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines

#mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

After that, everything worked! The truth is now at the time of each file portage.sqfs re-rebooting again this takes some time.
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Tue Jun 24, 2008 1:23 am    Post subject: Re: Thanks Reply with quote

init_6 wrote:
synss Thank you very much for the idea and its embodiment!
My five cents. At the moment I use sys-kernel/zen-sources-2.6.26_rc6-r10 I had a little rewrite your script to aufs all proper correction affected only lines

#mount -t unionfs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro unionfs $PORTDIR
mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR

After that, everything worked! The truth is now at the time of each file portage.sqfs re-rebooting again this takes some time.


Thank you! now I am not sure I understand your last sentence correctly: you do not need to reboot to update the image, you only need to restart the initscript, which can be done in a monthly cron job, too, BTW.
_________________
Compress portage tree
Elog viewer
Autodetect swap
Back to top
View user's profile Send private message
188562
Apprentice
Apprentice


Joined: 22 Jun 2008
Posts: 186

PostPosted: Tue Jun 24, 2008 3:24 pm    Post subject: Reply with quote

Every time I do a "reboot" or "halt" or "poweroff" script re-creates file portage.sqfs so if I update portage "emerge --sync"
Most likely this feature aufs but I'm not sure.
Back to top
View user's profile Send private message
IvanMajhen
Guru
Guru


Joined: 10 Jun 2006
Posts: 392
Location: Croatia

PostPosted: Tue Jun 24, 2008 11:38 pm    Post subject: Reply with quote

Add this
Code:
stop() {
   ebegin "Updating portage tree"
   rm -rf /dev/shm/.portage-rw/.wh..wh.plink
   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
   then
Back to top
View user's profile Send private message
188562
Apprentice
Apprentice


Joined: 22 Jun 2008
Posts: 186

PostPosted: Sat Jun 28, 2008 6:29 am    Post subject: Reply with quote

So here is the corrected version of the script for squashfs & aufs. All except out in Section stop working as it should work.

Code:
#!/sbin/runscript

source /etc/make.globals
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"

depend() {
    need localmount
}

start() {
    ebegin "Mounting read-only squashfs image"
    mount -rt squashfs -o loop $SQFS_CUR $PORTDIR
    retval=$?
    eend $retval
    [ $retval -ne 0 ] && return $retval

    ebegin "Mounting read-write with aufs"
    if [ ! $PORTAGE_RW ]
   then einfo " mounted in tmpfs (RAM)"
   PORTAGE_RW="${DEF_RW}"
    fi
    [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
    chmod 0750 $PORTAGE_RW
    chown portage:portage $PORTAGE_RW
    mount -t aufs -o nodev,noexec,dirs=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR
    eend $?
}

stop() {
#   ebegin "Updating portage tree"
#   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
#   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
#   then
#      einfo "  Syncing the tree"
#      mv -f $SQFS_NEW $SQFS_OLD
#      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
#      retval=$?
#      ln -sf $SQFS_NEW $SQFS_CUR
#   else
#      einfo "  Nothing to do"
#      retval=0
#   fi
#   eend $retval

    ebegin "Unmounting the tree"
    umount -t aufs $PORTDIR
    umount -t squashfs $PORTDIR
    rm -rf $PORTAGE_RW   
    eend 0
}



Here is this condition (if) why it is always true

Quote:
# ebegin "Updating portage tree"
# [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
# if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
Back to top
View user's profile Send private message
IvanMajhen
Guru
Guru


Joined: 10 Jun 2006
Posts: 392
Location: Croatia

PostPosted: Sun Jun 29, 2008 12:55 pm    Post subject: Reply with quote

This is wrong. Sqfs wont be updated when synced.

This is how my stop function looks:
Code:
stop() {
   ebegin "Updating portage tree"
   rm -rf /dev/shm/.portage-rw/.wh..wh.plink
   rm     /dev/shm/.portage-rw/.wh..wh.aufs

   [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
   if [ ! -z `ls -A $PORTAGE_RW | head -n1` ]
   then
      einfo "  Syncing the tree"
      mv -f $SQFS_NEW $SQFS_OLD
      mksquashfs $PORTDIR $SQFS_NEW -no-duplicates 2>/dev/null
      retval=$?
      ln -sf $SQFS_NEW $SQFS_CUR
   else
      einfo "  Nothing to do"
      retval=0
   fi
   eend $retval

   ebegin "Unmounting the tree"
   umount -t aufs  $PORTDIR
   umount -t squashfs $PORTDIR
   rm -rf $PORTAGE_RW
   eend 0
}
Back to top
View user's profile Send private message
NaiL
Apprentice
Apprentice


Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN

PostPosted: Sun Dec 14, 2008 11:56 am    Post subject: Reply with quote

I have using the MV's script for some time, (currently the last version).

I found a problem stopping the init script.

starting the script wrote:
/etc/init.d/squash_portage start
* Mounting /var/tmp/portage-current.sqfs as /usr/portage ... [ ok ]

mount command after the start wrote:
mount
/dev/sda3 on / type reiserfs (rw,noatime,notail)
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,gid=5,mode=620)
shm on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
securityfs on /sys/kernel/security type securityfs (rw,noexec,nosuid,nodev)
/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)
aufs on /usr/portage type aufs (rw,xino=/dev/shm/portage/.aufs.xino,br:/dev/shm/portage=rw:/tmp/squash_dir.readonly.4TQ2X8=ro)

stopping the script wrote:
# /etc/init.d/squash_portage stop
* Unmounting /usr/portage ...
umount: /var/tmp/portage-current.sqfs: not mounted
* umount -d -- /var/tmp/portage-current.sqfs failed [exit with 1] [ !! ]

mount command after the stop wrote:
mount
/dev/sda3 on / type reiserfs (rw,noatime,notail)
/proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,gid=5,mode=620)
shm on /dev/shm type tmpfs (rw,noexec,nosuid,nodev)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
securityfs on /sys/kernel/security type securityfs (rw,noexec,nosuid,nodev)
/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)

the config file wrote:
# cat /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

DIRECTORY="/usr/portage"
DIR_CHANGE="/dev/shm/portage"
FILE_SQFS="/var/tmp/portage-current.sqfs"
FILE_SQFS_OLD="/var/tmp/portage-old.sqfs"
DIR_TMP="/dev/shm"


Maybe is it a problem of configuration?
Back to top
View user's profile Send private message
Darknight
Guru
Guru


Joined: 26 Jan 2004
Posts: 483
Location: Italy

PostPosted: Wed Feb 25, 2009 2:49 pm    Post subject: Reply with quote

I have the same problem. As far as I know it used to work perfectly until some days ago but I may have missed that error for some time. Unfortunately I can't pinpoint any update, nor change the script to make it work with the new stuff...

EDIT: It has to do with temporary filenames. Set DIR_SQUASH as a workaround.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Feb 25, 2009 3:47 pm    Post subject: Reply with quote

Darknight wrote:
I have the same problem

Sorry, somehow I had missed NaiL's posting.

Anyway, I cannot reproduce the problem on my machine if I use exactly NaiL's config (only with a different DIRECTORY for testing purposes).
Do you use the most current version (7.4) of the script? (The version is in a comment near the beginning).

However, as remarked in the documentation, using the temporary filename feature for DIR_SQUASH is always hackish: It depends on the undocumented /etc/mtab format and thus it might depend on the mount command (i.e. on the util-linux version) whether this will work. During my test just now I had util-linux-2.14.1 installed: The critical part is whether in line 686-688 the correct filename can be read from /etc/mtab by the sed-expression.
Back to top
View user's profile Send private message
Darknight
Guru
Guru


Joined: 26 Jan 2004
Posts: 483
Location: Italy

PostPosted: Wed Feb 25, 2009 4:09 pm    Post subject: Reply with quote

I have the same util-linux version but somehow sed fails (I had nailed it, but I have not bash skillz to modify the script).
Anyway re-reading through the documentation gave me a solution that is good enough for me.
Back to top
View user's profile Send private message
NaiL
Apprentice
Apprentice


Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN

PostPosted: Sat Feb 28, 2009 2:43 pm    Post subject: Reply with quote

Hi,

I'm using the 7.4 version of the script and:

sys-apps/util-linux-2.14.1
sys-fs/squashfs-tools-3.3
sys-fs/aufs-20081208-r1
sys-kernel/gentoo-sources-2.6.27-r8

Do you need any other information?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Feb 28, 2009 3:58 pm    Post subject: Reply with quote

NaiL, apparently, the problem is your strange output of
mount wrote:
/dev/loop0 on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro)

This shouldn't be so. It should be instead something like
mount wrote:
/var/tmp/portage-current.sqfs on /tmp/squash_dir.readonly.4TQ2X8 type squashfs (ro,loop=/dev/loop0)

I cannot understand why your output is so. What happens after you do manually
Code:
mkdir /some_directory
mount -t squashfs -o loop,ro /var/tmp/portage-current.sqfs /some_directory

If this works as expected, the output of mount should then contain
mount wrote:
/var/tmp/portage-current.sqfs on /some_directory type squashfs (ro,loop=/dev/loop*)

and afterwards you should be able to umount by
Code:
umount /var/tmp/portage-current.sqfs

(of course, afterwards, you can remove /some_directory).

Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?
Back to top
View user's profile Send private message
NaiL
Apprentice
Apprentice


Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN

PostPosted: Sun Mar 01, 2009 12:19 pm    Post subject: Reply with quote

Quote:
I cannot understand why your output is so. What happens after you do manually
Code:
mkdir /some_directory
mount -t squashfs -o loop,ro /var/tmp/portage-current.sqfs /some_directory

If this works as expected, the output of mount should then contain
mount wrote:
/var/tmp/portage-current.sqfs on /some_directory type squashfs (ro,loop=/dev/loop*)

and afterwards you should be able to umount by
Code:
umount /var/tmp/portage-current.sqfs

(of course, afterwards, you can remove /some_directory).

The result was:
Quote:
/dev/loop1 on /some_directory type squashfs (ro)


Quote:
Perhaps it makes a change if you upgrade to sys-fs/squashfs-tools-3.4?

Affter the upgrade i get the same output.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Mar 01, 2009 12:44 pm    Post subject: Reply with quote

NaiL, I have no idea what is responsible on your system that the same mount-command (with same versions of mount and squashfs than here) works differently on your system. Do you have perhaps some of these terrible hal things activated or some special udev rules which could cause this behavior? Here are my useflags for util-linux:
Code:
crypt loop-aes nls slang unicode -old-linux -selinux -uclibc
Back to top
View user's profile Send private message
NaiL
Apprentice
Apprentice


Joined: 13 Feb 2003
Posts: 228
Location: Spain/BCN

PostPosted: Mon Mar 02, 2009 6:58 pm    Post subject: Reply with quote

Tnx! I finally found the problem.

I was some how related to "loop-aes" USE flag.

I had this USE flag disabled. By some reason it has to be activated, may be it will be good to document that some where.

I also noticed that. 8O
Quote:
* The loop-aes code has been split out of USE=crypt and into USE=loop-aes.
* If you need support for it, make sure to update your USE accordingly.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Mar 02, 2009 11:04 pm    Post subject: Reply with quote

In the new release of the script the temporary file name feature was removed for DIR_SQUASH. Instead, now a sane default is chosen for DIR_SQUASH in /var/run. Indeed, since even with the temporary file name feature the thing could only work by using a non-temporary file (in this case /etc/mtab), it is probably more reasonable to force directly to use a non-temporary directory. In this way the implementation is independent of undocumented behavior of the mount command.
Back to top
View user's profile Send private message
jowr
n00b
n00b


Joined: 27 Dec 2008
Posts: 52

PostPosted: Tue Mar 03, 2009 7:21 am    Post subject: Reply with quote

Where can I get a squashfs utils that makes a SquashFS 4 file system? Apparently 2.6.29 now only has v4, and there are no [known to me..] userland utils for making v4 file systems.
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Wed Apr 01, 2009 6:50 am    Post subject: Reply with quote

I've just upgraded to a 2.6.29 kernel, and I am now getting errors like this when trying to mount the squashfs:
Code:
SQUASHFS error: Major/Minor mismatch, older Squashfs 3.1 filesystems are unsupported

I may not be understanding correctly, but the information I'm reading on the Internet indicates that backward compatibility has been broken for squashfs as part of the process for bringing squashfs into the mainline kernel.

Here's a post from the author of squashfs:

http://lkml.indiana.edu/hypermail/linux/kernel/0901.1/02601.html

I'm not finding much information about this on squashfs' homepage on sourceforge

And I also see the entire set of sys-fs/squashfs-tools ebuilds have been hard-masked (as of the time of this post).

This looks like a bumpy ride for my squashfs portage directory.
Back to top
View user's profile Send private message
Darknight
Guru
Guru


Joined: 26 Jan 2004
Posts: 483
Location: Italy

PostPosted: Wed Apr 01, 2009 7:46 am    Post subject: Reply with quote

When I tried aufs didn't compile against 2.6.29, therefore I'm back to 2.6.27 for now.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5 ... 9, 10, 11  Next
Page 4 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum