Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage tree using squashfs
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 850
Location: PL

PostPosted: Sat Mar 25, 2006 1:14 pm    Post subject: Reply with quote

combined with cdb-portage tweak this thing is a killer :]

i used to keep my portage on reiser4 loopfile. and i use cdb. it was very fast. but squashfs metadata update with cdb is ~3x faster than that :]

(not mentioning the update-eix, with metadata setting for portage cache :D)

i wish it had support for overlays as well :D

oh, i use squashfs 3.0 from archck-sources 2.6.16-archck1. work s like a charm :]
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Sat Mar 25, 2006 3:10 pm    Post subject: Reply with quote

Well, I don't see any problems running the overlays on squashfs/unionfs, I already think how it will be better to partition here for this to work good.
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 850
Location: PL

PostPosted: Sun Mar 26, 2006 10:52 am    Post subject: Reply with quote

Gergan Penkov wrote:
Well, I don't see any problems running the overlays on squashfs/unionfs, I already think how it will be better to partition here for this to work good.
no, the problem was in my config. i had portage in /usr/portage/maintree and overlays in /usr/portage/overlays and whole /usr/portage was on a reiser4 loopback.

so i converted the entire loopback into squashfs, but the script expected to have portage only on that squashfs image. so my overlays got erased :/
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Tue Mar 28, 2006 1:50 pm    Post subject: Reply with quote

I am testing jffs2 for /usr/portage, it's +/- 75MB so 3x bigger than squasfs but it's rw. Jffs2 + cdb is like portage of 2000.
_________________
http://wojia.be
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Tue Mar 28, 2006 2:06 pm    Post subject: Reply with quote

Cool!
I had tried jffs2 before, but I never got it working particularly well.
Any tips?
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Thu Mar 30, 2006 11:19 am    Post subject: Reply with quote

ok I will do that once I have things sorted out
_________________
http://wojia.be
Back to top
View user's profile Send private message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Sun Apr 30, 2006 3:53 pm    Post subject: Reply with quote

I've updated emj's original script to include some error checking and error messages. It's probably still far from a pretty way to make Bash scripts, but it seems to work ;)
Before using you should set RAMSIZE, IMAGEDIR and RAMDISKDIR (in the script) to apropriate values. I haven't really tested this script yet, but it probably should work. It lets you know exactly what it is doing and when something goes wrong you (should) get a semi-descriptive error message.
I found that if you umount /usr/portage with the '-l' option (--lazy) the '-d' (clear loopdevice) option loses it functionality, thus filling up your loop devices. Uhm, so don't use that option :p

The script:
psync:
#!/bin/bash
#
#myemerge_sync.sh - script to emerge --sync from a squashfs
#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location
#emerge --sync's, then re-squashes portage and remounts
#also runs an update-eix to update eix database
#
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.
#I use the ArchCK ebuilds (search the gentoo forums) to get the above in the kernel.
#Also, remember that it's very important to have DISTFILES set somewhere
#other than /usr/portage/distfiles
#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf
#
#Preparatory steps to running this the first time are as follows:
#
#emerge squashfs-tools
#mkdir /images
#mksquashfs /usr/portage /images/portage.sqfs
#mv /usr/portage /usr/portage_
#mkdir /usr/portage
#echo "/images/portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
#mount /usr/portage
#
#...then try running this script. When you confirm that everything's working you
#can rm -rf /usr/portage_ to remove the old uncompressed portage tree, or restore
#it if the script doesn't work for you.
#
#~Ev
#
# Given a thorough make-over by Da Fox


#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.
#100M should suffice unless you've failed to sync for a very, very long time.
#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.
#It will be unloaded when done sync'ing.
RAMSIZE="128M"
IMAGEDIR="/var/ftp"
RAMDISKDIR="/var/tmp/ramdisk"

#######################

echo -n Checking for a place to store the SquashFS image...\
if mkdir -p ${IMAGEDIR} &> /dev/null ; then
  echo found ${IMAGEDIR}
  echo -n Checking for a place to mount the ramdisk...\ 
  if mkdir -p ${RAMDISKDIR} &> /dev/null ; then
    echo found ${RAMDISKDIR}
    echo -n Cleaning ${RAMDISKDIR}...\
    if rm -rf ${RAMDISKDIR}/* ; then
      echo done
      echo -n Checking if /usr/portage is a mounted SquashFS...\
      if cat /proc/mounts | grep squashfs | grep "\ \/usr\/portage\ " &> /dev/null ; then
        echo ok, it is.
        #Since the ramdisk is part of the unionfs we should make sure that the union
        #isn't active beforehand, otherwise we'll get a lot of stale mounts for ${RAMDISKDIR}
        if cat /proc/mounts | grep unionfs | grep "\ \/usr\/portage\ " &> /dev/null ; then
          #If the unionfs is already mounted on /usr/portage we can't just remount it
          #Try to umount it || die
          echo \*\* Warning: /usr/portage is already mounted as an UnionFS \*\*
          echo -n \ \ \ attempting to umount /usr/portage...\
          if umount -d /usr/portage &> /dev/null ; then
            echo ok. Proceeding as usual.
          else
          #Something bad has happened
            echo \ \ \ Failed!
            echo \ \ \ This really shouldn\'t happen. Please make sure /usr/portage is unmounted
            echo \ \ \ before running this script again.
            exit $?
          fi
        fi
        if umount -d ${RAMDISKDIR} &> /dev/null ; then
          echo Unmounted existing ramdisk from ${RAMDISKDIR}
        fi
        echo -n Mounting ramdisk of ${RAMSIZE} on ${RAMDISKDIR}...\
        if mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR &> /dev/null ; then
          echo ok
          echo -n Mounting UnionFS overlay on /usr/portage...\
          if mount -t unionfs -o dirs=${RAMDISKDIR}=rw:/usr/portage=ro none /usr/portage &> /dev/null ; then
            echo ok
            #Now we can finally sync the tree :D
            if which eix &> /dev/null ; then
              echo -n Now running eix-sync, please standby...\
              if eix-sync &> /dev/null ; then
                echo done
              else
                echo failed!
                echo An error occurred while running eix-sync.
                echo Please make sure that eix-sync works first!
                exit $?
              fi
            else
              echo \*\* Warning: You are running this script, but are not using eix \*\*
              echo \*\* eix is a really nice tool and you should emerge it asap. \;\) \*\*
              echo -n Now running emerge --sync, please standby...\
              if emerge --sync &> /dev/null ; then
                echo done
              else
                echo failed!
                echo An error occurred while running emerge --sync.
                echo Please make sure that emerge --sync works first!
                exit $?
              fi
            fi
            #create a backup of the old image, user *should* have write permissions
            echo -n Creating temporary backup of ${IMAGEDIR}/portage.sqfs...\
            if mv ${IMAGEDIR}/portage.sqfs ${IMAGEDIR}/portage.sqfs.bak &> /dev/null ; then
              echo ok
            else
              echo failed!
              echo Is your ${IMAGEDIR}/portage.sqfs missing?
              exit $?
            fi
            echo -n Updating portage SquashFS image...\
            if mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs -check_data &> /dev/null ; then
              echo ok
              echo -n Removing backup file...\
              if rm -f ${IMAGEDIR}/portage.sqfs.bak &> /dev/null ; then
                echo ok
              else
                echo failed!
                echo You should manually remove ${IMAGEDIR}/portage.sqfs.bak
              fi
              echo -n Now umounting /usr/portage union...\
              if umount -d /usr/portage &> /dev/null ; then
                echo ok
              else
                echo failed!
                echo Please make sure to umount the UnionFS from /usr/portage.
              fi
              echo -n Now umounting ${RAMDISKDIR}...\
              if umount -d ${RAMDISKDIR} &> /dev/null ; then
                echo ok
              else
                echo failed!
                echo Please make sure to umount the ramdisk from ${RAMDISKDIR}.
              fi
              echo -n Now umounting /usr/portage SquashFS...\
              if umount -d /usr/portage &> /dev/null ; then
                echo ok
                echo -n Now remounting /usr/portage SquashFS...\
                if mount -t squashfs -o ro,loop ${IMAGEDIR}/portage.sqfs /usr/portage &> /dev/null ; then
                  echo ok
                  echo All done, you may now enjoy your Updated Portage Tree :p
                else
                  echo failed!
                  echo Please make sure to mount ${IMAGEDIR}/portage.sqfs back on /usr/portage,
                  echo otherwise you won\'t have a Portage tree.
                fi
                exit $?
              else
                echo failed!
                echo Please make sure to umount \&\& mount the SquashFS from /usr/portage,
                echo in order to take advantage from the updated SquashFS image.
              fi
              exit $?
            else
              echo failed!
              echo
              echo Something funny has happened:
              echo You *should* have write permissions on ${IMAGEDIR}
              echo You *should* have mksquashfs
              echo You didn\'t cancel the creation process, now did you?
              echo -n restoring backup...\
              if  mv ${IMAGEDIR}/portage.sqfs.bak ${IMAGEDIR}/portage.sqfs &> /dev/null ; then
                echo ok
                echo please run \'mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs\' manually.
                echo \(Don\'t forget to make appropriate backups of ${IMAGEDIR}/portage.sqfs first!\)
              else
                echo failed!
                echo This is bad. Please review your config and make sure nothing
                echo funny is going on with ${IMAGEDIR}, and no other users/processes are
                echo writing to your SquashFS \(${IMAGEDIR}/portage.sqfs\)
                exit $?
              fi
            fi
          else
            echo failed!
            echo Please make sure you have UnionFS installed and it\'s module is loaded.
            echo emerge unionfs \&\& insmod /lib/modules/`uname -r`/fs/unionfs.ko
          fi
        else
          echo failed!
          echo Please make sure you have permission to mount ${RAMDISKDIR}.
        fi
      else
        echo failed!
        echo
        echo Before running this script you should:
        echo \ \ emerge squashfs-tools
        echo \ \ mkdir ${IMAGEDIR}
        echo \ \ mksquashfs /usr/portage ${IMAGEDIR}/portage.sqfs -check_data
        echo \ \ mv /usr/portage /usr/portage_
        echo \ \ mkdir /usr/portage
        echo \ \ echo \"${IMAGEDIR}/portage.sqfs /usr/portage squashfs ro,loop 0 0\" \>\> /etc/fstab
        echo \ \ mount /usr/portage
        echo \ \ Read further instructions in \'`pwd`/`echo ${0} | sed "s/.*\///"`\'
      fi
    else
      echo failed!
      echo Please make sure you have write permissions on ${RAMDISKDIR}.
    fi
  else
    echo failed!
    echo Please make sure you have write permissions on ${RAMDISKDIR}.
  fi
else
  echo failed!
  echo Please make sure you have write permissions on ${IMAGEDIR}.
fi
exit 0

_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator


Last edited by Da Fox on Fri May 05, 2006 10:48 am; edited 2 times in total
Back to top
View user's profile Send private message
BigMichi1
Tux's lil' helper
Tux's lil' helper


Joined: 25 Oct 2003
Posts: 82
Location: Dresden, Deutschland

PostPosted: Mon May 01, 2006 9:19 am    Post subject: Reply with quote

Quote:
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.

unionfs kernel module (if it's not already in the used kernel) can also be installed through
Code:
emerge unionfs


also there is something wrong in the Preparatory steps in the header of the script:
Code:
mksquashfs /usr/portage /images/usr.portage.sqfs
and
Code:
echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab

the usr.portage.sqfs should be portage.sqfs else the script fails when it tries to create the temporary backup
_________________
Internet is toll, da trifft man Leute von denen man nie gedacht hätte das es Sie gibt.
Back to top
View user's profile Send private message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Mon May 01, 2006 10:20 am    Post subject: Reply with quote

[quote="BigMichi1"]
Quote:
also there is something wrong in the Preparatory steps in the header of the script:
Code:
mksquashfs /usr/portage /images/usr.portage.sqfs
and
Code:
echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab

the usr.portage.sqfs should be portage.sqfs else the script fails when it tries to create the temporary backup


Thanks, I've made the changes in the posted script. It was called that in the original script, I really haven't change any of the introductory comments :)
_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator
Back to top
View user's profile Send private message
synss
Apprentice
Apprentice


Joined: 08 Mar 2006
Posts: 282
Location: Dijon > Berlin > Tokyo > Nürnberg > München

PostPosted: Tue May 02, 2006 12:47 am    Post subject: Reply with quote

Last edited Sat 2006-05-17 v. 0.1.2
Split and updated here

Very nice!!!

A simple init script will do, and I assume /dev/shm being tmpfs (that's the default) and the image is laready prepared, I might make it more general if I have time something like
Code:
if [ ! -e $PORTAGE_SQFS ]; then create it; fi


anyway

Code:
echo "unionfs" >> /etc/modules.autoload.d/kernel-2.6
echo "squashfs" >> /etc/modules.autoload.d/kernel-2.6


/etc/conf.d/portage.sqfs
Code:
PORTAGE_SQFS="/var/tmp/portage.sqfs"
PORTAGE_RO="/media/portage_ro"
PORTAGE_RW="/dev/shm/portage"
PORTAGE_TMP="/var/tmp/portage.tmp"
PORTDIR="/usr/portage"


/etc/init.d/portage.sqfs
Code:
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

depend() {
   need localmount
}

start() {
   ebegin "Mounting squashfs'ed portage tree"
   [ -d $PORTAGE_RO ] || mkdir -p $PORTAGE_RO
   mount -t squashfs -o loop,ro $PORTAGE_SQFS $PORTAGE_RO
   [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
   mount -t unionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR
   eend 0
}

stop() {
   ebegin "Updating portage tree"
   if [ "$(du -s $PORTAGE_RW | cut -f 1)" -gt 1 ]; then
      einfo "Syncing the tree"
      mksquashfs $PORTDIR $PORTAGE_TMP -check_data
   else
      einfo "Nothing to do"
   fi
   eend 0
   ebegin "Unmounting the tree"
   umount $PORTDIR
   umount $PORTAGE_RO
   rm -rf $PORTAGE_RW
   [ -e $PORTAGE_TMP ] && mv -f $PORTAGE_TMP $PORTAGE_SQFS
   eend 0
}

_________________
Compress portage tree
Elog viewer
Autodetect swap


Last edited by synss on Thu May 25, 2006 2:29 pm; edited 7 times in total
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 691

PostPosted: Fri May 05, 2006 7:43 am    Post subject: Reply with quote

Hi.
Can someone "unify" this stuff about portage with unionfs? New topic?


Oh, and squashfs-tools 3.0 broke my squashed portage...
Code:
dmesg | tail
*snip*
SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)
SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)

Ideas? New kernel module with 3.0 support? (or change to unionfs? ;-)

::edit::
Don't know why "watching" this topic isn't sending any mail.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Fri May 05, 2006 10:51 am    Post subject: Reply with quote

lazy_bum wrote:
Hi.
Can someone "unify" this stuff about portage with unionfs? New topic?


Oh, and squashfs-tools 3.0 broke my squashed portage...
Code:
dmesg | tail
*snip*
SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)
SQUASHFS error: Major/Minor mismatch, filesystem is (3:0), I support (1 : x) or (2 : <= 1)

Ideas? New kernel module with 3.0 support? (or change to unionfs? ;-)

::edit::
Don't know why "watching" this topic isn't sending any mail.


kernels <2.6.16 don't support SquashFS 3.0, this was added in 2.6.16. So you should update to a more recent kernel.
_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator
Back to top
View user's profile Send private message
bluefox81
n00b
n00b


Joined: 01 Feb 2006
Posts: 7

PostPosted: Mon May 08, 2006 10:45 am    Post subject: rsync freeze during emerge --sync Reply with quote

I'm using suspend-sources 2.6.14, unionfs 1.1.4.2-r2, rsync 2.6.8 with the rc scripts... well all seems working, no problem on mount squashfs filesystem (v 2.0...) nor with unionfs.
I updated my portage-tree once for testing... just adding some ebuilds and all done.
Now while trying a "serious" update rsync stop doing anything (without exiting, it just seems freezed) more precisely deleting files

Code:

deleting sys-apps/portage/files/2.1_pre7/1170_r3084_bug_128362.patch
deleting sys-apps/portage/files/2.1_pre7/1160_r3083_bug_126801.patch
deleting sys-apps/portage/files/2.1_pre7/1150_r3082_bug_117713.patch
deleting sys-apps/portage/files/2.1_pre7/1140_r3077_noclean.patch
deleting sys-apps/portage/files/2.1_pre7/1130_r3087_bug_129098.patch
deleting sys-apps/portage/files/2.1_pre7/1120_r3094_bug_129193.patch
deleting sys-apps/portage/files/2.1_pre7/1110_r3096_emerge_args_validation.patch
deleting sys-apps/portage/files/2.1_pre7/1100_r3063_bug_128506_distdir_error.patch
deleting sys-apps/portage/files/2.1_pre7/1090_r3055_bug_128284_get_open_fds.patch
deleting sys-apps/portage/files/2.1_pre7/1080_ensure_dirs.patch
deleting sys-apps/portage/files/2.1_pre7/1070_r3036_bug_127930_unpack_timestamp.patch
deleting sys-apps/portage/files/2.1_pre7/1060_r3034_bug_127897_realpath.patch
deleting sys-apps/portage/files/2.1_pre7/1050_forum_3210399_empty_digest.patch
deleting sys-apps/portage/files/2.1_pre7/1040_r3015_user_fetch.patch
deleting sys-apps/portage/files/2.1_pre7/1030_r3009_bug_127563_ccache_stat.patch
deleting sys-apps/portage/files/2.1_pre7/1020_r3006_bug_127573_cachedir.patch
deleting sys-apps/portage/files/2.1_pre7/1010_r3004_lazy_virtuals.patch
deleting sys-apps/portage/files/2.1_pre7/1000_r2994_workdir_mode.patch

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(242) [receiver=2.6.8]


it catch the ^C but no other consequences...
waiting without sending a signal rsync print timeout error at io.c [receiver=2.6.8]

kill -9 or other signals doesn't work... rsync cannot be stopped, and obviously unionfs connot be unmounted
I think it's a unionfs problem... but dmesg doesn't output anything, otherwhise the way rsync is locked bring me to think it's a kernel/module level problem...

any ideas...
pleaze... i really like and need a squashfs rw portage-tree....
thanks in advance at all guy!
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Mon May 15, 2006 12:05 pm    Post subject: portage and compressed filesystem Reply with quote

Ok as promised I have put a guide of portage on jffs2 on my home page
_________________
http://wojia.be
Back to top
View user's profile Send private message
bluefox81
n00b
n00b


Joined: 01 Feb 2006
Posts: 7

PostPosted: Mon May 15, 2006 9:48 pm    Post subject: Reply with quote

tnx... can u give an idea of how long takes mount this particular fs? advantages and disadvantages vs squash fs + unionfs (apart than proably with jffs2 all works properly ;-))... because the only time i correctly sync my portage tree (just some ebuild added, there wasn't need to delete anything) all works very fast (ie. dependencies deep scans, searches, etc) , also updating the squashfs file taken few seconds... (i tought REALLY GREAT!!!! until those problems appears... :-()... so if u give me... and us... a performance idea would be really nice
tanks in advance!
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Mon May 15, 2006 10:08 pm    Post subject: Reply with quote

jffs2 vs squashfs +unionfs

pro

  • no kernel patches
  • simple, just one package to emerge
  • less change for breakage
  • no need to merge back the changes


contra

  • long mount time, 15 sec to 1 min depending on the hardware
  • use more disk space than squashfs (+/- 3x)

xming
_________________
http://wojia.be
Back to top
View user's profile Send private message
bluefox81
n00b
n00b


Joined: 01 Feb 2006
Posts: 7

PostPosted: Mon May 15, 2006 10:39 pm    Post subject: Reply with quote

tnx... maybe the "long" time for mount the fs could be secondary... if i'm right we can mount /usr/portage only the first time we use emerge... so a simple script can do that without wait during the boot process...
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Mon May 15, 2006 11:16 pm    Post subject: Reply with quote

yes, you can use the code on my site and put it in a file and execute it before the 1st time you do a emerge
_________________
http://wojia.be
Back to top
View user's profile Send private message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Tue May 16, 2006 12:01 pm    Post subject: Reply with quote

Could you please explain why you would want to use jffs2 on your harddisk? From what I gather on the wikipedia entry it is designed for use on Flash devices.
There is also no mention of it being significantly faster (or slower for that matter) then any other filesystem, so why are you experiencing such speed up? If it is because you think that there is less disk accesses that has to be wrong, as the whole squashfs image only takes 33MB, which is easily chached in memory, and the tmpfs is also completely in memory, so there shouldn't be a lot of disk activity right? The one thing that does take some time is the (re)creation of the squashfs image.
_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Tue May 16, 2006 5:51 pm    Post subject: Reply with quote

First of all it only takes 90 MB instead of 500+ MB so there is more chance that thing gets cached and it's still 80 % saving on disk space. And there is a significant speed up, because there is less (80%) to read from the disk. I am sure that squashfs is faster but it's read only, and I have my doubts if squashfs + unionfs is faster that jffs2.

Anyway I am not forcing you to use jffs2.
_________________
http://wojia.be
Back to top
View user's profile Send private message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Tue May 16, 2006 8:55 pm    Post subject: Reply with quote

ofcourse you're not forcing anyone, it's all about choice 8)
I was just wondering why you were using jffs2, since it's designed for flashmemory. I have now also read some of your other reasons on your homepage, and they are good reasons. It's true that squashfs is readonly, but then again you don't need it to be read/write that often, only when you 'emerge --sync'. But most of the speed up is indeed because of caching and that is true for both filesystems. However I think that with the time it takes to mount jffs2 every time it might turn out that squashfs+unionfs is actually faster, rebuilding the image only takes about a minute. So most of the time as always lost in downloading the new files and updating the eix cache.
One last thing, squashfs 3.0 might not be in vanilla sources, but it definitely is in gentoo-sources, so you have that anyway. :)
_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 441

PostPosted: Wed May 17, 2006 1:18 pm    Post subject: Reply with quote

mount time is a non issue for me, reiser4 has also long mount time (less than jffs2 but longer than ext2, and even ext3 has longer mount time than ext2. I only have to mount once after reboot so on servers this is really not a problem and on my laptop I use software suspend so it's neither a isse here.

However I do emerge --syn approx. once a day, with squashfs I have to rebuild my fs each day, jffs2 just works. I did not choose jffs2 for any flash orientated features, I choose jffs for simplicity, less space (than ext/3, reiser) and speed.

I would have used e2compr if it was updated and in the vanilla. ARe there any other compress fs in the vanilla kernel?

xming
_________________
http://wojia.be
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Thu May 25, 2006 2:53 pm    Post subject: Reply with quote

In fact as the tree is already cached in the memory and only the changes are on disk (as unionfs overlay - in the way I use it), rebuilding the squashfs image is really fast <1min, on the other hand unionfs does not need patching - simple kernel-module re-compile.
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
jjlawren
n00b
n00b


Joined: 06 Feb 2006
Posts: 36

PostPosted: Fri Jun 23, 2006 9:12 pm    Post subject: Reply with quote

Assume I'm using squashfs mounted by NBD over a network. What's the best way for updating the filesystem without causing problems on the other systems? Do they need to unmount it first, or can the squash file simply be replaced?
Back to top
View user's profile Send private message
enderandrew
l33t
l33t


Joined: 25 Oct 2005
Posts: 731

PostPosted: Sat Jun 24, 2006 11:19 am    Post subject: Reply with quote

Color me stupid, but if you've got a small HDD and old hardware, but wouldn't the better solution be simply to use something like prlock ( https://forums.gentoo.org/viewtopic-t-55031-highlight-prlock.html ) to cut down portage? The guy who wrote the script said he cut down his portage tree to 95% of its size.

Pros:

Free up HDD space
Syncs are FASTER since you are excluding all kinds of things
Don't have to take forever to squash anything
Don't have to have the HDD space free for unsquashing
Works on all kernels

Cons:

If you want to emerge something new that has been removed from portage, you have to use prlock again to add the ebuild back in, or download the ebuild manually.

Given that we're talking older hardware, there is a good chance that you don't emerge completely new software all that often, if for no other reason that lack of HDD space. You probably keep what you've got, and perhaps upgrade from time to time.
_________________
Nihilism makes me smile.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 3 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum