Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage tree using squashfs
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 690

PostPosted: Thu Dec 22, 2005 8:04 am    Post subject: Reply with quote

frank_einstien wrote:
How do I get use squashfs without recompiling my kernal? Just asking...

Why not? You can add a module and then...
Code:
make modules && make modules_install


PS. squashed my portage and it works fine. With my 300 MHz on a home router i haven't noticed big slowdown (well "emerge sync" takes about... half an hour), and ~100 mb is nice cmp to >500 mb. Nice tip, thx.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
electrofreak
l33t
l33t


Joined: 30 Jun 2004
Posts: 713
Location: Ohio, USA

PostPosted: Thu Dec 22, 2005 6:26 pm    Post subject: Reply with quote

lazy_bum wrote:
frank_einstien wrote:
How do I get use squashfs without recompiling my kernal? Just asking...

Why not? You can add a module and then...
Code:
make modules && make modules_install


PS. squashed my portage and it works fine. With my 300 MHz on a home router i haven't noticed big slowdown (well "emerge sync" takes about... half an hour), and ~100 mb is nice cmp to >500 mb. Nice tip, thx.


I do stuff like that all the time with modules. Works fine.

As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method??
_________________
Desktop: ABit AN8, Athlon64 X2 4400+ 939 2.75GHz, 2x1GB Corsair XMS DDR400, 2x160GB SATA RAID-0, 2x20"W, Vista Ultimate x64
Laptop: 15.4" MacBook Pro 2.4Ghz, 2x1GB RAM, 160GB, Mac OS X 10.5.1
Server: PIII 550Mhz, 3x128MB RAM, 160GB, Ubuntu Server 7.10
Back to top
View user's profile Send private message
Liviu
n00b
n00b


Joined: 01 Mar 2005
Posts: 24
Location: Munich, Germany

PostPosted: Fri Dec 23, 2005 4:04 am    Post subject: Reply with quote

Alright! Really nice guide!

I got only around 6 gig for my linux partition on my Laptop (60G Harddrive).

I use my Home-Made P3-533 Gentoo Router to make my Squashed Portage Snapshots! It really doesn`t take too long.. perhaps 15 mins ?!?
Works perfectly for me so i don`t need to make one from a tar-snapshot or sync it and then squash it.

In my opinion this whole idea is very nice! I will distribute a copy of the .sqsh too .. I`ll just configure my cron...........*workinprogress*
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 690

PostPosted: Fri Dec 23, 2005 10:12 am    Post subject: Reply with quote

electrofreak wrote:

As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method??

Used the guide method. Router is online all the time so the emerge-sync-squashed is a cron job in the middle of the night.
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
bld
l33t
l33t


Joined: 26 Mar 2003
Posts: 759
Location: Outter Space

PostPosted: Fri Dec 23, 2005 4:58 pm    Post subject: Reply with quote

frank_einstien wrote:
Hi guys
How do I get use squashfs without recompiling my kernal? Just asking...
Thanks


NOTE: The squashFS patcheset is available for 2.6.13 and below. Hence I managed to rename the kernel version inside the patch. You should do this if your kernel is above 2.6.13 .. if something is not clear just ask.


# cd /root
# uname -r
2.6.14.2
# wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz
# tar zxvf squashfs2.2-r2.tar.gz
# cd squashfs2.2-r2/linux-2.6.13
# sed 's/2.6.13/2.6.14.2/g' squashfs2.2-patch > squashfs2.2-ready-patch
# cp squashfs2.2-ready-patch /usr/src/linux/ && cd /usr/src/linux && patch -p1 < squashfs2.2-ready-patch

done

choose, loop and squashFS as described below.. then type "make modules && make modules_install" inside the /usr/src/linux dir.. and you're all set..

Then follow the steps described above :-)
_________________
A happy GNU/Linux user!!
Back to top
View user's profile Send private message
Liviu
n00b
n00b


Joined: 01 Mar 2005
Posts: 24
Location: Munich, Germany

PostPosted: Wed Dec 28, 2005 12:22 am    Post subject: Reply with quote

Okay Folks !

A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):

http://home.arcor.de/liviufl/portage.sqsh/

I got 1 Gigabyte of Traffic for that webspace. It should be enough for around 33 Downloads ;)

have fun !
Back to top
View user's profile Send private message
emj
n00b
n00b


Joined: 01 Jan 2006
Posts: 5

PostPosted: Sun Jan 01, 2006 8:28 pm    Post subject: Reply with quote

Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all.

I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.

Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.

adsmith: I don't know why you got a kernel oops while testing unionfs. Perhaps it's because you didn't set the read/write layer to a tmpfs? Shouldn't matter, but... Or maybe it's just a bug in the code in your kernel... maybe try another kernel? unionfs has to be relatively stable, ordinarily, since most livecd are using it now as a fully filetree overlay, sooo... But there, it's a tmpfs overlay, again, so maybe it just likes that better. tmpfs is fine, really, since the changes accumulated during emerge syncs seem to only amount to a few megabytes of data in the ramdisk directory, which is cleared immediately after the sync, soo.

Anyway, after some testing, could you put this script into the main post? Thanks.

Code:

#!/bin/bash
#
#myemerge_sync.sh - script to emerge --sync from a squashfs
#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location
#emerge --sync's, then re-squashes portage and remounts
#also runs an update-eix to update eix database
#
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.
#I use the ArchCK ebuilds (search the gentoo forums) to get the above in the kernel.
#Also, remember that it's very important to have DISTFILES set somewhere
#other than /usr/portage/distfiles
#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf
#
#Preparatory steps to running this the first time are as follows:
#
#emerge squashfs-tools
#mkdir /images
#mksquashfs /usr/portage /images/usr.portage.sqfs
#mv /usr/portage /usr/portage_
#mkdir /usr/portage
#echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
#mount /usr/portage
#
#...then try running this script. When you confirm that everything's working you
#can rm -rf /usr/portage_ to remove the old uncompressed portage tree, or restore
#it if the script doesn't work for you.
#
#~Ev
#




#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.
#100M should suffice unless you've failed to sync for a very, very long time.
#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.
#It will be unloaded when done sync'ing.
RAMSIZE="100M"
IMAGEDIR="/images"
RAMDISKDIR="$IMAGEDIR/ramdisk"

#######################

mkdir -p $IMAGEDIR &> /dev/null
mkdir -p $RAMDISKDIR &> /dev/null
rm -rf $RAMDISKDIR/*

#only proceed if squashfs already mounted..
if [ "`cat /proc/mounts | grep squashfs | grep \/usr\/portage\ `" ]
then umount $RAMDISKDIR &> /dev/null
mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR
echo "** tmpfs of $RAMSIZE mounted on: $RAMDISKDIR"
mount -t unionfs -o dirs=$RAMDISKDIR=rw:/usr/portage=ro none /usr/portage
echo "** unionfs overlay mounted over /usr/portage"
emerge --sync
update-eix
mv -f /images/usr.portage.sqfs /images/usr.portage1.sqfs
mksquashfs /usr/portage /images/usr.portage.sqfs
umount -dl /usr/portage &> /dev/null
umount -dl /usr/portage &> /dev/null
umount -dl $RAMDISKDIR &> /dev/null
mount -t squashfs -o ro,loop /images/usr.portage.sqfs /usr/portage
fi


Last edited by emj on Tue Jan 10, 2006 1:31 am; edited 1 time in total
Back to top
View user's profile Send private message
electrofreak
l33t
l33t


Joined: 30 Jun 2004
Posts: 713
Location: Ohio, USA

PostPosted: Sun Jan 01, 2006 8:46 pm    Post subject: Reply with quote

looks like a nice script. I will try it soon (when I have time)
_________________
Desktop: ABit AN8, Athlon64 X2 4400+ 939 2.75GHz, 2x1GB Corsair XMS DDR400, 2x160GB SATA RAID-0, 2x20"W, Vista Ultimate x64
Laptop: 15.4" MacBook Pro 2.4Ghz, 2x1GB RAM, 160GB, Mac OS X 10.5.1
Server: PIII 550Mhz, 3x128MB RAM, 160GB, Ubuntu Server 7.10
Back to top
View user's profile Send private message
opentaka
l33t
l33t


Joined: 18 Feb 2005
Posts: 840
Location: Japan

PostPosted: Mon Jan 02, 2006 7:30 am    Post subject: Reply with quote

great howto, this helped my router server with 5GB HDD. yes, 500MB is huuuge :D
_________________
"Being defeated is often a temporary condition. Giving up is what makes it permanent" - Marilyn vos Savant
Back to top
View user's profile Send private message
bld
l33t
l33t


Joined: 26 Mar 2003
Posts: 759
Location: Outter Space

PostPosted: Sat Jan 07, 2006 10:43 am    Post subject: Reply with quote

emj wrote:
Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all.

I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.

Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.

adsmith: I don't know why you got a kernel oops while testing unionfs. Perhaps it's because you didn't set the read/write layer to a tmpfs? Shouldn't matter, but... Or maybe it's just a bug in the code in your kernel... maybe try another kernel? unionfs has to be relatively stable, ordinarily, since most livecd are using it now as a fully filetree overlay, sooo... But there, it's a tmpfs overlay, again, so maybe it just likes that better. tmpfs is fine, really, since the changes accumulated during emerge syncs seem to only amount to a few megabytes of data in the ramdisk directory, which is cleared immediately after the sync, soo.

Anyway, after some testing, could you put this script into the main post? Thanks.

Code:

#!/bin/bash
#
#myemerge_sync.sh - script to emerge --sync from a squashfs
#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location
#emerge --sync's, then re-squashes portage and remounts
#also runs an update-eix to update eix database
#
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.
#Also, remember that it's very important to have DISTFILES set somewhere
#other than /usr/portage/distfiles
#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf

#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.
#100M should suffice unless you've failed to sync for a very, very long time.
#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.
#It will be unloaded when done sync'ing.
RAMSIZE="100M"
IMAGEDIR="/images"
RAMDISKDIR="$IMAGEDIR/ramdisk"

#######################

mkdir -p $IMAGEDIR &> /dev/null
mkdir -p $RAMDISKDIR &> /dev/null
rm -rf $RAMDISKDIR/*

#only proceed if squashfs already mounted..
if [ "`cat /proc/mounts | grep squashfs | grep \/usr\/portage\ `" ]
then umount $RAMDISKDIR &> /dev/null
mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR
echo "** tmpfs of $RAMSIZE mounted on: $RAMDISKDIR"
mount -t unionfs -o dirs=$RAMDISKDIR=rw:/usr/portage=ro none /usr/portage
echo "** unionfs overlay mounted over /usr/portage"
emerge --sync
update-eix
mv -f /images/usr.portage.sqfs /images/usr.portage1.sqfs
mksquashfs /usr/portage /images/usr.portage.sqfs
umount -dl /usr/portage &> /dev/null
umount -dl /usr/portage &> /dev/null
umount -dl $RAMDISKDIR &> /dev/null
mount -t squashfs -o ro,loop /images/usr.portage.sqfs /usr/portage
fi


While trying your script I got those errors:

Quote:

mkstemp "/usr/portage/sys-apps/baselayout/.Manifest.l5EAf8" failed: Read-only file system 1317 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r1.ebuild.WlejER" failed: Read-only file system 18930 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r2.ebuild.bvd3yB" failed: Read-only file system 19000 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.14.ebuild.SuXJZl" failed: Read-only file system 18962 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.12.0_pre13-r1.ebuild.ZvcoW6" failed: Read-only file system


Are you sure that we need to mount the fs as read only?
_________________
A happy GNU/Linux user!!
Back to top
View user's profile Send private message
chojin
n00b
n00b


Joined: 26 Jun 2005
Posts: 42

PostPosted: Sat Jan 07, 2006 2:17 pm    Post subject: Reply with quote

al this looked very appealing to me so I tried it right away..
It all seemed to work alright until I tried the above myemerge-sync.sh script.
The script seems to mount the tmpfs and the unionfs
and while the scripts is running I see this in df -h:
Code:
 # df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/sil_afabafddahff5
                       19G   11G  6,7G  62% /
udev                  252M  292K  252M   1% /dev
/dev/mapper/sil_afabafddahff6
                      9,2G  8,3G  443M  96% /home
/dev/mapper/sil_afabafddahff8
                      184G  175G     0 100% /mnt/backup
/dev/mapper/sil_afabafddahff9
                      143G  131G  5,3G  97% /mnt/storage
/dev/mapper/sil_afabafddahff2
                       19G   11G  8,6G  55% /mnt/windows
none                  252M     0  252M   0% /dev/shm
none                  252M     0  252M   0% /dev/shm
/images/usr.portage.sqfs
                      127M   28M   99M  22% /usr/portage
filekeeper:/mnt/storage
                      113G  106G  1,9G  99% /mnt/filekeeper
tmpfs                 100M  1,2M   99M   2% /images/ramdisk
none                  127M   28M   99M  22% /usr/portage


and rsync starts, without problems.. doing some files and suddenly
Code:
.....
dev-libs/libdaemon/libdaemon-0.8.ebuild
         980 100%    0.00kB/s    0:00:00
dev-libs/mpfr/ChangeLog
        1231 100%    0.00kB/s    0:00:00
dev-libs/mpfr/Manifest
         980 100%    0.00kB/s    0:00:00
dev-libs/mpfr/files/2.2.0/patch06
         835 100%    0.00kB/s    0:00:00
dev-libs/mpfr/files/2.2.0/patch07
        1651 100%    0.00kB/s    0:00:00
dev-libs/mpfr/files/digest-mpfr-2.2.0_p7
         120 100%    0.00kB/s    0:00:00
metadata/cache/media-tv/
metadata/cache/media-video/
metadata/cache/net-analyzer/
metadata/cache/net-dialup/
metadata/cache/net-dns/
metadata/cache/net-firewall/
metadata/cache/net-fs/
metadata/cache/net-ftp/
metadata/cache/net-im/
metadata/cache/net-irc/
metadata/cache/net-libs/
metadata/cache/net-mail/
metadata/cache/net-misc/
rsync: connection unexpectedly closed (3238834 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...


>>> Starting retry 1 of 3 with rsync://81.223.20.162/gentoo-portage
>>> checking server timestamp ...

This is mirror.inode.at, your local friendly mirror in the neighbourhood.

Connection sponsored by inode. See http://www.inode.at/

Have a look at http://mirror.inode.at/ for further information and statistics on the server.

Please contact mirror@inode.at in case of problems.

*** Checksumming (-c) is disabled, so don't use it ***

Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162

receiving file list ...
1 file to consider
timestamp.chk
          32 100%    0.00kB/s    0:00:00

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 223
Total bytes read: 542

wrote 223 bytes  read 542 bytes  510.00 bytes/sec
total size is 32  speedup is 0.04

This is mirror.inode.at, your local friendly mirror in the neighbourhood.

Connection sponsored by inode. See http://www.inode.at/

Have a look at http://mirror.inode.at/ for further information and statistics on the server.

Please contact mirror@inode.at in case of problems.

*** Checksumming (-c) is disabled, so don't use it ***

Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162

receiving file list ...
133128 files to consider
./
app-admin/
app-admin/conky/
app-admin/conky/files/
app-admin/denyhosts/
app-admin/gentoo-bugger/
app-admin/gentoo-bugger/files/
app-admin/modlogan/
app-admin/modlogan/files/
app-admin/superadduser/
app-admin/superadduser/files/
app-admin/usermin/
app-admin/usermin/files/
app-admin/webalizer/
app-admin/webalizer/files/
app-admin/webalizer/files/2.01.10/
app-admin/zope-config/
app-admin/zope-config/files/
app-arch/
app-arch/bsdsfv/
app-arch/bsdsfv/files/
app-arch/sharutils/
app-arch/sharutils/files/
app-backup/
app-backup/sarab/
app-backup/sarab/files/
app-benchmarks/
app-cdr/
app-cdr/k9copy/
app-cdr/k9copy/files/
app-crypt/
app-crypt/heimdal/
app-crypt/heimdal/files/
app-crypt/johntheripper/
app-crypt/johntheripper/files/
app-dicts/
app-dicts/aspell-et/
app-dicts/aspell-et/files/
app-doc/
app-doc/doxygen/
app-doc/edox-data/
app-doc/xmltoman/
>>> retry ...


>>> Starting retry 2 of 3 with rsync://62.197.40.130/gentoo-portage
>>> checking server timestamp ...
,----------------
|  Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!

|  For info on Gentoo Linux, see <http://www.gentoo.org/>.

|  For more on this server, see <http://vegetable.org.uk/>.
|  For server-admin policy, see <http://gentoo.vegetable.org.uk/>.


   Please note: an excessive rate of connections to this server does not
   benefit anyone - a couple of times per day is normally sufficient. A
   small amount of profiling is undertaken to determine potential abusers
   and rate-limiting will be enforced if needed.


receiving file list ...
1 file to consider
timestamp.chk
          32 100%    0.00kB/s    0:00:00

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 223
Total bytes read: 699

wrote 223 bytes  read 699 bytes  1844.00 bytes/sec
total size is 32  speedup is 0.03
,----------------
|  Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!

|  For info on Gentoo Linux, see <http://www.gentoo.org/>.

|  For more on this server, see <http://vegetable.org.uk/>.
|  For server-admin policy, see <http://gentoo.vegetable.org.uk/>.


   Please note: an excessive rate of connections to this server does not
   benefit anyone - a couple of times per day is normally sufficient. A
   small amount of profiling is undertaken to determine potential abusers
   and rate-limiting will be enforced if needed.

receiving file list ...
133128 files to consider
./
app-admin/
app-admin/modlogan/files/
app-admin/superadduser/files/
app-admin/usermin/files/
app-admin/zope-config/files/
app-arch/
app-arch/bsdsfv/files/
app-benchmarks/
app-cdr/
app-crypt/
app-crypt/heimdal/files/
app-dicts/
app-doc/
io timeout after 358 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)
io timeout after 180 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)


And from that point I see this with ps -A |grep rsync:
Code:
$ ps -A |grep rsync
 8698 pts/0    00:00:02 rsync
 8765 pts/0    00:00:00 rsync <defunct>


THen when shutting down the pc, it can't unmount the loopback filesystems, complains a lot, even lets me use the emergency maintenance console (or press Ctrl-D to continue)...probably because rsync is keeping the filesystem busy and doesn't want to get killed.

Does anybody know why this is happening?
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Sat Jan 07, 2006 3:06 pm    Post subject: Reply with quote

For some reason, the email notification hadn't been telling me about these recent posts, so I missed this discussion.


Thanks for the help with unionfs. I'll try it out again shortly. If it works, I'll toss your script back up in the main post.
Back to top
View user's profile Send private message
emj
n00b
n00b


Joined: 01 Jan 2006
Posts: 5

PostPosted: Sun Jan 08, 2006 8:21 am    Post subject: Reply with quote

chojin:

I can only presume that perhaps you don't have enough free, unswapped, ram available. Either that, or your unionfs is bugged somehow. did "dmesg" give any errors? What kernel are you using? I'm using 2.6.14-archck5. Haven't tried it with any other. I'd be surprised if unionfs is broken on many kernels... I suspect it's just the same code copy in the various new kernels...

It would be useful to check and see the output of df -h during the bugged part of the emerge --sync .. that could let us know how full the ramdisk is. Oh, and a free -m also, so we can see the ram situation.

I'm operating with a gig of ram, just for reference... but at the end of a normal emerge --sync the ramdisk for the unionfs overlay is only using 11mb, soo...

bld:

The fact that it's complaining that the file system is read-only is probably an indication that you don't have unionfs support enabled / working. Try checking to make sure it's enabled in the kernel, and/or modprobing it if you selected it as a module.

I suppose I could add something to the script that does "grep /usr/src/linux/.config -ie UNIONFS | grep "yes"" or something...

Anyway, thanks for testing! If it's working for some, post and say so :P I'm continuing to use the script and it's working great.

~Ev
Back to top
View user's profile Send private message
chojin
n00b
n00b


Joined: 26 Jun 2005
Posts: 42

PostPosted: Sun Jan 08, 2006 1:28 pm    Post subject: Reply with quote

I have a total 512mb Ram and 2Gb of Swap so it would surprise me if I ran out of ram/swap with this script.

As you can see in the output of df -h in my previous post only 1.2M was taken on the ramdisk at the moment rsync started to act funny.

I'm running kernel 2.6.14-gentoo-r5-mh2 #9 PREEMPT but I didn't find anything about unionfs in the kernel config itself. I had to emerge sys-fs/unionfs-1.0.14 (which is masked ~x86) and this gave me the unionfs module, which I load in /etc/modules.autoload/kernel-2.6 together with the squashfs and loop module.

this is the output of free -m while rsync was crashing
Code:
$ free -m
             total       used       free     shared    buffers     cached
Mem:           503        495          7          0        153        103
-/+ buffers/cache:        239        264
Swap:         1906          0       1906


and df -h at the same moment
Code:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/sil_afabafddahff5
                       19G   12G  6,0G  66% /
udev                  252M  292K  252M   1% /dev
/dev/mapper/sil_afabafddahff6
                      9,2G  8,5G  277M  97% /home
/dev/mapper/sil_afabafddahff8
                      184G  175G     0 100% /mnt/backup
/dev/mapper/sil_afabafddahff9
                      143G  131G  5,3G  97% /mnt/storage
/dev/mapper/sil_afabafddahff2
                       19G   11G  8,6G  55% /mnt/windows
none                  252M     0  252M   0% /dev/shm
/images/usr.portage.sqfs
                      127M   29M   99M  23% /usr/portage
filekeeper:/mnt/storage
                      113G  106G  1,9G  99% /mnt/filekeeper
tmpfs                 100M  1,9M   99M   2% /images/ramdisk
none                  127M   29M   99M  23% /usr/portage


As you can see, I have plenty of free swap space and only 1.9M used on the ramdisk...
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Sun Jan 08, 2006 2:24 pm    Post subject: Reply with quote

I had a kernel panic yet again, about half-way through the sync. My r/w image was just a directory on the HD, so no risk of running out of space. I'm using vanilla 2.6.14.2. I'll look into it later this week, but I'll be busy for a few days.
Back to top
View user's profile Send private message
emj
n00b
n00b


Joined: 01 Jan 2006
Posts: 5

PostPosted: Sun Jan 08, 2006 6:16 pm    Post subject: Reply with quote

chojin:

Oh, ok, sorry. I thought that your previous df -h was at the point where the sync went buggy.

Hmmm, weird. I'll switch to a few different kernels and see how it works then. Oh, and I'll try the unionfs in portage. Maybe the problem's there. There's gotta be some buggy unionfs code somewhere. I'll post later tonight with results.

adsmith:

The kernel panic may be 'cause your r/w image was a directory on the HD... it should work, of course, but who knows? I'll give that a try too, if I have time.

~Ev
Back to top
View user's profile Send private message
emj
n00b
n00b


Joined: 01 Jan 2006
Posts: 5

PostPosted: Tue Jan 10, 2006 1:25 am    Post subject: Reply with quote

Well, I tried to set up the overlay and whatnot on my epia router, but to get squashfs, I ended up going with the ArchCK kernel again (that's what I'm using on my desktop as well), because mm-sources and regular ck-sources don't have that patch included, and I was too lazy to patch a kernel myself. I added squashfs and unionfs and proceeded.

Anyway, my router worked fine as well.

I just did:
emerge squashfs-tools
mkdir /images
mksquashfs /usr/portage /images/usr.portage.sqfs
mv /usr/portage /usr/portage_
mkdir /usr/portage
echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
mount /usr/portage

then: myemerge_sync.sh

And I didn't have any errors. I'll edit my original post to include those preparatory steps.

~Ev
Back to top
View user's profile Send private message
lazy_bum
l33t
l33t


Joined: 16 Feb 2005
Posts: 690

PostPosted: Mon Feb 06, 2006 9:51 am    Post subject: Reply with quote

Liviu wrote:
Okay Folks !
A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):
http://home.arcor.de/liviufl/portage.sqsh/
I got 1 Gigabyte of Traffic for that webspace. It should be enough for around 33 Downloads ;)
have fun !

It would be nice to tell when the snapshot from "daily" is turning to old, dead and buried... synced with this and i have ~20 packets to "downgrade"...
_________________
roslin uberlay | grubelek
Back to top
View user's profile Send private message
Boris27
Guru
Guru


Joined: 05 Nov 2003
Posts: 562
Location: Almelo, The Netherlands

PostPosted: Mon Feb 06, 2006 9:59 am    Post subject: Re: TIP: Compressing portage tree using squashfs Reply with quote

frostschutz wrote:
adsmith wrote:
Why?
Wow, the portage tree is huge! It takes around 500MB, which is a huge chunk of the storage space on many systems.


500MB is huge :?:

I'm somewhat unsatisfied by this description. Hard disks are so huge nowadays (they broke like the 100GB barrier ages ago) that it seems hardly imaginable that 500MB would pose that much of a problem. The only application which would be more limited in terms of disk space that I can think of, would be a virtual server. And for a Gentoo installation/update on those I would go about in a completely different manner.

So can you give a real world example where SquashFSing the Portage tree would actually be useful? It has so many downsides (hard to update, requires a lot of free space anyway during update), plus the Portage tree isn't even the biggest part about Gentoo (my distfiles usually grows bigger than that, as well as the work directory during a compile).

If you need the free space only while you are not updating your system, and you are only updating like once a week, and you don't mind to download a tarballed tree then, you could just delete the whole thing (and thus achieve a even better compression ration than SquashFS without all the hassle). :lol:


My server only has a 10 GB disk, so every byte is welcome. I even looked into changing the blocksize on my EXT3 and having it create less inodes, which take 128 bytes apiece.
_________________
we are microsoft, lower your firewalls and surrender your pc's. we will add your biological and technological distinctiveness to our own. your culture will adapt and service us. resistance is futile.
Back to top
View user's profile Send private message
attrezzo
n00b
n00b


Joined: 24 Feb 2004
Posts: 31

PostPosted: Mon Feb 13, 2006 2:48 pm    Post subject: MY laptop has a weak little 3 gig hard drive` Reply with quote

MY laptop has a little 3 gig hard drive and I'm getting around the problem by nfs mounting those partitions when I need to update or do a compile. /usr/portage & /usr/src
Just throw an extra partition on a bigger box with more hd space and let nfs do all the work when you have time at your home network.

Often I can't do any installs when I'm on the road anyway cause the proc is so stinkin slow it simply takes too long. I usually end up trying to find bin versions and grabbing those.
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Sat Mar 18, 2006 1:39 pm    Post subject: Reply with quote

I hope my guide about different filesystems for portage does fit here quite well.

Discussion on gentoo forums can be found here

Thanks for any visit.

Phlogiston
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Sat Mar 18, 2006 2:44 pm    Post subject: Reply with quote

But you only work with conventional filesystems, not any compressed filesystems like squashfs.

You should do this benchmark with squashfs. I've found it to be extremely fast... empty metadata update in exactly 2 minutes with a 1GHz laptop with a slow hard drive. The "du -h" test, which is silly in this case, takes <2 seconds.


Also, to what extent does the "emerge -S" test depend on the filesystem for /var/cache/edb, eh?
In any case, that test gave 1min29sec on this machine.
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Thu Mar 23, 2006 2:11 pm    Post subject: Reply with quote

I have a little bit different partitioning scheme and using unionfs to create the squashfs on the fly with emerge --sync.
I have distfiles and the unionfs-rw-overlay on a standalone partiotions, like this in fstab:
Code:
/dev/hda5               /var/snapshot           ext2            noatime                 0 0
/var/portage.sqsh       /var/portage    squashfs        ro,loop                 0 0
/usr/portage           /usr/portage    unionfs dirs=/var/snapshot=rw:/var/portage=ro,delete=whiteout   0 0
/dev/hda6               /usr/portage/distfiles          ext3            noatime                 0 0


So no need to change the distdir in make.conf, you simply need more partitions - but this is good as the fragmentation is smaller.

modules needed as follows
Code:
loop
squashfs
unionfs


/etc/conf.d/squash_portage

Code:
#!/bin/bash

PORTAGE_METADIR="/var"
PORTAGE="/usr/portage"

#don't change these values, unless you know what are you doing
PORTAGE_DISTDIR="${PORTAGE}/distfiles"
PORTAGE_SQUASH="${PORTAGE_METADIR}/portage"
PORTAGE_UNIONFS="${PORTAGE_METADIR}/snapshot"
SQUASH_FILE="${PORTAGE_METADIR}/portage.sqsh"
SQUASH_TEMPFILE="${PORTAGE_METADIR}/portage_snapshot.sqsh"


/etc/conf.d/local.stop - we need this to properly umount the partitions on shut-down

Code:
# /etc/conf.d/local.stop

# This is a good place to unload any misc.
# programs you started above.
# For example, if you are using OSS and have
# "/usr/local/bin/soundon" above, put
# "/usr/local/bin/soundoff" here.

if [[ -e /etc/conf.d/squash_portage ]] ; then
    source /etc/conf.d/squash_portage
    umount "${PORTAGE_DISTDIR}"
    umount "${PORTAGE}"
    umount "${PORTAGE_SQUASH}"
else
    eend $? "failed to import squash_portage"
fi


and here is the script, which I use to update the portage:

Code:
#!/bin/bash

#tests if the config exists
if [[ -e /etc/conf.d/squash_portage ]] ; then
    source /etc/conf.d/squash_portage
else
    echo "failed sourcing /etc/conf.d/squash_portage"
    exit 127
fi

#tests if the dirs exist
if ! [[ -d "${PORTAGE_METADIR}" && -d "${PORTAGE_DISTDIR}" && -d "${PORTAGE}" && \
    -d "${PORTAGE_SQUASH}" && -d "${PORTAGE_UNIONFS}" && -n ${SQUASH_TEMPFILE} ]] ; then
    echo "!!! the directories do not exist check your fstab"
    echo "!!! or some of the following variables are not defined"
    echo "!!! PORTAGE_DISTDIR PORTAGE PORTAGE_SQUASH PORTAGE_UNIONFS"
    echo "!!! PORTAGE_METADIR SQUASH_TEMPFILE"
    exit 127
fi

#test if the file is in place
if ! [[ -e "${SQUASH_FILE}" ]] ; then
    echo "!!! the SQUASH_FILE is not defined or the file does not exist, aborting"
    exit 127
fi

#remove stale temp, if exists
if [[ -e "${SQUASH_TEMPFILE}" ]] ; then
    echo "removing stale ${SQUASH_TEMPFILE}"
    rm -f "${SQUASH_TEMPFILE}"
fi

echo "emerge syncing"
torify /usr/bin/emerge --sync
echo "emerge --sync ready..."
echo "building the squashfs image"
umount "${PORTAGE_DISTDIR}"
/usr/bin/mksquashfs "${PORTAGE}" "${SQUASH_TEMPFILE}" -check_data
umount "${PORTAGE}"
umount "${PORTAGE_SQUASH}"
rm -fvR "${PORTAGE_UNIONFS}"/*
rm -f "${SQUASH_FILE}"
mv "${SQUASH_TEMPFILE}" "${SQUASH_FILE}"
mount "${PORTAGE_SQUASH}"
mount "${PORTAGE}"
mount "${PORTAGE_DISTDIR}"
echo "ready..."


The first squashfs image must be created with the normal method.
Strangely enough the squashfs creation is a lot faster in this way probably, because it could read faster :)
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Thu Mar 23, 2006 2:54 pm    Post subject: Reply with quote

Which kernel are you using?
I always failed to get the unionfs patch working on vanilla kernels.
I'm curious if this has been fixed in the move from 2.6.14 to 2.6.15
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Thu Mar 23, 2006 5:31 pm    Post subject: Reply with quote

I'm using the standalone unionfs from the portage (unionfs-1.1.3), it's not patched in the kernel.
I don't know which kernels have this, (gentoo is not patched), but I have seen it somewhere earlier, probably in nitro.
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum