View previous topic :: View next topic |
Author |
Message |
lazy_bum l33t
Joined: 16 Feb 2005 Posts: 691
|
Posted: Thu Dec 22, 2005 8:04 am Post subject: |
|
|
frank_einstien wrote: | How do I get use squashfs without recompiling my kernal? Just asking... |
Why not? You can add a module and then...
Code: | make modules && make modules_install |
PS. squashed my portage and it works fine. With my 300 MHz on a home router i haven't noticed big slowdown (well "emerge sync" takes about... half an hour), and ~100 mb is nice cmp to >500 mb. Nice tip, thx. _________________ roslin uberlay | grubelek |
|
Back to top |
|
|
electrofreak l33t
Joined: 30 Jun 2004 Posts: 713 Location: Ohio, USA
|
Posted: Thu Dec 22, 2005 6:26 pm Post subject: |
|
|
lazy_bum wrote: | frank_einstien wrote: | How do I get use squashfs without recompiling my kernal? Just asking... |
Why not? You can add a module and then...
Code: | make modules && make modules_install |
PS. squashed my portage and it works fine. With my 300 MHz on a home router i haven't noticed big slowdown (well "emerge sync" takes about... half an hour), and ~100 mb is nice cmp to >500 mb. Nice tip, thx. |
I do stuff like that all the time with modules. Works fine.
As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method?? _________________ Desktop: ABit AN8, Athlon64 X2 4400+ 939 2.75GHz, 2x1GB Corsair XMS DDR400, 2x160GB SATA RAID-0, 2x20"W, Vista Ultimate x64
Laptop: 15.4" MacBook Pro 2.4Ghz, 2x1GB RAM, 160GB, Mac OS X 10.5.1
Server: PIII 550Mhz, 3x128MB RAM, 160GB, Ubuntu Server 7.10 |
|
Back to top |
|
|
Liviu n00b
Joined: 01 Mar 2005 Posts: 24 Location: Munich, Germany
|
Posted: Fri Dec 23, 2005 4:04 am Post subject: |
|
|
Alright! Really nice guide!
I got only around 6 gig for my linux partition on my Laptop (60G Harddrive).
I use my Home-Made P3-533 Gentoo Router to make my Squashed Portage Snapshots! It really doesn`t take too long.. perhaps 15 mins ?!?
Works perfectly for me so i don`t need to make one from a tar-snapshot or sync it and then squash it.
In my opinion this whole idea is very nice! I will distribute a copy of the .sqsh too .. I`ll just configure my cron...........*workinprogress* |
|
Back to top |
|
|
lazy_bum l33t
Joined: 16 Feb 2005 Posts: 691
|
Posted: Fri Dec 23, 2005 10:12 am Post subject: |
|
|
electrofreak wrote: |
As for the squashed protage... did you make it so it can be written to? or did you use the guide's original method?? |
Used the guide method. Router is online all the time so the emerge-sync-squashed is a cron job in the middle of the night. _________________ roslin uberlay | grubelek |
|
Back to top |
|
|
bld l33t
Joined: 26 Mar 2003 Posts: 759 Location: Outter Space
|
Posted: Fri Dec 23, 2005 4:58 pm Post subject: |
|
|
frank_einstien wrote: | Hi guys
How do I get use squashfs without recompiling my kernal? Just asking...
Thanks |
NOTE: The squashFS patcheset is available for 2.6.13 and below. Hence I managed to rename the kernel version inside the patch. You should do this if your kernel is above 2.6.13 .. if something is not clear just ask.
# cd /root
# uname -r
2.6.14.2
# wget http://switch.dl.sourceforge.net/sourceforge/squashfs/squashfs2.2-r2.tar.gz
# tar zxvf squashfs2.2-r2.tar.gz
# cd squashfs2.2-r2/linux-2.6.13
# sed 's/2.6.13/2.6.14.2/g' squashfs2.2-patch > squashfs2.2-ready-patch
# cp squashfs2.2-ready-patch /usr/src/linux/ && cd /usr/src/linux && patch -p1 < squashfs2.2-ready-patch
done
choose, loop and squashFS as described below.. then type "make modules && make modules_install" inside the /usr/src/linux dir.. and you're all set..
Then follow the steps described above _________________ A happy GNU/Linux user!! |
|
Back to top |
|
|
Liviu n00b
Joined: 01 Mar 2005 Posts: 24 Location: Munich, Germany
|
Posted: Wed Dec 28, 2005 12:22 am Post subject: |
|
|
Okay Folks !
A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):
http://home.arcor.de/liviufl/portage.sqsh/
I got 1 Gigabyte of Traffic for that webspace. It should be enough for around 33 Downloads
have fun ! |
|
Back to top |
|
|
emj n00b
Joined: 01 Jan 2006 Posts: 5
|
Posted: Sun Jan 01, 2006 8:28 pm Post subject: |
|
|
Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all.
I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.
Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.
adsmith: I don't know why you got a kernel oops while testing unionfs. Perhaps it's because you didn't set the read/write layer to a tmpfs? Shouldn't matter, but... Or maybe it's just a bug in the code in your kernel... maybe try another kernel? unionfs has to be relatively stable, ordinarily, since most livecd are using it now as a fully filetree overlay, sooo... But there, it's a tmpfs overlay, again, so maybe it just likes that better. tmpfs is fine, really, since the changes accumulated during emerge syncs seem to only amount to a few megabytes of data in the ramdisk directory, which is cleared immediately after the sync, soo.
Anyway, after some testing, could you put this script into the main post? Thanks.
Code: |
#!/bin/bash
#
#myemerge_sync.sh - script to emerge --sync from a squashfs
#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location
#emerge --sync's, then re-squashes portage and remounts
#also runs an update-eix to update eix database
#
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.
#I use the ArchCK ebuilds (search the gentoo forums) to get the above in the kernel.
#Also, remember that it's very important to have DISTFILES set somewhere
#other than /usr/portage/distfiles
#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf
#
#Preparatory steps to running this the first time are as follows:
#
#emerge squashfs-tools
#mkdir /images
#mksquashfs /usr/portage /images/usr.portage.sqfs
#mv /usr/portage /usr/portage_
#mkdir /usr/portage
#echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
#mount /usr/portage
#
#...then try running this script. When you confirm that everything's working you
#can rm -rf /usr/portage_ to remove the old uncompressed portage tree, or restore
#it if the script doesn't work for you.
#
#~Ev
#
#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.
#100M should suffice unless you've failed to sync for a very, very long time.
#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.
#It will be unloaded when done sync'ing.
RAMSIZE="100M"
IMAGEDIR="/images"
RAMDISKDIR="$IMAGEDIR/ramdisk"
#######################
mkdir -p $IMAGEDIR &> /dev/null
mkdir -p $RAMDISKDIR &> /dev/null
rm -rf $RAMDISKDIR/*
#only proceed if squashfs already mounted..
if [ "`cat /proc/mounts | grep squashfs | grep \/usr\/portage\ `" ]
then umount $RAMDISKDIR &> /dev/null
mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR
echo "** tmpfs of $RAMSIZE mounted on: $RAMDISKDIR"
mount -t unionfs -o dirs=$RAMDISKDIR=rw:/usr/portage=ro none /usr/portage
echo "** unionfs overlay mounted over /usr/portage"
emerge --sync
update-eix
mv -f /images/usr.portage.sqfs /images/usr.portage1.sqfs
mksquashfs /usr/portage /images/usr.portage.sqfs
umount -dl /usr/portage &> /dev/null
umount -dl /usr/portage &> /dev/null
umount -dl $RAMDISKDIR &> /dev/null
mount -t squashfs -o ro,loop /images/usr.portage.sqfs /usr/portage
fi
|
Last edited by emj on Tue Jan 10, 2006 1:31 am; edited 1 time in total |
|
Back to top |
|
|
electrofreak l33t
Joined: 30 Jun 2004 Posts: 713 Location: Ohio, USA
|
Posted: Sun Jan 01, 2006 8:46 pm Post subject: |
|
|
looks like a nice script. I will try it soon (when I have time) _________________ Desktop: ABit AN8, Athlon64 X2 4400+ 939 2.75GHz, 2x1GB Corsair XMS DDR400, 2x160GB SATA RAID-0, 2x20"W, Vista Ultimate x64
Laptop: 15.4" MacBook Pro 2.4Ghz, 2x1GB RAM, 160GB, Mac OS X 10.5.1
Server: PIII 550Mhz, 3x128MB RAM, 160GB, Ubuntu Server 7.10 |
|
Back to top |
|
|
opentaka l33t
Joined: 18 Feb 2005 Posts: 840 Location: Japan
|
Posted: Mon Jan 02, 2006 7:30 am Post subject: |
|
|
great howto, this helped my router server with 5GB HDD. yes, 500MB is huuuge _________________ "Being defeated is often a temporary condition. Giving up is what makes it permanent" - Marilyn vos Savant
|
|
Back to top |
|
|
bld l33t
Joined: 26 Mar 2003 Posts: 759 Location: Outter Space
|
Posted: Sat Jan 07, 2006 10:43 am Post subject: |
|
|
emj wrote: | Here's a script to enable portage syncs using a unionfs overlay. Using this method, squashfs as the filesystem for the portage tree basically has absolutely no disadvantages at all.
I'm still testing this method myself, but so far it works great. Basically, the script that follows makes a tmpfs ramdisk to use as a read/write overlay, and uses unionfs as the glue to make the portage tree writable without unmounting and copying off all 500 megabytes of the portage tree to somewhere on the hard drive that's writable.
Before this, I wrote a script that made a ramdisk to copy the contents of the squashfs to, emerge --sync'd to the ramdisk, and then resquashed. The disadvantage was that it required 500mb of free ram. I have that, but most people don't. Using unionfs, the ram usage is next to nothing for the ramdisk. If you update portage often, we're talking a few megs of memory, temporarily, during a sync.
adsmith: I don't know why you got a kernel oops while testing unionfs. Perhaps it's because you didn't set the read/write layer to a tmpfs? Shouldn't matter, but... Or maybe it's just a bug in the code in your kernel... maybe try another kernel? unionfs has to be relatively stable, ordinarily, since most livecd are using it now as a fully filetree overlay, sooo... But there, it's a tmpfs overlay, again, so maybe it just likes that better. tmpfs is fine, really, since the changes accumulated during emerge syncs seem to only amount to a few megabytes of data in the ramdisk directory, which is cleared immediately after the sync, soo.
Anyway, after some testing, could you put this script into the main post? Thanks.
Code: |
#!/bin/bash
#
#myemerge_sync.sh - script to emerge --sync from a squashfs
#mounts a unionfs overlay onto /usr/portage, with a tmpfs ramdisk as the read/write location
#emerge --sync's, then re-squashes portage and remounts
#also runs an update-eix to update eix database
#
#REQUIREMENTS: tmpfs, unionfs, and squashfs in your kernel, or as modules.
#Also, remember that it's very important to have DISTFILES set somewhere
#other than /usr/portage/distfiles
#so set, for example, DISTDIR=/var/tmp/distfiles in /etc/make.conf
#RAMSIZE is the size of the tmpfs ramdisk that will be created for use as the unionfs's read/write area.
#100M should suffice unless you've failed to sync for a very, very long time.
#RAMDISKDIR can be changed if you already have a ramdisk you use, or left at default.
#It will be unloaded when done sync'ing.
RAMSIZE="100M"
IMAGEDIR="/images"
RAMDISKDIR="$IMAGEDIR/ramdisk"
#######################
mkdir -p $IMAGEDIR &> /dev/null
mkdir -p $RAMDISKDIR &> /dev/null
rm -rf $RAMDISKDIR/*
#only proceed if squashfs already mounted..
if [ "`cat /proc/mounts | grep squashfs | grep \/usr\/portage\ `" ]
then umount $RAMDISKDIR &> /dev/null
mount -t tmpfs tmpfs -o size=$RAMSIZE,nr_inodes=200000 $RAMDISKDIR
echo "** tmpfs of $RAMSIZE mounted on: $RAMDISKDIR"
mount -t unionfs -o dirs=$RAMDISKDIR=rw:/usr/portage=ro none /usr/portage
echo "** unionfs overlay mounted over /usr/portage"
emerge --sync
update-eix
mv -f /images/usr.portage.sqfs /images/usr.portage1.sqfs
mksquashfs /usr/portage /images/usr.portage.sqfs
umount -dl /usr/portage &> /dev/null
umount -dl /usr/portage &> /dev/null
umount -dl $RAMDISKDIR &> /dev/null
mount -t squashfs -o ro,loop /images/usr.portage.sqfs /usr/portage
fi
|
|
While trying your script I got those errors:
Quote: |
mkstemp "/usr/portage/sys-apps/baselayout/.Manifest.l5EAf8" failed: Read-only file system 1317 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r1.ebuild.WlejER" failed: Read-only file system 18930 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.13-r2.ebuild.bvd3yB" failed: Read-only file system 19000 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.11.14.ebuild.SuXJZl" failed: Read-only file system 18962 100% 0.00kB/s 0:00:00mkstemp "/usr/portage/sys-apps/baselayout/.baselayout-1.12.0_pre13-r1.ebuild.ZvcoW6" failed: Read-only file system |
Are you sure that we need to mount the fs as read only? _________________ A happy GNU/Linux user!! |
|
Back to top |
|
|
chojin n00b
Joined: 26 Jun 2005 Posts: 42
|
Posted: Sat Jan 07, 2006 2:17 pm Post subject: |
|
|
al this looked very appealing to me so I tried it right away..
It all seemed to work alright until I tried the above myemerge-sync.sh script.
The script seems to mount the tmpfs and the unionfs
and while the scripts is running I see this in df -h:
Code: | # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sil_afabafddahff5
19G 11G 6,7G 62% /
udev 252M 292K 252M 1% /dev
/dev/mapper/sil_afabafddahff6
9,2G 8,3G 443M 96% /home
/dev/mapper/sil_afabafddahff8
184G 175G 0 100% /mnt/backup
/dev/mapper/sil_afabafddahff9
143G 131G 5,3G 97% /mnt/storage
/dev/mapper/sil_afabafddahff2
19G 11G 8,6G 55% /mnt/windows
none 252M 0 252M 0% /dev/shm
none 252M 0 252M 0% /dev/shm
/images/usr.portage.sqfs
127M 28M 99M 22% /usr/portage
filekeeper:/mnt/storage
113G 106G 1,9G 99% /mnt/filekeeper
tmpfs 100M 1,2M 99M 2% /images/ramdisk
none 127M 28M 99M 22% /usr/portage
|
and rsync starts, without problems.. doing some files and suddenly
Code: | .....
dev-libs/libdaemon/libdaemon-0.8.ebuild
980 100% 0.00kB/s 0:00:00
dev-libs/mpfr/ChangeLog
1231 100% 0.00kB/s 0:00:00
dev-libs/mpfr/Manifest
980 100% 0.00kB/s 0:00:00
dev-libs/mpfr/files/2.2.0/patch06
835 100% 0.00kB/s 0:00:00
dev-libs/mpfr/files/2.2.0/patch07
1651 100% 0.00kB/s 0:00:00
dev-libs/mpfr/files/digest-mpfr-2.2.0_p7
120 100% 0.00kB/s 0:00:00
metadata/cache/media-tv/
metadata/cache/media-video/
metadata/cache/net-analyzer/
metadata/cache/net-dialup/
metadata/cache/net-dns/
metadata/cache/net-firewall/
metadata/cache/net-fs/
metadata/cache/net-ftp/
metadata/cache/net-im/
metadata/cache/net-irc/
metadata/cache/net-libs/
metadata/cache/net-mail/
metadata/cache/net-misc/
rsync: connection unexpectedly closed (3238834 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...
>>> Starting retry 1 of 3 with rsync://81.223.20.162/gentoo-portage
>>> checking server timestamp ...
This is mirror.inode.at, your local friendly mirror in the neighbourhood.
Connection sponsored by inode. See http://www.inode.at/
Have a look at http://mirror.inode.at/ for further information and statistics on the server.
Please contact mirror@inode.at in case of problems.
*** Checksumming (-c) is disabled, so don't use it ***
Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162
receiving file list ...
1 file to consider
timestamp.chk
32 100% 0.00kB/s 0:00:00
Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 223
Total bytes read: 542
wrote 223 bytes read 542 bytes 510.00 bytes/sec
total size is 32 speedup is 0.04
This is mirror.inode.at, your local friendly mirror in the neighbourhood.
Connection sponsored by inode. See http://www.inode.at/
Have a look at http://mirror.inode.at/ for further information and statistics on the server.
Please contact mirror@inode.at in case of problems.
*** Checksumming (-c) is disabled, so don't use it ***
Gentoo-Users: You've hit rsync1.at.gentoo.org/81.223.20.162
receiving file list ...
133128 files to consider
./
app-admin/
app-admin/conky/
app-admin/conky/files/
app-admin/denyhosts/
app-admin/gentoo-bugger/
app-admin/gentoo-bugger/files/
app-admin/modlogan/
app-admin/modlogan/files/
app-admin/superadduser/
app-admin/superadduser/files/
app-admin/usermin/
app-admin/usermin/files/
app-admin/webalizer/
app-admin/webalizer/files/
app-admin/webalizer/files/2.01.10/
app-admin/zope-config/
app-admin/zope-config/files/
app-arch/
app-arch/bsdsfv/
app-arch/bsdsfv/files/
app-arch/sharutils/
app-arch/sharutils/files/
app-backup/
app-backup/sarab/
app-backup/sarab/files/
app-benchmarks/
app-cdr/
app-cdr/k9copy/
app-cdr/k9copy/files/
app-crypt/
app-crypt/heimdal/
app-crypt/heimdal/files/
app-crypt/johntheripper/
app-crypt/johntheripper/files/
app-dicts/
app-dicts/aspell-et/
app-dicts/aspell-et/files/
app-doc/
app-doc/doxygen/
app-doc/edox-data/
app-doc/xmltoman/
>>> retry ...
>>> Starting retry 2 of 3 with rsync://62.197.40.130/gentoo-portage
>>> checking server timestamp ...
,----------------
| Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!
|
| For info on Gentoo Linux, see <http://www.gentoo.org/>.
|
| For more on this server, see <http://vegetable.org.uk/>.
| For server-admin policy, see <http://gentoo.vegetable.org.uk/>.
|
Please note: an excessive rate of connections to this server does not
benefit anyone - a couple of times per day is normally sufficient. A
small amount of profiling is undertaken to determine potential abusers
and rate-limiting will be enforced if needed.
receiving file list ...
1 file to consider
timestamp.chk
32 100% 0.00kB/s 0:00:00
Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 32
Total bytes written: 223
Total bytes read: 699
wrote 223 bytes read 699 bytes 1844.00 bytes/sec
total size is 32 speedup is 0.03
,----------------
| Welcome to rsync1.uk.gentoo.org, located in Telehouse, London, UK!
|
| For info on Gentoo Linux, see <http://www.gentoo.org/>.
|
| For more on this server, see <http://vegetable.org.uk/>.
| For server-admin policy, see <http://gentoo.vegetable.org.uk/>.
|
Please note: an excessive rate of connections to this server does not
benefit anyone - a couple of times per day is normally sufficient. A
small amount of profiling is undertaken to determine potential abusers
and rate-limiting will be enforced if needed.
receiving file list ...
133128 files to consider
./
app-admin/
app-admin/modlogan/files/
app-admin/superadduser/files/
app-admin/usermin/files/
app-admin/zope-config/files/
app-arch/
app-arch/bsdsfv/files/
app-benchmarks/
app-cdr/
app-crypt/
app-crypt/heimdal/files/
app-dicts/
app-doc/
io timeout after 358 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)
io timeout after 180 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)
|
And from that point I see this with ps -A |grep rsync:
Code: | $ ps -A |grep rsync
8698 pts/0 00:00:02 rsync
8765 pts/0 00:00:00 rsync <defunct> |
THen when shutting down the pc, it can't unmount the loopback filesystems, complains a lot, even lets me use the emergency maintenance console (or press Ctrl-D to continue)...probably because rsync is keeping the filesystem busy and doesn't want to get killed.
Does anybody know why this is happening? |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
Posted: Sat Jan 07, 2006 3:06 pm Post subject: |
|
|
For some reason, the email notification hadn't been telling me about these recent posts, so I missed this discussion.
Thanks for the help with unionfs. I'll try it out again shortly. If it works, I'll toss your script back up in the main post. |
|
Back to top |
|
|
emj n00b
Joined: 01 Jan 2006 Posts: 5
|
Posted: Sun Jan 08, 2006 8:21 am Post subject: |
|
|
chojin:
I can only presume that perhaps you don't have enough free, unswapped, ram available. Either that, or your unionfs is bugged somehow. did "dmesg" give any errors? What kernel are you using? I'm using 2.6.14-archck5. Haven't tried it with any other. I'd be surprised if unionfs is broken on many kernels... I suspect it's just the same code copy in the various new kernels...
It would be useful to check and see the output of df -h during the bugged part of the emerge --sync .. that could let us know how full the ramdisk is. Oh, and a free -m also, so we can see the ram situation.
I'm operating with a gig of ram, just for reference... but at the end of a normal emerge --sync the ramdisk for the unionfs overlay is only using 11mb, soo...
bld:
The fact that it's complaining that the file system is read-only is probably an indication that you don't have unionfs support enabled / working. Try checking to make sure it's enabled in the kernel, and/or modprobing it if you selected it as a module.
I suppose I could add something to the script that does "grep /usr/src/linux/.config -ie UNIONFS | grep "yes"" or something...
Anyway, thanks for testing! If it's working for some, post and say so I'm continuing to use the script and it's working great.
~Ev |
|
Back to top |
|
|
chojin n00b
Joined: 26 Jun 2005 Posts: 42
|
Posted: Sun Jan 08, 2006 1:28 pm Post subject: |
|
|
I have a total 512mb Ram and 2Gb of Swap so it would surprise me if I ran out of ram/swap with this script.
As you can see in the output of df -h in my previous post only 1.2M was taken on the ramdisk at the moment rsync started to act funny.
I'm running kernel 2.6.14-gentoo-r5-mh2 #9 PREEMPT but I didn't find anything about unionfs in the kernel config itself. I had to emerge sys-fs/unionfs-1.0.14 (which is masked ~x86) and this gave me the unionfs module, which I load in /etc/modules.autoload/kernel-2.6 together with the squashfs and loop module.
this is the output of free -m while rsync was crashing
Code: | $ free -m
total used free shared buffers cached
Mem: 503 495 7 0 153 103
-/+ buffers/cache: 239 264
Swap: 1906 0 1906
|
and df -h at the same moment
Code: | $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sil_afabafddahff5
19G 12G 6,0G 66% /
udev 252M 292K 252M 1% /dev
/dev/mapper/sil_afabafddahff6
9,2G 8,5G 277M 97% /home
/dev/mapper/sil_afabafddahff8
184G 175G 0 100% /mnt/backup
/dev/mapper/sil_afabafddahff9
143G 131G 5,3G 97% /mnt/storage
/dev/mapper/sil_afabafddahff2
19G 11G 8,6G 55% /mnt/windows
none 252M 0 252M 0% /dev/shm
/images/usr.portage.sqfs
127M 29M 99M 23% /usr/portage
filekeeper:/mnt/storage
113G 106G 1,9G 99% /mnt/filekeeper
tmpfs 100M 1,9M 99M 2% /images/ramdisk
none 127M 29M 99M 23% /usr/portage
|
As you can see, I have plenty of free swap space and only 1.9M used on the ramdisk... |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
Posted: Sun Jan 08, 2006 2:24 pm Post subject: |
|
|
I had a kernel panic yet again, about half-way through the sync. My r/w image was just a directory on the HD, so no risk of running out of space. I'm using vanilla 2.6.14.2. I'll look into it later this week, but I'll be busy for a few days. |
|
Back to top |
|
|
emj n00b
Joined: 01 Jan 2006 Posts: 5
|
Posted: Sun Jan 08, 2006 6:16 pm Post subject: |
|
|
chojin:
Oh, ok, sorry. I thought that your previous df -h was at the point where the sync went buggy.
Hmmm, weird. I'll switch to a few different kernels and see how it works then. Oh, and I'll try the unionfs in portage. Maybe the problem's there. There's gotta be some buggy unionfs code somewhere. I'll post later tonight with results.
adsmith:
The kernel panic may be 'cause your r/w image was a directory on the HD... it should work, of course, but who knows? I'll give that a try too, if I have time.
~Ev |
|
Back to top |
|
|
emj n00b
Joined: 01 Jan 2006 Posts: 5
|
Posted: Tue Jan 10, 2006 1:25 am Post subject: |
|
|
Well, I tried to set up the overlay and whatnot on my epia router, but to get squashfs, I ended up going with the ArchCK kernel again (that's what I'm using on my desktop as well), because mm-sources and regular ck-sources don't have that patch included, and I was too lazy to patch a kernel myself. I added squashfs and unionfs and proceeded.
Anyway, my router worked fine as well.
I just did:
emerge squashfs-tools
mkdir /images
mksquashfs /usr/portage /images/usr.portage.sqfs
mv /usr/portage /usr/portage_
mkdir /usr/portage
echo "/images/usr.portage.sqfs /usr/portage squashfs ro,loop 0 0" >> /etc/fstab
mount /usr/portage
then: myemerge_sync.sh
And I didn't have any errors. I'll edit my original post to include those preparatory steps.
~Ev |
|
Back to top |
|
|
lazy_bum l33t
Joined: 16 Feb 2005 Posts: 691
|
Posted: Mon Feb 06, 2006 9:51 am Post subject: |
|
|
Liviu wrote: | Okay Folks !
A daily updated, squashed portage snapshot is availible here too (my home Router syncs every day, creates the squashfs and uploads it to my webspace):
http://home.arcor.de/liviufl/portage.sqsh/
I got 1 Gigabyte of Traffic for that webspace. It should be enough for around 33 Downloads ;)
have fun ! |
It would be nice to tell when the snapshot from "daily" is turning to old, dead and buried... synced with this and i have ~20 packets to "downgrade"... _________________ roslin uberlay | grubelek |
|
Back to top |
|
|
Boris27 Guru
Joined: 05 Nov 2003 Posts: 562 Location: Almelo, The Netherlands
|
Posted: Mon Feb 06, 2006 9:59 am Post subject: Re: TIP: Compressing portage tree using squashfs |
|
|
frostschutz wrote: | adsmith wrote: | Why?
Wow, the portage tree is huge! It takes around 500MB, which is a huge chunk of the storage space on many systems. |
500MB is huge
I'm somewhat unsatisfied by this description. Hard disks are so huge nowadays (they broke like the 100GB barrier ages ago) that it seems hardly imaginable that 500MB would pose that much of a problem. The only application which would be more limited in terms of disk space that I can think of, would be a virtual server. And for a Gentoo installation/update on those I would go about in a completely different manner.
So can you give a real world example where SquashFSing the Portage tree would actually be useful? It has so many downsides (hard to update, requires a lot of free space anyway during update), plus the Portage tree isn't even the biggest part about Gentoo (my distfiles usually grows bigger than that, as well as the work directory during a compile).
If you need the free space only while you are not updating your system, and you are only updating like once a week, and you don't mind to download a tarballed tree then, you could just delete the whole thing (and thus achieve a even better compression ration than SquashFS without all the hassle). |
My server only has a 10 GB disk, so every byte is welcome. I even looked into changing the blocksize on my EXT3 and having it create less inodes, which take 128 bytes apiece. _________________ we are microsoft, lower your firewalls and surrender your pc's. we will add your biological and technological distinctiveness to our own. your culture will adapt and service us. resistance is futile. |
|
Back to top |
|
|
attrezzo n00b
Joined: 24 Feb 2004 Posts: 31
|
Posted: Mon Feb 13, 2006 2:48 pm Post subject: MY laptop has a weak little 3 gig hard drive` |
|
|
MY laptop has a little 3 gig hard drive and I'm getting around the problem by nfs mounting those partitions when I need to update or do a compile. /usr/portage & /usr/src
Just throw an extra partition on a bigger box with more hd space and let nfs do all the work when you have time at your home network.
Often I can't do any installs when I'm on the road anyway cause the proc is so stinkin slow it simply takes too long. I usually end up trying to find bin versions and grabbing those. |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Sat Mar 18, 2006 1:39 pm Post subject: |
|
|
I hope my guide about different filesystems for portage does fit here quite well.
Discussion on gentoo forums can be found here
Thanks for any visit.
Phlogiston _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
Posted: Sat Mar 18, 2006 2:44 pm Post subject: |
|
|
But you only work with conventional filesystems, not any compressed filesystems like squashfs.
You should do this benchmark with squashfs. I've found it to be extremely fast... empty metadata update in exactly 2 minutes with a 1GHz laptop with a slow hard drive. The "du -h" test, which is silly in this case, takes <2 seconds.
Also, to what extent does the "emerge -S" test depend on the filesystem for /var/cache/edb, eh?
In any case, that test gave 1min29sec on this machine. |
|
Back to top |
|
|
Gergan Penkov Veteran
Joined: 17 Jul 2004 Posts: 1464 Location: das kleinste Kuhdorf Deutschlands :)
|
Posted: Thu Mar 23, 2006 2:11 pm Post subject: |
|
|
I have a little bit different partitioning scheme and using unionfs to create the squashfs on the fly with emerge --sync.
I have distfiles and the unionfs-rw-overlay on a standalone partiotions, like this in fstab:
Code: | /dev/hda5 /var/snapshot ext2 noatime 0 0
/var/portage.sqsh /var/portage squashfs ro,loop 0 0
/usr/portage /usr/portage unionfs dirs=/var/snapshot=rw:/var/portage=ro,delete=whiteout 0 0
/dev/hda6 /usr/portage/distfiles ext3 noatime 0 0
|
So no need to change the distdir in make.conf, you simply need more partitions - but this is good as the fragmentation is smaller.
modules needed as follows
Code: | loop
squashfs
unionfs
|
/etc/conf.d/squash_portage
Code: | #!/bin/bash
PORTAGE_METADIR="/var"
PORTAGE="/usr/portage"
#don't change these values, unless you know what are you doing
PORTAGE_DISTDIR="${PORTAGE}/distfiles"
PORTAGE_SQUASH="${PORTAGE_METADIR}/portage"
PORTAGE_UNIONFS="${PORTAGE_METADIR}/snapshot"
SQUASH_FILE="${PORTAGE_METADIR}/portage.sqsh"
SQUASH_TEMPFILE="${PORTAGE_METADIR}/portage_snapshot.sqsh"
|
/etc/conf.d/local.stop - we need this to properly umount the partitions on shut-down
Code: | # /etc/conf.d/local.stop
# This is a good place to unload any misc.
# programs you started above.
# For example, if you are using OSS and have
# "/usr/local/bin/soundon" above, put
# "/usr/local/bin/soundoff" here.
if [[ -e /etc/conf.d/squash_portage ]] ; then
source /etc/conf.d/squash_portage
umount "${PORTAGE_DISTDIR}"
umount "${PORTAGE}"
umount "${PORTAGE_SQUASH}"
else
eend $? "failed to import squash_portage"
fi
|
and here is the script, which I use to update the portage:
Code: | #!/bin/bash
#tests if the config exists
if [[ -e /etc/conf.d/squash_portage ]] ; then
source /etc/conf.d/squash_portage
else
echo "failed sourcing /etc/conf.d/squash_portage"
exit 127
fi
#tests if the dirs exist
if ! [[ -d "${PORTAGE_METADIR}" && -d "${PORTAGE_DISTDIR}" && -d "${PORTAGE}" && \
-d "${PORTAGE_SQUASH}" && -d "${PORTAGE_UNIONFS}" && -n ${SQUASH_TEMPFILE} ]] ; then
echo "!!! the directories do not exist check your fstab"
echo "!!! or some of the following variables are not defined"
echo "!!! PORTAGE_DISTDIR PORTAGE PORTAGE_SQUASH PORTAGE_UNIONFS"
echo "!!! PORTAGE_METADIR SQUASH_TEMPFILE"
exit 127
fi
#test if the file is in place
if ! [[ -e "${SQUASH_FILE}" ]] ; then
echo "!!! the SQUASH_FILE is not defined or the file does not exist, aborting"
exit 127
fi
#remove stale temp, if exists
if [[ -e "${SQUASH_TEMPFILE}" ]] ; then
echo "removing stale ${SQUASH_TEMPFILE}"
rm -f "${SQUASH_TEMPFILE}"
fi
echo "emerge syncing"
torify /usr/bin/emerge --sync
echo "emerge --sync ready..."
echo "building the squashfs image"
umount "${PORTAGE_DISTDIR}"
/usr/bin/mksquashfs "${PORTAGE}" "${SQUASH_TEMPFILE}" -check_data
umount "${PORTAGE}"
umount "${PORTAGE_SQUASH}"
rm -fvR "${PORTAGE_UNIONFS}"/*
rm -f "${SQUASH_FILE}"
mv "${SQUASH_TEMPFILE}" "${SQUASH_FILE}"
mount "${PORTAGE_SQUASH}"
mount "${PORTAGE}"
mount "${PORTAGE_DISTDIR}"
echo "ready..."
|
The first squashfs image must be created with the normal method.
Strangely enough the squashfs creation is a lot faster in this way probably, because it could read faster _________________ "I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
Posted: Thu Mar 23, 2006 2:54 pm Post subject: |
|
|
Which kernel are you using?
I always failed to get the unionfs patch working on vanilla kernels.
I'm curious if this has been fixed in the move from 2.6.14 to 2.6.15 |
|
Back to top |
|
|
Gergan Penkov Veteran
Joined: 17 Jul 2004 Posts: 1464 Location: das kleinste Kuhdorf Deutschlands :)
|
Posted: Thu Mar 23, 2006 5:31 pm Post subject: |
|
|
I'm using the standalone unionfs from the portage (unionfs-1.1.3), it's not patched in the kernel.
I don't know which kernels have this, (gentoo is not patched), but I have seen it somewhere earlier, probably in nitro. _________________ "I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|