Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage tree using squashfs
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Da Fox
Guru
Guru


Joined: 06 Jul 2005
Posts: 341

PostPosted: Sat Jun 24, 2006 8:55 pm    Post subject: Reply with quote

You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.
Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope ;)) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.

In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it :D
_________________
"Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami

JGBE, a Java based GameBoy Emulator
Back to top
View user's profile Send private message
Gergan Penkov
Veteran
Veteran


Joined: 17 Jul 2004
Posts: 1464
Location: das kleinste Kuhdorf Deutschlands :)

PostPosted: Sat Jun 24, 2006 10:00 pm    Post subject: Reply with quote

And not to forget the fragmentation, and this on old hardware - the portage fragments extremely the hard disk with squashfs and probably with in memory mounted unionfs overlay there will be virtually no chances for fragmentation :) which is IMHO the best thing out of this and the speed of course :)
_________________
"I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack
Back to top
View user's profile Send private message
enderandrew
l33t
l33t


Joined: 25 Oct 2005
Posts: 731

PostPosted: Sun Jun 25, 2006 3:42 am    Post subject: Reply with quote

Da Fox wrote:
You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.
Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope ;)) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.

In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it :D


In the prlock thread, the example was going down to 18 megs without any compression.

It is faster, and it uses less bandwidth which makes it more friendly on the various Gentoo mirrors.

As far as work, prlock has an easier setup, and is also a one-time thing that ends up saving you time (no compress/expand/recompress, as well as faster syncs and searches). Having a smaller portage naturally is a win-win-win.

If you *REALLY* wanted, you could combine both methods and shrink portage down very small.
_________________
Nihilism makes me smile.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 850
Location: PL

PostPosted: Tue Jun 27, 2006 8:21 pm    Post subject: Reply with quote

this method is way better https://forums.gentoo.org/viewtopic-t-465367-highlight-squashfs.html

i use it on daily basis now.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
sfragis
Tux's lil' helper
Tux's lil' helper


Joined: 24 Mar 2005
Posts: 95
Location: RE < IT < Europe

PostPosted: Tue Aug 08, 2006 4:17 pm    Post subject: Reply with quote

Phlogiston wrote:
I hope my guide about different filesystems for portage does fit here quite well.

Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:

  • block size of 65536 bytes (default) => 38 MB
  • block size of 16384 bytes => 44 MB
  • block size of 4096 bytes (minimum accepted by mksquashfs) => 55 MB

Hence the best value for the block size is 64KB which seems in contraddiction with the result of Phlogiston's guide 8O (smaller block size leads to less filesystem usage). Has anyone tried to tune this or other parameters in order to achieve different results?
_________________
Regards
Fabio Strozzi
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Tue Aug 08, 2006 7:48 pm    Post subject: Reply with quote

sfragis wrote:
Phlogiston wrote:
I hope my guide about different filesystems for portage does fit here quite well.

Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:

  • block size of 65536 bytes (default) => 38 MB
  • block size of 16384 bytes => 44 MB
  • block size of 4096 bytes (minimum accepted by mksquashfs) => 55 MB

Hence the best value for the block size is 64KB which seems in contraddiction with the result of Phlogiston's guide 8O (smaller block size leads to less filesystem usage). Has anyone tried to tune this or other parameters in order to achieve different results?


Hmm probably this is different when using squashfs, but actually I can't explain.

But my question: Is there any howto that sums this stuff here up and gives clear instructions about using portage on a squashfs? I mean what are the results of that thread? Squasfs + unionfs ->rw and then just portage on it?

Thanks!
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Tue Aug 08, 2006 7:57 pm    Post subject: Reply with quote

Option 1: (easy, read-only)

download http://www.math.duke.edu/~adsmith/portage.sqsh periodically (it's updated weekly), and mount it to /usr/portage.


Option 2: (slightly more complicated, read-write)

use unionfs over squashfs, see here: https://forums.gentoo.org/viewtopic-p-3338393.html
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Tue Aug 08, 2006 8:03 pm    Post subject: Reply with quote

adsmith wrote:
Option 1: (easy, read-only)

download http://www.math.duke.edu/~adsmith/portage.sqsh periodically (it's updated weekly), and mount it to /usr/portage.


Option 2: (slightly more complicated, read-write)

use unionfs over squashfs, see here: https://forums.gentoo.org/viewtopic-p-3338393.html


Thanks! I'd like to sync it myself but I don't like the init scripts because I never shutdown actually.
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Tue Aug 08, 2006 8:15 pm    Post subject: Reply with quote

This is quite interesting:
Code:

Creating little endian 3.0 filesystem on /tmp/images/portage.sqfs, block size 65536.

Little endian filesystem, data block size 65536, compressed data, compressed metadata, compressed fragments
Filesystem size 38251.43 Kbytes (37.35 Mbytes)
        25.09% of uncompressed filesystem size (152465.07 Kbytes)
Inode table size 1652513 bytes (1613.78 Kbytes)
        35.97% of uncompressed inode table size (4594717 bytes)
Directory table size 1407370 bytes (1374.38 Kbytes)
        40.76% of uncompressed directory table size (3453208 bytes)
Number of duplicate files found 12688
Number of inodes 146450
Number of files 123037
Number of fragments 2275
Number of symbolic links  1
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 23412
Number of uids 2
        root (0)
        portage (250)
Number of gids 0


Especially the duplicate files 8)

I'm thinking more and more that the way of portage syncing and storing is quite deprecated. Using portage and syncing just the compressed images cuts down the traffic incredibly.

/edit: Wow this rocks:

Code:

time /usr/local/sbin/psync.2
[...]
All done, you may now enjoy your Updated Portage Tree :p <-this actually not  8)

real    2m19.984s
user    0m27.094s
sys     0m18.061s


Thanks a lot... Although I think that the first emerge -Duvat world afterwards is not really faster...
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
sfragis
Tux's lil' helper
Tux's lil' helper


Joined: 24 Mar 2005
Posts: 95
Location: RE < IT < Europe

PostPosted: Tue Aug 08, 2006 10:56 pm    Post subject: Reply with quote

I wrote a small script to wrap the portage tree sync and automagically mount/umount the unionfs overlay and keep the squashfs image up to date. Code uses parts of scripts published in this and other threads of the Gentoo forum. There's a small header in it with a brief explanation of what I did to get things working. When I'll have more spare time I'll write a detailed html page. Feel free to trash it, use it and edit it :wink:
_________________
Regards
Fabio Strozzi
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Wed Aug 09, 2006 5:27 am    Post subject: Reply with quote

Hmm I have a new problem now: After activating squashfs and unionfs in my kernel config, suspend2 does not work any longer. It hangs at doing atomic copy. Any hints?
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
sfragis
Tux's lil' helper
Tux's lil' helper


Joined: 24 Mar 2005
Posts: 95
Location: RE < IT < Europe

PostPosted: Wed Aug 09, 2006 7:12 am    Post subject: Reply with quote

Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted?
_________________
Regards
Fabio Strozzi
Back to top
View user's profile Send private message
micko
Tux's lil' helper
Tux's lil' helper


Joined: 16 Dec 2004
Posts: 121
Location: Finland / Turku

PostPosted: Wed Aug 09, 2006 9:18 am    Post subject: Reply with quote

Quote:
Phlogiston wrote:

Option 2: (slightly more complicated, read-write)

use unionfs over squashfs, see here: https://forums.gentoo.org/viewtopic-p-3338393.html


Thanks! I'd like to sync it myself but I don't like the init scripts because I never shutdown actually.


I wrote an extremely simple script that sync's portage and restarts the service, so the squashfs-image gets updated. If I forget to use the script and sync normally nothing will go wrong, unless my computer crashes, and even then everything is fixed by a new sync. I never shut down either and sync maybe about 4-5 times a week. Again, if I forget to use the script once but remember the next time, it will update the image then.

This unionfs-method is great also because it works well with layman.

That is for my fastest computer. For my server and laptop I've written a script that umounts portage dl's the portage.sqfs with scp from my desktop and mounts the new one. Everything works just great.
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Wed Aug 09, 2006 11:36 am    Post subject: Reply with quote

sfragis wrote:
Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted?


I compiled the stuff directly into the kernel. (I hate modules 8) ) But I will find out if it`s squashfs or unionfs, and then try with modules. PS: It didn't help to umount portage before. So even I don't use sqaushfs or unionfs it breaks hibernation. That is at least a bug in my opinion.

BTW using latest beyond3 sources.

/update: I just found out that its not squashfs nor unionfs related. It just has to do with my kernel, I don't know, using an older one now.
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
tost
Guru
Guru


Joined: 10 Dec 2005
Posts: 506

PostPosted: Sun Aug 13, 2006 11:02 am    Post subject: Reply with quote

Now I´m a bit confused

I really like the idea of using squashfs for Portage, but for me it seems that it doesn´t really work.
I activated all the stuff in the kernel and after this viewPortageX (that Superkaramba theme) shows me that some updates are available.
So I run emerge-sync-squash.sh. But at the end of it emerge -NuDpv world says that no updates are available.
And now the things get strange, because emerge -pv vcdimager wants to upgrade the package ?

Whats going wrong ?
Back to top
View user's profile Send private message
adsmith
Veteran
Veteran


Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA

PostPosted: Sun Aug 13, 2006 1:09 pm    Post subject: Reply with quote

I'm not sure I entirely understand your problem. If you jsut man that a world update is missing things, it may be totally unrelated to squashfs.

1) Is "getbinpkg" activated in FEATURES in your make.conf? That sometimes causes missed packages in world updates

2) are you sure the squashed portage tree is complete and mounted properly, and that PORTDIR is set properly?

3) Is your world file actually up-to-date?
Back to top
View user's profile Send private message
tost
Guru
Guru


Joined: 10 Dec 2005
Posts: 506

PostPosted: Sun Aug 13, 2006 2:05 pm    Post subject: Reply with quote

mmhh I thought the problem is a consequence of my compressed portage tree, but now I saw that it doesn´t work either when I use portage in "default-mode" (uncompressed)

Quote:
3) Is your world file actually up-to-date?


I´ll check it, thanks

tost
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Mon Sep 04, 2006 11:34 am    Post subject: Reply with quote

I'm still using DaFox script to update my portage and it works well (though, any updates?)

But today I noticed an ugly slowdown: emerge -ua world takes minutes to calcuate the updates. I don't know if its the portage version I am using( Installed: 2.1.1_rc1-r2) or the latest kernel update.

Anyone else noticed this?
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
Phlogiston
Veteran
Veteran


Joined: 27 Jan 2004
Posts: 1925
Location: Europe, Swizerland

PostPosted: Mon Feb 19, 2007 10:11 pm    Post subject: Reply with quote

Hey squashfs portage user whats up? I'm still using the psync script here and it works perfectly. Is anyone else still using it or is there any better way to do it? 8)
_________________
Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4
Page 4 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum