View previous topic :: View next topic |
Author |
Message |
Da Fox Guru
Joined: 06 Jul 2005 Posts: 341
|
Posted: Sat Jun 24, 2006 8:55 pm Post subject: |
|
|
You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.
Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope ) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.
In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it _________________ "Man fears the darkness, and so he scrapes away at the edges of it with fire."
- Rei Ayanami
JGBE, a Java based GameBoy Emulator |
|
Back to top |
|
|
Gergan Penkov Veteran
Joined: 17 Jul 2004 Posts: 1464 Location: das kleinste Kuhdorf Deutschlands :)
|
Posted: Sat Jun 24, 2006 10:00 pm Post subject: |
|
|
And not to forget the fragmentation, and this on old hardware - the portage fragments extremely the hard disk with squashfs and probably with in memory mounted unionfs overlay there will be virtually no chances for fragmentation which is IMHO the best thing out of this and the speed of course _________________ "I knew when an angel whispered into my ear,
You gotta get him away, yeah
Hey little bitch!
Be glad you finally walked away or you may have not lived another day."
Godsmack |
|
Back to top |
|
|
enderandrew l33t
Joined: 25 Oct 2005 Posts: 731
|
Posted: Sun Jun 25, 2006 3:42 am Post subject: |
|
|
Da Fox wrote: | You don't actually need any HD space for unsquashing, as you can build the new portage.squash directly from the old (mounted) portage.squash+unionfs.
Also squashfs(3.0) is in the standard (gentoo-sources) kernel. Even on old hardware a 30MB portage tree is _very_ small, there really isn't much point in making it even smaller than that because compared to the overall size of the install that's small enough. The sync speed shouldn't be much of a problem, it is already speeded up because most work is now being done 'in memory', squashing really doesn't take that long (even on older hardware I image it shouldn't take very long after the initial squash) and you only sync once a week (or so I hope ) so the time 'lost' is minimal. Besides syncing can be done while you continue to work on your computer.
In the end I think it really comes down to exactly _how_ small your HDD is (A few hundred MB perhaps?) and how much additional work you are willing to do. With squashfs you set it up once and it runs beautifully on it's own without ever having to do any sort of maintenance or adding/removing ebuilds. But if one was really desperate for the fastest and smallest possible portage on old hardware, go for it |
In the prlock thread, the example was going down to 18 megs without any compression.
It is faster, and it uses less bandwidth which makes it more friendly on the various Gentoo mirrors.
As far as work, prlock has an easier setup, and is also a one-time thing that ends up saving you time (no compress/expand/recompress, as well as faster syncs and searches). Having a smaller portage naturally is a win-win-win.
If you *REALLY* wanted, you could combine both methods and shrink portage down very small. _________________ Nihilism makes me smile. |
|
Back to top |
|
|
yoshi314 l33t
Joined: 30 Dec 2004 Posts: 850 Location: PL
|
|
Back to top |
|
|
sfragis Tux's lil' helper
Joined: 24 Mar 2005 Posts: 95 Location: RE < IT < Europe
|
Posted: Tue Aug 08, 2006 4:17 pm Post subject: |
|
|
Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:
- block size of 65536 bytes (default) => 38 MB
- block size of 16384 bytes => 44 MB
- block size of 4096 bytes (minimum accepted by mksquashfs) => 55 MB
Hence the best value for the block size is 64KB which seems in contraddiction with the result of Phlogiston's guide (smaller block size leads to less filesystem usage). Has anyone tried to tune this or other parameters in order to achieve different results? _________________ Regards
Fabio Strozzi |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Tue Aug 08, 2006 7:48 pm Post subject: |
|
|
sfragis wrote: |
Inspired by this guide I tried to create a smaller squashfs image of the portage tree using smaller values for the block size. Here's the result:
- block size of 65536 bytes (default) => 38 MB
- block size of 16384 bytes => 44 MB
- block size of 4096 bytes (minimum accepted by mksquashfs) => 55 MB
Hence the best value for the block size is 64KB which seems in contraddiction with the result of Phlogiston's guide (smaller block size leads to less filesystem usage). Has anyone tried to tune this or other parameters in order to achieve different results? |
Hmm probably this is different when using squashfs, but actually I can't explain.
But my question: Is there any howto that sums this stuff here up and gives clear instructions about using portage on a squashfs? I mean what are the results of that thread? Squasfs + unionfs ->rw and then just portage on it?
Thanks! _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Tue Aug 08, 2006 8:03 pm Post subject: |
|
|
Thanks! I'd like to sync it myself but I don't like the init scripts because I never shutdown actually. _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Tue Aug 08, 2006 8:15 pm Post subject: |
|
|
This is quite interesting:
Code: |
Creating little endian 3.0 filesystem on /tmp/images/portage.sqfs, block size 65536.
Little endian filesystem, data block size 65536, compressed data, compressed metadata, compressed fragments
Filesystem size 38251.43 Kbytes (37.35 Mbytes)
25.09% of uncompressed filesystem size (152465.07 Kbytes)
Inode table size 1652513 bytes (1613.78 Kbytes)
35.97% of uncompressed inode table size (4594717 bytes)
Directory table size 1407370 bytes (1374.38 Kbytes)
40.76% of uncompressed directory table size (3453208 bytes)
Number of duplicate files found 12688
Number of inodes 146450
Number of files 123037
Number of fragments 2275
Number of symbolic links 1
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 23412
Number of uids 2
root (0)
portage (250)
Number of gids 0
|
Especially the duplicate files
I'm thinking more and more that the way of portage syncing and storing is quite deprecated. Using portage and syncing just the compressed images cuts down the traffic incredibly.
/edit: Wow this rocks:
Code: |
time /usr/local/sbin/psync.2
[...]
All done, you may now enjoy your Updated Portage Tree :p <-this actually not 8)
real 2m19.984s
user 0m27.094s
sys 0m18.061s
|
Thanks a lot... Although I think that the first emerge -Duvat world afterwards is not really faster... _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
sfragis Tux's lil' helper
Joined: 24 Mar 2005 Posts: 95 Location: RE < IT < Europe
|
Posted: Tue Aug 08, 2006 10:56 pm Post subject: |
|
|
I wrote a small script to wrap the portage tree sync and automagically mount/umount the unionfs overlay and keep the squashfs image up to date. Code uses parts of scripts published in this and other threads of the Gentoo forum. There's a small header in it with a brief explanation of what I did to get things working. When I'll have more spare time I'll write a detailed html page. Feel free to trash it, use it and edit it _________________ Regards
Fabio Strozzi |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Wed Aug 09, 2006 5:27 am Post subject: |
|
|
Hmm I have a new problem now: After activating squashfs and unionfs in my kernel config, suspend2 does not work any longer. It hangs at doing atomic copy. Any hints? _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
sfragis Tux's lil' helper
Joined: 24 Mar 2005 Posts: 95 Location: RE < IT < Europe
|
Posted: Wed Aug 09, 2006 7:12 am Post subject: |
|
|
Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted? _________________ Regards
Fabio Strozzi |
|
Back to top |
|
|
micko Tux's lil' helper
Joined: 16 Dec 2004 Posts: 121 Location: Finland / Turku
|
Posted: Wed Aug 09, 2006 9:18 am Post subject: |
|
|
Quote: |
Thanks! I'd like to sync it myself but I don't like the init scripts because I never shutdown actually. |
I wrote an extremely simple script that sync's portage and restarts the service, so the squashfs-image gets updated. If I forget to use the script and sync normally nothing will go wrong, unless my computer crashes, and even then everything is fixed by a new sync. I never shut down either and sync maybe about 4-5 times a week. Again, if I forget to use the script once but remember the next time, it will update the image then.
This unionfs-method is great also because it works well with layman.
That is for my fastest computer. For my server and laptop I've written a script that umounts portage dl's the portage.sqfs with scp from my desktop and mounts the new one. Everything works just great. |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Wed Aug 09, 2006 11:36 am Post subject: |
|
|
sfragis wrote: | Does hibernation fail even if you don't load any squashfs or unionfs module, simply booting with the compiled new kernel? Or, does the problem raise after you've loaded squashfs and unionfs modules or after these filesystems have been mounted? |
I compiled the stuff directly into the kernel. (I hate modules ) But I will find out if it`s squashfs or unionfs, and then try with modules. PS: It didn't help to umount portage before. So even I don't use sqaushfs or unionfs it breaks hibernation. That is at least a bug in my opinion.
BTW using latest beyond3 sources.
/update: I just found out that its not squashfs nor unionfs related. It just has to do with my kernel, I don't know, using an older one now. _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
tost Guru
Joined: 10 Dec 2005 Posts: 506
|
Posted: Sun Aug 13, 2006 11:02 am Post subject: |
|
|
Now I´m a bit confused
I really like the idea of using squashfs for Portage, but for me it seems that it doesn´t really work.
I activated all the stuff in the kernel and after this viewPortageX (that Superkaramba theme) shows me that some updates are available.
So I run emerge-sync-squash.sh. But at the end of it emerge -NuDpv world says that no updates are available.
And now the things get strange, because emerge -pv vcdimager wants to upgrade the package ?
Whats going wrong ? |
|
Back to top |
|
|
adsmith Veteran
Joined: 26 Sep 2004 Posts: 1386 Location: NC, USA
|
Posted: Sun Aug 13, 2006 1:09 pm Post subject: |
|
|
I'm not sure I entirely understand your problem. If you jsut man that a world update is missing things, it may be totally unrelated to squashfs.
1) Is "getbinpkg" activated in FEATURES in your make.conf? That sometimes causes missed packages in world updates
2) are you sure the squashed portage tree is complete and mounted properly, and that PORTDIR is set properly?
3) Is your world file actually up-to-date? |
|
Back to top |
|
|
tost Guru
Joined: 10 Dec 2005 Posts: 506
|
Posted: Sun Aug 13, 2006 2:05 pm Post subject: |
|
|
mmhh I thought the problem is a consequence of my compressed portage tree, but now I saw that it doesn´t work either when I use portage in "default-mode" (uncompressed)
Quote: | 3) Is your world file actually up-to-date? |
I´ll check it, thanks
tost |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Mon Sep 04, 2006 11:34 am Post subject: |
|
|
I'm still using DaFox script to update my portage and it works well (though, any updates?)
But today I noticed an ugly slowdown: emerge -ua world takes minutes to calcuate the updates. I don't know if its the portage version I am using( Installed: 2.1.1_rc1-r2) or the latest kernel update.
Anyone else noticed this? _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
Phlogiston Veteran
Joined: 27 Jan 2004 Posts: 1925 Location: Europe, Swizerland
|
Posted: Mon Feb 19, 2007 10:11 pm Post subject: |
|
|
Hey squashfs portage user whats up? I'm still using the psync script here and it works perfectly. Is anyone else still using it or is there any better way to do it? _________________ Workstation: 5.1 SurroundSound, LIRC remote control; Laptop [IBM-T43]: patched sources, s2disk/ram, fingerprint sensor |
|
Back to top |
|
|
|