TIP: Compressing portage using squashfs: initscript method

[nato.Otan]

This sound an excelent alternative to run portage lean and swift. The method used here reminds me about puppyos. PuppyOS is extremely small distro that runs off RAM, and when you click on the programs they start instantly in the range of nano-seconds. Pretty cool.

Here I give the link to the descriptions of the bowels of puppyOS, and perhaps it could be implemented on this script to make it even Faster Than Light!!! And rename portage to Tachyon!!!

How Puppy Works

[synss]

Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2 if that would work that would obsolete these scripts (for the best I must say) however, I cannot compile compFUSEd. And I do not know how to deal with that kind of problems. If anybody is successful with that, I would be very interested.

Being able to mount a freshly downloaded tarball and update it would be great IMHO. The author uses it with his kernel sources and says he can compile into the archive and install his kernel. There are many things I can think of with these FUSE-related fs.

[mv]

Well, now things like compFUSEd claim they do support mounting + read-write of archives in .tar.bz2

As I understood, they write in their own archive format (based on lzo). I have doubts that any writable filesystem can be fast and simultaneously compress many small files effectively. However, I would like that somebody proves that I am wrong with this conjecture.

I cannot compile compFUSEd. [...]There are many things I can think of with these FUSE-related fs

Yes, it is a pity that for most FUSE filesystems there are no .ebuilds in the portage tree.
Actually, I would also prefer to use an FUSE-based unionfs http://freshmeat.net/projects/unionfs-fuse/, because it needs no kernel module. However, I am too lazy to write an .ebuild.

[synss]

Actually, I would also prefer to use an FUSE-based unionfs http://freshmeat.net/projects/unionfs-fuse/, because it needs no kernel module. However, I am too lazy to write an .ebuild.

Agreed, however, there is an ebuild for funionfs in the overlay sunrise, I will see whether it works here and try to use it.

As of compFUSEd, I believe the author says he does support tar.bz2 without restrictions. There are other compressed fs who only work on lzo/gz but this one claims it can deal with .tar.bz2, so it would be worth trying. I contacted the author yesterday asking for support, I also suggested him to commit an ebuild (he is a gentoo user) to bugzy, that would earn him feedback and support, and if it turns out to work well enough, well, that would be really nice.

As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think. Only size on the HD matters. And for compressing other things, well then you can chose speed or compression.

BTW, I do not really understand why you have implemented tarred backups in the script: you can just download a new image of the tree from the ftp servers around here if ever something breaks badly, you do not need to do these time-consuming backups, do you? That's why experimenting new fs is fun with the tree, if you break it, just download a new one! Risk-free.

My point above you did not understand (probably was not clear) was that, in your check_config() function, if [[ ! -e "{PORTAGE_SQFS} ]]; blah, instead of echoing a message, you should actually squash the tree to a new image and then mount it. That would allow complete newbies to just download the script and run it and compression would be done automatically upon the first use.

Thank you for mentioning me and I think you've changed the identation and now it is easier to read


[EDIT] well, funionfs compiled flawlessly (x86) and did not complain when I asked it to mount my tree, thus

Code: Select all

emerge layman
layman -a sunrise
/etc/init.d/squash_portage stop
modprobe -r unionfs
echo "~sys-fs/funionfs-0.4.2" >> /etc/portage/package.keywords
emerge funionfs

Code: Select all

	#mount -t funionfs -o dirs=$PORTAGE_RW=rw:$PORTAGE_RO=ro unionfs $PORTDIR    
	funionfs -o dirs=$PORTAGE_RW: $PORTAGE_RO=RO -o allow_other NONE $PORTDIR

that is in the start() thing.

Code: Select all

/etc/init.d/squash_portage start

Does not complain, I will try syncing later though, for I just did and syncing several times a day is bad practice. But it does look good! If syncing works I will update my HOWTO, chmod'ing as you do is also a good idea, obviously, I will probably add a few of your things in mine, but I'll leave the tarred backups out...

[SECOND EDIT]sys-fs/fuse-2.6.0_rc1
stable, from emerge (not the one in the kernel) although form their website, they say funionfs should work with older versions of fuse, + my kernel is stable gentoo-source-2.6.17-r8

[THIRD EDIT]https://bugs.gentoo.org/show_bug.cgi?id=143026
an ebuild for UnionFsFuse don't ask me what the difference is or which is best between funionfs and UnionFsFuse (but give me the answer)

[mv]

As of performance, I do not think it matters so much here: you have eix for fast searches and the tree is actually small and cached pretty fast if you use emerge, I think.

The point where I expect an enormous speed loss for .tar.bz2 files (if this would really be supported) is when the data is written to the tree, i.e. for emerge --sync. Just to have some numbers (for emerge --sync with a fast server and a fast connection):

• Standard (without squash_portage): 3-6 minutes, depending on the filesystem fragmentation.
  
  With squash_portage: <1 minute (that's why I like it so much: It saves not only space but also time, even if one includes the compression time at the shutdown).
  
  With pure .tar.bz2: ??? (I expect 30 minutes or more unless you have very much ram/swap, because every time the (uncompressed!) data cannot be held in ram anymore, practically the whole tree has to be compressed, since any change in the data in the middle of a .tar.bz2 file implies changes until the end of the file by the structure of .tar.bz2 files).

you can just download a new image of the tree from the ftp servers around here

Not every computer has easy access to the internet. Moreover, the portage tree is rather large (compared to the few data for a typical emerge --sync): It can really cost money if you have to pay for the bandwith.

you do not need to do these time-consuming backups, do you?

A backup can be also handy if the previous emerge --sync dropped some ebuild/patch which you wanted. Yes, there is a cvs of the tree, but a backup is more convenient in such cases.

instead of echoing a message, you should actually squash the tree to a new image and then mount it.

An automatic action for such a fundamental change is very dangerous: Once the new tree is mounted, a complete newbie will not be able to remove the old tree anymore. The script could of course also remove the old tree, but then if something goes wrong with the mounting (wrong kernel, insufficient disk space, ...) the user is left without a portage tree. I think the printed instruction is clear enough, and in this way the user can understand what is going on and why, and he has no bad surprise just by trying the script.

[synss]

Update!

I tried to include the feedback posted in this thread (including some that were posted a long time ago...) plus a few cosmetic changes and a bit more informative messages.

Changelog:
2006-11-19
Security enhancements thanks to mv
Only two sqfs images will now reside on the system: portage-old.sqfs and portage-current.sqfs
changed name of sqfs-related variables
more informational messages (it is made clear that the read-only image is mounted first and that it exists even if unionfs fails, that has always been the way it works but it seems not everybody got it)
correct usage of eend (i.e. !! when mount fails instead of OK)
tried to keep it short and simple without too many bashisms

The updated scripts are in the first post. Comments and bugs welcome.

[mv]

Update also for the other script.
It allows now to choose alternatively funionfs instead of unionfs, see [bug=151673]funionfs.ebuild[/bug].
funionfs is based on fuse and thus works with all new kernels.
Since, in contrast, unionfs has not yet been ported for the stable kernel 2.8.19, funionfs is currently the only way to use this approach with the newest kernels (and it has the advantage that probably newer kernel releases won't break it again). The disadvantage is that funionfs is way slower than unionfs.

[NaiL]

Maybe the next thing is to build the squash_portage.ebuild that install all the config files and check for modules and with use flag to select from unionfs or funionfs

[NaiL]

Why don't do the same with:
/var/cache/edb
/var/db/pkg

Maybe it's a noob question

[synss]

Why don't do the same with:
/var/cache/edb
/var/db/pkg

Maybe it's a noob question

No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...

And mery christmas!

[NaiL]

No, it should work and that is one idea I had (also add /usr/src/linux) but no time for it. It would not be very diffictult to implement either, like only another for loop. Later, maybe...

And mery christmas!

Why with /usr/src/linux? because there are lots of files?

mery christmas to you! too :p

[mv]

Why don't do the same with:
/var/cache/edb
/var/db/pkg

These two are not worth the trouble, because they do not take too much space. It would probably be more interesting to use it for /usr/share/doc or /usr/src/linux, because these two should compress very well. In principle, you can simply use the same script repeatedly only with different directory/filename settings. Only the previous variable names are then misleading...

Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/ ... index.html is now called squash_dir and has no misleadings variable names.

Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are:

• 1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs).
  
  2. You can explicitly avoid recompression during shutdown (by creating a certain magic file). In particular, if you use this script to compress /usr/src/linux, it is usually not worth to recompress /usr/src/linux after a kernel recompilation: Instead it is more reasonable to leave the modified files in a separate directory and simply use this directory on the next start (thus never do any recompression during shutdown).

However, despite these changes, I am not sure whether it is a good idea to compress /usr/src/linux: If you compiled a new kernel but e.g. unionfs does not run on it yet, you might get troubles to access your original kernel read-writable so that you are able to compile unionfs...
Anyway, /usr/share/doc should cause less troubles (but it may also cause some, because e.g. the ebuild for unionfs might want to install something there).

[NaiL]

For me the first important thing is to avoid disk fragmentation in the measure of possible, secondly speed up the portage metadata updates.
Thats why i think to do that with all portage related directories.

The fact of gain disk space, for me is a welcomed secondary effect.

[mv]

secondly speed up the portage metadata updates

Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient.

Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up.

[NaiL]

secondly speed up the portage metadata updates

Do you have a particular reason to do metadata updates? For most users, FEATURES='-metadata-transfer' and choosing the metadata_overlay cache is sufficient.

I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well?

Anyway, if you really need metadata updates, you should use the script with PREFER=unionfs (in contrast to funionfs) - funionfs will probably slow down things instead of speeding them up.

I'm currently only using unionfs with the synss script. I'll try your script when i have some time! (It looks very well. "all about choice")

[mv]

I didn't know this feature, I'm using it now. thanks. but it works with portage overlays as well?

For most users it works. Some things like eclasses in overlays can cause troubles. If you need the latter, an alternative is to use the sqlite cache method.

"all about choice"

You will finally probably end up writing your own script - after all, these are all just very simple scripts with a very limited purpose. It's only the idea which is important...

[kuku]

Christmas Present: The new version of the init-script on http://www.mathematik.uni-wuerzburg.de/ ... index.html is now called squash_dir and has no misleadings variable names.

Moreover, the new script has some new configuration options which are handy for some applications. The most important ones are:

• 1. You can explicitly mount a directory read-only (which needs no unionfs/funionfs).

Hi - I've tried your script and i got some errors:

Code: Select all

 ./squash_portage restart
 * Caching service dependencies ...                                       [ ok ]
 * Mounting /var/portage/portage.sqfs as /usr/portage ...
fusermount: mountpoint is not empty
fusermount: if you are sure this is safe, use the 'nonempty' mount option
 * Failed mounting /var/portage/change with funionfs [exit with 1]
mount: unknown filesystem type 'unionfs'
 * Failed mounting /var/portage/change with unionfs [exit with 32]
mount: wrong fs type, bad option, bad superblock on ,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

 * Failed even rbinding /usr/portage [exit with 32]   

this is my /etc/conf.d/squash_portage

Code: Select all

DIRECTORY="$(. /etc/make.conf ; echo "${PORTDIR}")"
FILE_SQFS="/var/portage/portage.sqfs"
FILE_SQFS_OLD="/var/portage/portage.old.sqfs"
PREFER="funionfs"
DIR_CHANGE="/var/portage/change"

any hint ?

[mv]

Thanks for the feedback. It seems that I introduced an error briefly before release but after testing the special case of empty DIR_SQUASH (which I do not recommend, anyway). Setting DIR_SQUASH to some empty directory should fix your problem.
Today's version of the script should hopefully also fix the bug.

I must say that I had some bad experience with funionfs: It seems that sometimes(?) portage cannot "see" the files mounted by funionfs, e.g. it occassionally complains about missing eclasses or even a wrong profile link (although an ls shows that the files are there): I guess some directory function is not always properly mapped by funionfs.
Does anybody have an idea what might be wrong?

So, in the moment, I can recommend only unionfs which you apparently haven't installed:

Code: Select all

mount: unknown filesystem type 'unionfs'

BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work...

[kuku]

BTW: If there is a need of unionfs for kernel 2.6.19, I might publish some quick-and-dirty hacks with instructions how to make them work...

I only use funionfs of 2.6.19 so it might be helpfull

[mv]

See unionfs.tar.gz on http://www.mathematik.uni-wuerzburg.de/ ... index.html for ebuild/instructions on how to run the current unionfs snapshot with kernel 2.6.19.

[Maxwell]

Hi!


I'm trying to use an old pc as an host for some network services and i wanted to use its (64M) ram to speed things up. So when i searched the forums i found this.

As you all have tried several ways of doing this (sort of), how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%?


Thanks for the help!

[synss]

As you all have tried several ways of doing this (sort of), how can i put something in ram when i boot from grub and then use the ram as the root filesystem? How much can i compress the data? 20% or 30%?

This is going seriously off-topic so you might as well start a new thread, but what you are looking for is initramfs. Data compression can be achieved in there with squashfs or cramfs, look for live CDs too, some of them (knoppix for example) do that.

[Maxwell]

Thanks for the tip!

This is going seriously off-topic

Yeah, i know, but i've already started a thread, but no replies were found. Sometimes i just prefer to ask the right guy

but what you are looking for is initramfs.

Ok then!


BTW, how much compression can you get from the portage tree? how much ram is being used?


thanks for the help

[jpmayer87]

With squashfs I reduced the entire portage tree to about 44 Meg, that includes overlays as well.

JP

[mv]

Update: The new initscript version on http://www.mathematik.uni-wuerzburg.de/ ... index.html now also supports (and even prefers) aufs over unionfs. An ebuild for aufs is available from [bug=160065]here[/bug].