Squashfs with LZMA compression [ebuild]

[cyrillic]

Updated ebuild several posts down


I'm not sure how much interest there is in squashfs these days, and there doesn't seem to be too much activity upstream ...

But hey, I use squashfs on a regular basis, so I cooked up some kernel patches (because I prefer not to build stuff outside the kernel tree), and fixed (butchered) the squashfs-tools ebuild to include LZMA support.

These patches are for 2.6.22 and 2.6.23 vanilla kernels respectively.
sqlzma.22.patch 162KB
sqlzma.23.patch 162KB
If your kernel (gentoo-sources, etc.) already includes partial squashfs support, then these may not apply cleanly.

sys-fs/squashfs-tools/squashfs-tools-3.2_p2.ebuild

Code: Select all

# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-fs/squashfs-tools/squashfs-tools-3.2_p2.ebuild,v 1.1 2007/05/16 19:19:07 wolf31o2 Exp $
# Added LZMA stuff 2007/10/14 cyrillic

inherit eutils toolchain-funcs

MY_PV=${PV/_p/-r}
DESCRIPTION="Tool for creating compressed filesystem type squashfs"
HOMEPAGE="http://www.squashfs-lzma.org/"
# SRC_URI="mirror://sourceforge/squashfs/squashfs${MY_PV}.tar.gz"
SRC_URI="http://www.squashfs-lzma.org/dl/squashfs3.2-r2.tar.gz
http://www.squashfs-lzma.org/dl/lzma443.tar.bz2
http://www.squashfs-lzma.org/dl/sqlzma3.2-r2b.tar.bz2"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86"
IUSE=""

RDEPEND="sys-libs/zlib"

# S=${WORKDIR}/squashfs${PV/_p/-r}/squashfs-tools

src_unpack() {
        cd "${WORKDIR}"
        unpack squashfs3.2-r2.tar.gz sqlzma3.2-r2b.tar.bz2
        mkdir lzma443
        cd lzma443
        unpack lzma443.tar.bz2
        cd "${WORKDIR}"
        epatch sqlzma1-443.patch
        epatch sqlzma2u-3.2-r2.patch
        sed -i "s:-O2:${CFLAGS}:" squashfs3.2-r2/squashfs-tools/Makefile
        sed -i "s:-O2:${CFLAGS}:" lzma443/C/7zip/Compress/LZMA_C/makefile.gcc
        sed -i "s:-O2:${CFLAGS}:" lzma443/C/7zip/Compress/LZMA_Alone/makefile.gcc
        sed -i "s:KDir =:# KDir =:" Makefile
        sed -i "s:BuildSquashfs =:# BuildSquashfs =:" Makefile
}

src_compile() {
        emake CC="$(tc-getCC)" || die
}

src_install() {
        cd squashfs3.2-r2/squashfs-tools
        dobin mksquashfs unsquashfs || die
        cd ..
        dodoc README ACKNOWLEDGEMENTS CHANGES PERFORMANCE.README README-3.2
        cd ..
        dodoc sqlzma.txt
} 

For more info, look here : http://www.squashfs-lzma.org

EDIT: Better CFLAGS support in ebuild.

EDIT2 : Added patch for 2.6.24-rc kernels.

EDIT3 : Scrapped patch for 2.6.24* -- it compiled fine, but caused a kernel panic if you actually tried to mount something.

[hirakendu]

thanks a lot for making it trivial! shall try it sometime .

[likewhoa]

sweet, time to test on my livecd.

[yoshi314]

so there is support for lzma as an unofficial patch for squashfs. it packs pretty good, but *very* slow at packing - i keep my portage and overlays on squashfs loopback, and i saw a benefit of ~5-10% size-wise. and a huge increase in compression time :/

can i somehow use other compressiom methods with squashfs, other than the default one (what is the default method btw?) or lzma?

[hirakendu]

I guess the default squashfs is pretty fast at packing too. I think it uses gzip or lzo (whatever is supported by kernel) and compresses well too. (I get livecd images of 1.9 gb from my full blown system of size 6.4 gb with default squashfs, all in under 25 minutes on an athlon64x2 and sata hdd @ 63 MB/s.)

Although I didn't try sqlzma as yet, but from my experiences of lzma, I would expect it to be slow at compressing. (Typically, 'tar -czf ...' would take 10-15 minutes, while '... lzma e -si -so ...' would take 2 hours for compressing my 6 gb system. The output tar.gz used to be 1.8 gb, while tar.lzma used to be 1.3 gb. In contrast, bzip would take 35 minutes to give out tar.bz2 of 1.6 gb. And hence, as it would anyway require a dvd to store, I always stuck to default squashfs / tar.gz for backup.)

But there are two bright points. One is that if you make the image(s) seldom, lzma is pretty good choice - we can compress a further lot more by increasing dictionary sizes (I haven't tried this for my impatience). Second is that decompression is lightning fast - as fast or better than gzip. This is where bzip2 is bad imo - it takes larger time to even decompress. Especially the small sizes in conjunction with slow i/o speed of dvd-drives, high cpu power (and fast hard-drive writes) makes lzma look even better than what it is, compared to others.

Aside, I would go on to add that as of now, I am extremely happy with regular squashfs combined with my usb flash drive. I use a usb flash drive of > 2gb (its a sandisk cruzer 4gb which gives a read speed of about 17 MB/s) to store my 1.9 gb image and its screaming faster than booting from my hard drive! (Must also add that the true random access nature of flash drives makes it more lethal than what its already impressive 17 MB/s seems!) Not to mention the no rotating parts noise. Wonder what would sqlzma make it look like .

[hirakendu]

Finally tried and tested it. Many thanks for making it work easily! (Used the patch for 2.6.22.10 .)

Although the overall result has been a bit disappointing for me compared to the much hype I had come across. My 6.4 gb of system was compressed to 1.7 gb by lzma instead of 1.95 gb with default zlib. With lzma, I was expecting something less than 1.5 gb at least . And it took 2 full hrs to do that instead of 25 minutes with zlib.

Of course, the brighter side is that its completely backward compatible. (Not only did the old squashfs images work with the new kernel, even the new ones created with -nolzma option worked fine with old kernel. So just happy with -nolzma at the moment .)

[baeksu]

I can't access the kernel patches (sharedzilla is down?). Could someone give another source for them, or at least for .23 kernels?

[hirakendu]

http://hirakendu.mooo.com/sqlzma-temp/

[baeksu]

http://hirakendu.mooo.com/sqlzma-temp/

Excellent! Thank you very much.

[yoshi314]

let's hope squashfs3.3 ebuilds get posted quickly, as it's already out.

[hirakendu]

Thats nice. Especially I wanted to try out blocksizes of 128K and more. But I don't think it will lead to any drastic improvements. And as mentioned earlier, lzma didn't turn out to be dramatic (but the time taken to compress will definitely surprise 1st timers ).

[yoshi314]

Thats nice. Especially I wanted to try out blocksizes of 128K and more. But I don't think it will lead to any drastic improvements. And as mentioned earlier, lzma didn't turn out to be dramatic (but the time taken to compress will definitely surprise 1st timers ).

it _does_ see slax6 blog for results. they managed to reduce filesystem from 227mb to 167mb by using 1mb blocksize.

http://blog.slax.org/2007/11/02/mksquas ... omparison/

[hirakendu]

Yep, I had made that comment after reading that blog only . That 167 mb is using 1mb + lzma. The result with 1mb + gzip is hardly impressive, compared to 64kb. Besides I don't know the increase (hopefully decrease) in time that would incur.

[cyrillic]

I normally compress the portage tree, and it looks like 1MB blocks really help LZMA to compress better :
44MB with zlib + default 128K blocks
32MB with lzma + 1M blocks

This is the next generation of sqlzma, based on squashfs-3.3 and the latest LZMA patches that just came out a couple of days ago.

You can find the ebuild and kernel patches here : ftp://71.184.254.182
shh ... don't tell my ISP that I am hosting files at home.

sys-fs/squashfs-tools/squashfs-tools-3.3.ebuild

Code: Select all

# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

inherit eutils toolchain-funcs

SQUASH_PV="squashfs3.3"
LZMA_PV="lzma449"

DESCRIPTION="Tool for creating compressed filesystem type squashfs with LZMA"
HOMEPAGE="http://www.squashfs-lzma.org/"
SRC_URI="mirror://sourceforge/squashfs/${SQUASH_PV}.tar.gz
mirror://sourceforge/sevenzip/${LZMA_PV}.tar.bz2
http://www.squashfs-lzma.org/dl/${SQUASH_PV}.tar.gz
http://www.squashfs-lzma.org/dl/${LZMA_PV}.tar.bz2
http://www.squashfs-lzma.org/dl/sqlzma3.3-fixed.tar.bz2"

LICENSE="GPL-2"
SLOT="0"
KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86"
IUSE=""

RDEPEND="sys-libs/zlib"

src_unpack() {
        cd ${WORKDIR}
        unpack ${SQUASH_PV}.tar.gz sqlzma3.3-fixed.tar.bz2
        mkdir ${LZMA_PV}
        cd ${LZMA_PV}
        unpack ${LZMA_PV}.tar.bz2
        cd ..
        epatch sqlzma1-449.patch
        epatch sqlzma2u-3.3.patch
        sed -i "s:-O2:${CFLAGS}:" ${SQUASH_PV}/squashfs-tools/Makefile
        sed -i "s:-O2:${CFLAGS}:" ${LZMA_PV}/C/Compress/Lzma/sqlzma.mk
        sed -i "s:-O2:${CFLAGS}:" ${LZMA_PV}/CPP/7zip/Compress/LZMA_Alone/makefile.gcc
        sed -i "s:KDir =:# KDir =:" Makefile
        sed -i "s:BuildSquashfs =:# BuildSquashfs =:" Makefile
}

src_compile() {
        emake CC="$(tc-getCC)" || die
}

src_install() {
        cd ${SQUASH_PV}/squashfs-tools
        dobin mksquashfs unsquashfs || die
        cd ..
        dodoc README ACKNOWLEDGEMENTS CHANGES COPYING PERFORMANCE.README README-3.3
        cd ..
        dodoc sqlzma.txt
} 

EDIT : ebuild cleanup

[uli]

Hi cyrillic,

I'm not an advanced Gentoo user. So up to now I have simply emerged packages but also managed to make an overlay.

But I have not tried to use just an ebuild. So I'm interested in squashfs/lzma together with 2.6.24-rc2-rt1 (patched with rt1). You ftp-link now shows the ebuild 3.3 and then some kernel patches and also some squashfs/lzma files.

Can you please describe the steps to patch the kernel and to run the ebuild in the proper way?

Thx

Uli

[cyrillic]

This is how you patch your 2.6.24-* kernel.

Code: Select all

# cd /usr/src/linux
# patch -p1 < /path_to_patch/sqlzma-ng.24.patch 

If your overlay is located at /usr/local/portage you can just add the ebuild like this.

Code: Select all

# mkdir -p /usr/local/portage/sys-fs/squashfs-tools
# cp squashfs-tools-3.3.ebuild /usr/local/portage/sys-fs/squashfs-tools/
# ebuild /usr/local/portage/sys-fs/squashfs-tools/squashfs-tools-3.3.ebuild digest
# emerge squashfs-tools 

[hirakendu]

Seems that sqlzma-3.3 is borked for me. I am guessing only the kernel part. The mksquashfs part works fine (even with lzma I believe). Even the mounting part works fine. The trouble is when I chroot into the mounted system, the filesystem is all messed up. Namely, files are spuriously linked to each other. Example, things are as crazy as /etc/mtab linked to /usr/bin/make. And I believe it happens only from within chroot. So those using it for just squashing their portage-tree etc may not notice it. However, its required for livedisks. Has anyone else noticed this. Btw, this happens even with the old squashfs images created with squashfs-tools-3.2 and using kernel with new sqlzma patches. And using vanilla squashfs-3.3 with the kernel, things are back to normal. So I guess its very much something with sqlzma 3.3 patches for kernel. I have noticed this 'bug' with both sqlzma patch in this thread for 2.6.24 and zen-sources (2.6.24-zen1). Is there possibly any workaround or mount options to get around this?

As of now, using plain vanilla squashfs-3.3 for the kernel part and squashfs-tools with lzma (using ebuild in this thread). Using '-nolzma' and '-b 1024KB' and things look pretty good.

[prymitive]

I'm using squashfs with lzma to make initrd images for booting servers from network using PXE, to allow kernel to load lzma squashfs at boot You need this simple patch:
http://prymitive.com/files/lzma_ramdisk_boot.diff

[hirakendu]

I am guessing you didn't get my problem. I don't have issues with booting and/or mounting. Just that the filesystem is borked - many files are wrongly linked to each other.

EDIT : Found the bug filed with debian : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456489.

[prymitive]

I am guessing you didn't get my problem. I don't have issues with booting and/or mounting. Just that the filesystem is borked - many files are wrongly linked to each other.

EDIT : Found the bug filed with debian : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456489.

Did I said that this will help You ? I just posted this patch becouse I think that someone maybe interesed, it's not related to any post, only the topic.

[hirakendu]

Apologies . I thought 'You' = 'me' .

[cyrillic]

So I guess its very much something with sqlzma 3.3 patches for kernel. I have noticed this 'bug' with both sqlzma patch in this thread for 2.6.24 and zen-sources (2.6.24-zen1).

Yeah, I noticed some portage tree corruption with sqlzma, but it seemed to vary depending on the kernel version.
I did not manage to track down what was causing this (due to laziness and lack of programming skills), and for now I have discontinued using squashfs.

Maybe when I have more time, I will try to dissect a SLAX CD, and see what they are doing differently (or see if they are experiencing the same buggy behavior).

[hirakendu]

hmm, i thought corruption was only when i chroot. externally, all seemed to look ok. yeah, i too thought of rev-engg the slax cd, but short on time. i heard a big one is coming on wednesday .

[avx]

Thanks very much, it's working fine for me :)

Compressed portage is now down to 4.7MB, has been 5.5MB before. If only it weren't that slow, but that's the price.

[mikegpitt]

Do you still have the patch for the 2.6.23 sources? It doesn't appear to be available on the sqashlzma site anymore, and the d/l links you posted seem to be dead.