Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
FuseCompress - compressed filesystem overlay.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Nov 25, 2005 5:16 pm    Post subject: FuseCompress - compressed filesystem overlay. Reply with quote

FuseCompress is a compression filesystem, simply mount and go and every file you store there will be compressed.

URL : http://www.miio.net/fusecompress/
svn url : http://svn.berlios.de/svnroot/repos/fusecompress/trunk
(the svn is a LOT more up to date than the latest release is, highly recomended).

This fuse filesystem is a merge between Milan Svoboda's fusecompress implementation and my (Anders Aagaard) lzofs implementation (http://forums.gentoo.org/viewtopic-t-388855-highlight-lzofs.html). It is a LOT more robust and "simpler" than my lzofs, and it survives far more tests than lzofs ever did.

To test:
Code:

svn checkout http://svn.berlios.de/svnroot/repos/fusecompress/trunk
cd trunk
make # NOTE, if your doing performance tests, do make -f Makefile.release to disable debugging.
mkdir raw compressed
./fusecomp compressed raw

#Any file you write to the compressed dir after this will be stored in raw/ and be compressed.


Note that there are a lot of room for improvements in performance here, but it should be quite good from the start. Also this fs does not need a "clean" raw directory. So a simple:
mv /usr/portage /usr/portage.RAW
mkdir /usr/portage
fusecomp /usr/portage /usr/portage.RAW should work. (note that current build is forced to run foreground, so you can't close the terminal while your using it)

NOTE1: This filesystem is still experimental, and should not be used to store anything you care about.
NOTE2: If your performance testing things, dont rm -rf and rsync a lot of times, it's MEAN to the rsync mirrors. Make a local mirror instead (or rsync a directory on your harddrive).
NOTE3: If your compressing your portage tree, you wont see much gain. Your compressing TINY files, and (unless your using reiser with tail packing) files under 4096 bytes will still take 4096 bytes on your harddrive.

Tests passed on revision 42:
rsync (both with and without checksum checks)
100 rsync's running at the same time.
100 diff's running at the same time.


Last edited by neuron on Wed Nov 30, 2005 7:58 pm; edited 5 times in total
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Nov 25, 2005 5:17 pm    Post subject: Reply with quote

Making usefull bugreports:
If something goes wrong the first thing you can do is check /var/log/syslog or the output from fusecomp. It should have some info on where it failed.

Another extreamly helpfull thing is to run it with core dumps enabled (if you manage to get it to crash):
ulimit -c unlimited
fusecomp /a /b
(and to turn coredumps off afterwards do ulimit -c 0)

This'll produce a core.$PID file, which we can get a lot of info from (if your interested yourself do gdb --core=`ls -1 core.*` fusecomp and write bt (backtrace))
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Nov 25, 2005 6:15 pm    Post subject: Reply with quote

Is the new version incompatible with FuseCompress-0.6.1? When I tried to mount some data I was testing and I got a segmentation fault in fusecompress:

Code:
./fusecomp ../../mythburn.new/ ../../mythburn/
fusecompress.c:889(main()) : Program started
lists.c:129(compress_sometime()) : ../../mythburn/replex-0.1.4/ts.o added to 'comp' queue.l
ists.c:129(compress_sometime()) : ../../mythburn/replex-0.1.4/avi.c added to 'comp' queue.l
ists.c:129(compress_sometime()) : ../../mythburn/replex-0.1.4/ringbuffer.c added to 'comp'
queue.lists.c:129(compress_sometime()) : ../../mythburn/replex-0.1.4/element.c added to 'co
mp' queue.lists.c:129(compress_sometime()) : ../../mythburn/replex-0.1.4/mpg_common.c added
 to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/fonts/read_me.html added
 to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/install.sh.new added to
'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/scripts/apply_cutlist.sh add
ed to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/scripts/remplex.sh add
ed to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/scripts/mpeg2cut-mod a
dded to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/scripts/getmyth.sh a
dded to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/mythwebburn/mythtvbu
rndvd_createdvd.php added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn
/mythwebburn/mythtvburndvd_confirm.php added to 'comp' queue.lists.c:129(compress_sometime(
)) : ../../mythburn/mythwebburn/themes/Default/mythtvburndvd_confirm.php added to 'comp' qu
eue.lists.c:129(compress_sometime()) : ../../mythburn/mythwebburn/themes/Default/mythtvburn
dvd.php added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/mythwebburn
/mythtvburndvd.php added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/
install.sh added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/gop_fixu
p/gop_fixup added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/myth-mp
lex/inc/ctools.h added to 'comp' queue.lists.c:129(compress_sometime()) : ../../mythburn/my
th-mplex/inc/cpptools.hh added to 'comp' queue.lists.c:129(compress_sometime()) : ../../myt
hburn/myth-mplex/inc/transform.h added to 'comp' queue.compress.c:69(thread_compress()) : G
ot ../../mythburn/replex-0.1.4/ts.o
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpS1OkYacompress.c:6
9(thread_compress()) : Got ../../mythburn/replex-0.1.4/avi.c
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpuedE3zcompress.c:6
9(thread_compress()) : Got ../../mythburn/replex-0.1.4/ringbuffer.c
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpRGhe9Ycompress.c:6    9(thread_compress()) : Got ../../mythburn/replex-0.1.4/element.c
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpwMmZeocompress.c:6    9(thread_compress()) : Got ../../mythburn/replex-0.1.4/mpg_common.c
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmp1Tg2kNlists.c:129(    compress_sometime()) : ../../mythburn/myth-mplex/Makefile added to 'comp' queue.lists.c:129    (compress_sometime()) : ../../mythburn/myth-mplex/bits.cpp added to 'comp' queue.lists.c:12    9(compress_sometime()) : ../../mythburn/myth-mplex/mplex.cpp added to 'comp' queue.lists.c:    129(compress_sometime()) : ../../mythburn/myth-mplex/domplex.cpp added to 'comp' queue.list    s.c:129(compress_sometime()) : ../../mythburn/myth-mplex/systems.o added to 'comp' queue.li    sts.c:129(compress_sometime()) : ../../mythburn/myth-mplex/bits.o added to 'comp' queue.lis    ts.c:129(compress_sometime()) : ../../mythburn/myth-mplex/common.h added to 'comp' queue.li    sts.c:129(compress_sometime()) : ../../mythburn/myth-mplex/consts.h added to 'comp' queue.l    ists.c:129(compress_sometime()) : ../../mythburn/myth-mplex/timecode.cpp added to 'comp' qu    eue.lists.c:129(compress_sometime()) : ../../mythburn/myth-mplex/ringbuffy.c added to 'comp    ' queue.compress.c:69(thread_compress()) : Got ../../mythburn/fonts/read_me.html
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpSXhfrclists.c:129(    compress_sometime()) : ../../mythburn/changes.txt added to 'comp' queue.compress.c:69(threa    d_compress()) : Got ../../mythburn/install.sh.new
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmp1CmGxBcompress.c:6    9(thread_compress()) : Got ../../mythburn/scripts/apply_cutlist.sh
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpQAcmE0compress.c:6    9(thread_compress()) : Got ../../mythburn/scripts/remplex.sh
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpPnbcLpcompress.c:6    9(thread_compress()) : Got ../../mythburn/scripts/mpeg2cut-mod
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpwI3aSOcompress.c:6    9(thread_compress()) : Got ../../mythburn/scripts/getmyth.sh
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmpmA0iZdcompress.c:6    9(thread_compress()) : Got ../../mythburn/mythwebburn/mythtvburndvd_createdvd.php
compress.c:123(thread_compress()) : Using temp file ../../mythburn/._.tmp4NVz6Ccompress.c:6    9(thread_compress()) : Got ../../mythburn/mythwebburn/mythtvburndvd_confirm.php
Segmentation fault

_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Nov 25, 2005 6:21 pm    Post subject: Reply with quote

Is this the svn version? I'd REALLY use svn instead of 0.6.2, as a LOT of code changes has been done since then, and tons of code has been rewritten. So bugreports for 0.6.2 isn't worth much unfortunatly. If this is the svn, then mail me a coredump (aagaande@gmail.com) and then wipe the directory and see if you can reproduce it without having old files around. I'm not 100% sure if we're compatible with the older versions or not.

//edit, most likely we're not compatible, but it still shouldn't cause segfaults.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Nov 25, 2005 6:35 pm    Post subject: Reply with quote

This was the svn code from a few minutes ago.

Dumb question. I'm not getting a coredump, at least it is not in the current folder. How do I fix this??
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Nov 25, 2005 6:43 pm    Post subject: Reply with quote

see my post number two, you have to set ulimit first to allow it.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Nov 25, 2005 6:45 pm    Post subject: Reply with quote

Sorry, I missed that. :oops:
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Nov 25, 2005 7:06 pm    Post subject: Reply with quote

I believe the error was caused by specifying ../.. in the path instead of an absolute path. The following did not crash (these are the same folders):
Code:
md0 trunk # ./fusecomp /root/projects/mythburn.new /root/projects/mythburn/
fusecompress.c:889(main()) : Program started

_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Nov 25, 2005 7:12 pm    Post subject: Reply with quote

ah, brilliant, I'll add that to the original post ;)
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sat Nov 26, 2005 7:42 pm    Post subject: Reply with quote

quite big changes in the svn. In the merge we had two database systems, they've now been merged to one. Which should increase performance a LOT (it does half as many searches). Also there is no thread polling in the background now, which should also increase performance considerably. That means files that arn't compressed (if your mounting over an old directory) wont be immediatly compressed. But the first time they are opened they will be.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Wed Nov 30, 2005 7:58 pm    Post subject: Reply with quote

new stable version out 0.8, a TON of chances has been done, this is getting quite usable :), no known bugs.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Dec 16, 2005 4:08 pm    Post subject: Reply with quote

Any way to select the compressor yet? Also is there a way to see the compression levels of the files with the fusecompress mounted?

I'd like to see a way to possibly schedule a cron job that would compress (possibly recompress) some files using the compessor I choose. In most cases this would be the one that is most likely to produce the tightest compression.
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Fri Dec 16, 2005 4:22 pm    Post subject: Reply with quote

drescherjm wrote:
Any way to select the compressor yet? Also is there a way to see the compression levels of the files with the fusecompress mounted?


nope, it's on the todo list though. You can change it but you'll have to do it in source, it's hardcoded right now.

drescherjm wrote:
I'd like to see a way to possibly schedule a cron job that would compress (possibly recompress) some files using the compessor I choose. In most cases this would be the one that is most likely to produce the tightest compression.


Do a find /mnt/compressed -exec ls -l \{\} \; if files are STAT'ed it'll check if it's compressed or not, and if not pass it on to the background compressor. But you shouldn't have to do that except when you first mount a raw directory.
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Fri Dec 16, 2005 4:34 pm    Post subject: Reply with quote

Quote:
Do a find /mnt/compressed -exec ls -l \{\} \; if files are STAT'ed it'll check if it's compressed or not, and if not pass it on to the background compressor. But you shouldn't have to do that except when you first mount a raw directory.


At first I was going to say what's that? But I figured it out thanks. :D
_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
drescherjm
Advocate
Advocate


Joined: 05 Jun 2004
Posts: 2768
Location: Pittsburgh, PA, USA

PostPosted: Tue Jan 10, 2006 4:02 pm    Post subject: Reply with quote

Please add log.h to the repository when you get a chance. Sorry if this was too quick after the latest update...

Code:
# make
cc -c -g -D_REENTRANT -O3 -D_FILE_OFFSET_BITS=64 -Wall background_compress.c
background_compress.c:37:17: log.h: No such file or directory
background_compress.c: In function `background_compress':
background_compress.c:50: warning: implicit declaration of function `STAT_'
background_compress.c:50: error: `STAT_BACKGROUND_COMPRESS_QUEUE' undeclared (first use in this function)
background_compress.c:50: error: (Each undeclared identifier is reported only once
background_compress.c:50: error: for each function it appears in.)
background_compress.c:58: warning: implicit declaration of function `DEBUG_'
background_compress.c: In function `do_compress':
background_compress.c:104: warning: implicit declaration of function `CRIT_'
background_compress.c:112: warning: implicit declaration of function `WARN_'
background_compress.c:159: error: `STAT_BACKGROUND_COMPRESS' undeclared (first use in this function)
make: *** [background_compress] Error 1

_________________
John

My gentoo overlay
Instructons for overlay
Back to top
View user's profile Send private message
mauricev
Tux's lil' helper
Tux's lil' helper


Joined: 22 Mar 2004
Posts: 107

PostPosted: Mon Nov 26, 2007 7:45 am    Post subject: Reply with quote

Seems that this software is in active development, with a new version released just 2 weeks ago. Is anyone using it? If my understanding is correct, it's an overlay, so that means it doesn't actually implement a filesystem, but overlays an existing one. It's not clear if it supports all the common Linux filesystems (e.g., ext3, reiserfs, jfs, xfs) or not.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Mon Nov 26, 2007 7:57 am    Post subject: Reply with quote

mauricev wrote:
Seems that this software is in active development, with a new version released just 2 weeks ago. Is anyone using it? If my understanding is correct, it's an overlay, so that means it doesn't actually implement a filesystem, but overlays an existing one. It's not clear if it supports all the common Linux filesystems (e.g., ext3, reiserfs, jfs, xfs) or not.


It does work, and it is an overlay, so it'll work with any (working :P ) underlaying filesystem. Works very well with reiserfs and tail packing, since compressed a lot of files will be smaller than 4kb (blocksize). On ext3 files under 4kb will still take 4kb.
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5737
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun May 20, 2012 3:26 pm    Post subject: Reply with quote

dig dig dig :P


anyone uses this filesystem ?

if yes - from what overlay ?



as a sideeffect that I currently can't buy bigger harddrives I'm forced to get a filesystem with reasonable performance and compression (preferrably lzma)

there's version 2.6 in the zugaina overlay (http://gpo.zugaina.org/sys-fs/fusecompress/ChangeLog)

but that looks rather old when comparing with 0.9.x (http://code.google.com/p/fusecompress/)




thanks !
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.3.0-r2
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sun May 20, 2012 5:48 pm    Post subject: Reply with quote

Performance wize I'd strongly recomend going btrfs+lzma instead of fusecompress. Fusecompress is going through fuse (!), and the performance is therefor limited.

(On a side note, I wrote quite a bit of patches for 0.9 dealing with performance, and I used it on the portage tree and on usb storage with great success).
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5737
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Sun May 20, 2012 6:24 pm    Post subject: Reply with quote

neuron wrote:
Performance wize I'd strongly recomend going btrfs+lzma instead of fusecompress. Fusecompress is going through fuse (!), and the performance is therefor limited.

(On a side note, I wrote quite a bit of patches for 0.9 dealing with performance, and I used it on the portage tree and on usb storage with great success).


you mean transparent lzma compression in btrfs ? this would be the optimal solution for me right now

I haven't seen any patches that would add that additional compression algorithm, discussions about lzma are going back 2-4 years


thanks !


*edit:*

I've got to backup around 1 TB through rsync so compression on a file-per-file basis (if it isn't supported by the kernel or filesystem) unfortunately isn't an option
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.3.0-r2
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum