Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compressed r/w FS or alternatives to aufs?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Tue Feb 19, 2013 5:52 pm    Post subject: Compressed r/w FS or alternatives to aufs? Reply with quote

Thunderbird mail storage directory becomes HUGE in size, most of the content being text; so I was looking forward towards a solution.

A long time ago, I compiled fusecompress, at that time it was at version 2.

Now I cant get any of available fusecompress ebuilds to compile, all cause of boost errors.

Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird.

:sight:

Are there any options remaining?
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1237
Location: 42.68n 85.41w

PostPosted: Tue Feb 19, 2013 5:56 pm    Post subject: Reply with quote

after seeing 100s of episodes of hoarders, i decided to unsubscribe to everything, start monitoring my email traffic and snap it up when it comes. i read and purge it.

did you try to unmask the very latest boost and compile that? or go back to previous versions of boost? id just start purging and unsubscribing, only keeping the very most important mail.
_________________
cat /etc/*-release
Funtoo Linux - baselayout 2.2.0
consider this warning no. 1
https://wiki.gentoo.org/index.php?title=Special:Contributions/666threesixes666&offset=&limit=500&target=666threesixes666
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Wed Feb 20, 2013 9:49 am    Post subject: Reply with quote

It doesn't matter. It's not about the new mails, it's about the mails I already have in the inbox and various other places.

Speaking of which, I'll see to all that.

Since this's a very much critical requirements, I'll try the new boost.
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Wed Feb 20, 2013 11:58 am    Post subject: Reply with quote

Upgrading boost didn't work.

I think the project is dead; it never compiles and doesn't have a homepage.
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Wed Feb 20, 2013 12:12 pm    Post subject: Reply with quote

btrfs?
_________________
http://wojia.be
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2244

PostPosted: Wed Feb 20, 2013 1:08 pm    Post subject: Re: Compressed r/w FS or alternatives to aufs? Reply with quote

dE_logics wrote:
A long time ago, I compiled fusecompress, at that time it was at version 2.

dE_logics ... looking at the useflags for sys-fs/fusecompress::foo-overlay it uses bzip2, lzma, lzo, and zlib for compression, so it may be the file is simply compressed with one or other of these compression types and that fusecompress then fuse mounts it ... you might run 'file' and see it this is the case, as you may then be able to access (via bzip2, xz, etc) the data without the need of fusecompress.

dE_logics wrote:
Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird.

Sounds like it uses 'mbox' and that the mailboxes are compressed on the fly ... not sure why more space would be consumed by the process, other than the fact that the data is copied, it should however compact the data into whatever compression method is used once Thunderbird quits. That said, given the nature of the application it may be that deleted mail is not actually deleted but stored in a Trash mailbox that needs to be explicitly purged.

dE_logics wrote:
Are there any options remaining?

Not having ever used Thunderbird I can't say, but given that mail is a standard format (mbox, or maildir, most likely) an archive should be maintainable whatever application is used. My archived mail is stored in gzip'ed mbox, with current mail in maildir, archived mail can be accessed, and appended too from within mutt with the following hooks:

Code:
# gzip
open-hook \\.gz$ "gzip -cd %f > %t"
close-hook \\.gz$ "gzip -c %t > %f"
append-hook \\.gz$ "gzip -c %t >> %f"

# bzip2:
# no appending possible with bzip2
open-hook \\.bz2$ "bzip2 -cd %f > %t"
close-hook \\.bz2$ "bzip2 -c %t > %f"

I've been maintaining this setup for ages, and given the compression format is standard, easily managed, and accessable outside of a mail client, its fairly future proof.

best ... khay
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Wed Feb 20, 2013 2:51 pm    Post subject: Re: Compressed r/w FS or alternatives to aufs? Reply with quote

khayyam wrote:
dE_logics wrote:
A long time ago, I compiled fusecompress, at that time it was at version 2.

dE_logics ... looking at the useflags for sys-fs/fusecompress::foo-overlay it uses bzip2, lzma, lzo, and zlib for compression, so it may be the file is simply compressed with one or other of these compression types and that fusecompress then fuse mounts it ... you might run 'file' and see it this is the case, as you may then be able to access (via bzip2, xz, etc) the data without the need of fusecompress.

dE_logics wrote:
Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird.

Sounds like it uses 'mbox' and that the mailboxes are compressed on the fly ... not sure why more space would be consumed by the process, other than the fact that the data is copied, it should however compact the data into whatever compression method is used once Thunderbird quits. That said, given the nature of the application it may be that deleted mail is not actually deleted but stored in a Trash mailbox that needs to be explicitly purged.


They don't appear to be compressed, it's plane text which I can read using a text editor and highly compressible, and yet it's mbox, but mbox is not a compressed format.
khayyam wrote:

dE_logics wrote:
Are there any options remaining?

Not having ever used Thunderbird I can't say, but given that mail is a standard format (mbox, or maildir, most likely) an archive should be maintainable whatever application is used. My archived mail is stored in gzip'ed mbox, with current mail in maildir, archived mail can be accessed, and appended too from within mutt with the following hooks:

Code:
# gzip
open-hook \\.gz$ "gzip -cd %f > %t"
close-hook \\.gz$ "gzip -c %t > %f"
append-hook \\.gz$ "gzip -c %t >> %f"

# bzip2:
# no appending possible with bzip2
open-hook \\.bz2$ "bzip2 -cd %f > %t"
close-hook \\.bz2$ "bzip2 -c %t > %f"

I've been maintaining this setup for ages, and given the compression format is standard, easily managed, and accessable outside of a mail client, its fairly future proof.

best ... khay


But thunderbird needs plane/abstract rw access on the FS.
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
khayyam
Advocate
Advocate


Joined: 07 Jun 2012
Posts: 2244

PostPosted: Wed Feb 20, 2013 3:49 pm    Post subject: Re: Compressed r/w FS or alternatives to aufs? Reply with quote

dE_logics wrote:
They don't appear to be compressed, it's plane text which I can read using a text editor and highly compressible, and yet it's mbox, but mbox is not a compressed format.

dE_logics ... I had read your initial message as suggesting you were using fusecompress for mail storeage, and that as you were now unable to compile it, were unable to access the data. That's somewhat the result of your presentation ... but I think what your actually asking is how you might setup a compressed fs, or whats on offer in that regard. Anyhow, squash_dir (sys-fs/squash_dir::mv) is one possible example. Perhaps, as has been suggested, using btrfs on a loop/file might also be a possible solution, but I'm not sure how much you'll save in terms of diskspace space.

dE_logics wrote:
But thunderbird needs plane/abstract rw access on the FS.

Yes, but it wasn't altogether clear from your inital post what you we're doing or trying to achieve. I posted the mutt hooks meerly as an example of how deal with it.

best ... khay
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Wed Feb 20, 2013 5:30 pm    Post subject: Reply with quote

What I'm trying to compress here is the ~/.thunderbird; that's exactly not a 'mail directory' as you may be thinking; it's the place where thunderbird also stores settings and all, and among that are mails in mbox format.

Also squash_dir uses some kind of union FS which has issues with my setup (as stated in the main question). BUT the RDEPEND in the squash_dir ebuild has provided hints to aufs alternatives which I'll try. Thanks for that!
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
dE_logics
Advocate
Advocate


Joined: 02 Jan 2009
Posts: 2175
Location: $TERM

PostPosted: Thu Feb 21, 2013 4:41 pm    Post subject: Reply with quote

None of those work as per my expectations.

In the mean time I delete those mbox files if it takes too much space.
_________________
Buy from companies supporting opensource -- IBM, Dell, HP, Hitachi, Google etc...
Disfavor companies supporting only Win -- Logitech, Epson, Adobe, Autodesk, Pioneer, Kingston, WD, Yahoo, MSI, XFX
My blog
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum