| View previous topic :: View next topic |
| Author |
Message |
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
 Posted: Tue Feb 19, 2013 5:52 pm Post subject: Compressed r/w FS or alternatives to aufs? |
 |
|
Thunderbird mail storage directory becomes HUGE in size, most of the content being text; so I was looking forward towards a solution.
A long time ago, I compiled fusecompress, at that time it was at version 2.
Now I cant get any of available fusecompress ebuilds to compile, all cause of boost errors.
Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird.
:sight:
Are there any options remaining?
_________________
My blog |
|
| Back to top |
|
 |
666threesixes666
Veteran
Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w
|
| Posted: Tue Feb 19, 2013 5:56 pm Post subject: |
|
|
after seeing 100s of episodes of hoarders, i decided to unsubscribe to everything, start monitoring my email traffic and snap it up when it comes. i read and purge it.
did you try to unmask the very latest boost and compile that? or go back to previous versions of boost? id just start purging and unsubscribing, only keeping the very most important mail. |
|
| Back to top |
|
|
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
| Posted: Wed Feb 20, 2013 9:49 am Post subject: |
|
|
It doesn't matter. It's not about the new mails, it's about the mails I already have in the inbox and various other places.
Speaking of which, I'll see to all that.
Since this's a very much critical requirements, I'll try the new boost.
_________________
My blog |
|
| Back to top |
|
|
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
| Posted: Wed Feb 20, 2013 11:58 am Post subject: |
|
|
Upgrading boost didn't work.
I think the project is dead; it never compiles and doesn't have a homepage.
_________________
My blog |
|
| Back to top |
|
|
xming
Guru
Joined: 02 Jul 2002
Posts: 441
|
| Posted: Wed Feb 20, 2013 12:12 pm Post subject: |
|
|
btrfs?
_________________
http://wojia.be |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Wed Feb 20, 2013 1:08 pm Post subject: Re: Compressed r/w FS or alternatives to aufs? |
|
|
| dE_logics wrote: | | A long time ago, I compiled fusecompress, at that time it was at version 2. |
dE_logics ... looking at the useflags for sys-fs/fusecompress::foo-overlay it uses bzip2, lzma, lzo, and zlib for compression, so it may be the file is simply compressed with one or other of these compression types and that fusecompress then fuse mounts it ... you might run 'file' and see it this is the case, as you may then be able to access (via bzip2, xz, etc) the data without the need of fusecompress.
| dE_logics wrote: | | Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird. |
Sounds like it uses 'mbox' and that the mailboxes are compressed on the fly ... not sure why more space would be consumed by the process, other than the fact that the data is copied, it should however compact the data into whatever compression method is used once Thunderbird quits. That said, given the nature of the application it may be that deleted mail is not actually deleted but stored in a Trash mailbox that needs to be explicitly purged.
| dE_logics wrote: | | Are there any options remaining? |
Not having ever used Thunderbird I can't say, but given that mail is a standard format (mbox, or maildir, most likely) an archive should be maintainable whatever application is used. My archived mail is stored in gzip'ed mbox, with current mail in maildir, archived mail can be accessed, and appended too from within mutt with the following hooks:
| Code: | # gzip
open-hook \\.gz$ "gzip -cd %f > %t"
close-hook \\.gz$ "gzip -c %t > %f"
append-hook \\.gz$ "gzip -c %t >> %f"
# bzip2:
# no appending possible with bzip2
open-hook \\.bz2$ "bzip2 -cd %f > %t"
close-hook \\.bz2$ "bzip2 -c %t > %f" |
I've been maintaining this setup for ages, and given the compression format is standard, easily managed, and accessable outside of a mail client, its fairly future proof.
best ... khay |
|
| Back to top |
|
|
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
| Posted: Wed Feb 20, 2013 2:51 pm Post subject: Re: Compressed r/w FS or alternatives to aufs? |
|
|
| khayyam wrote: | | dE_logics wrote: | | A long time ago, I compiled fusecompress, at that time it was at version 2. |
dE_logics ... looking at the useflags for sys-fs/fusecompress::foo-overlay it uses bzip2, lzma, lzo, and zlib for compression, so it may be the file is simply compressed with one or other of these compression types and that fusecompress then fuse mounts it ... you might run 'file' and see it this is the case, as you may then be able to access (via bzip2, xz, etc) the data without the need of fusecompress.
| dE_logics wrote: | | Thunderbird stores mails in a single large text file which's the main content to be compressed; with aufs (+squashfs), if one of those files changes by a bit, the whole file is copied to the r/w location by aufs, making it effectively consume more space with every run of Thunderbird. |
Sounds like it uses 'mbox' and that the mailboxes are compressed on the fly ... not sure why more space would be consumed by the process, other than the fact that the data is copied, it should however compact the data into whatever compression method is used once Thunderbird quits. That said, given the nature of the application it may be that deleted mail is not actually deleted but stored in a Trash mailbox that needs to be explicitly purged. |
They don't appear to be compressed, it's plane text which I can read using a text editor and highly compressible, and yet it's mbox, but mbox is not a compressed format.
| khayyam wrote: |
| dE_logics wrote: | | Are there any options remaining? |
Not having ever used Thunderbird I can't say, but given that mail is a standard format (mbox, or maildir, most likely) an archive should be maintainable whatever application is used. My archived mail is stored in gzip'ed mbox, with current mail in maildir, archived mail can be accessed, and appended too from within mutt with the following hooks:
| Code: | # gzip
open-hook \\.gz$ "gzip -cd %f > %t"
close-hook \\.gz$ "gzip -c %t > %f"
append-hook \\.gz$ "gzip -c %t >> %f"
# bzip2:
# no appending possible with bzip2
open-hook \\.bz2$ "bzip2 -cd %f > %t"
close-hook \\.bz2$ "bzip2 -c %t > %f" |
I've been maintaining this setup for ages, and given the compression format is standard, easily managed, and accessable outside of a mail client, its fairly future proof.
best ... khay |
But thunderbird needs plane/abstract rw access on the FS.
_________________
My blog |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Wed Feb 20, 2013 3:49 pm Post subject: Re: Compressed r/w FS or alternatives to aufs? |
|
|
| dE_logics wrote: | | They don't appear to be compressed, it's plane text which I can read using a text editor and highly compressible, and yet it's mbox, but mbox is not a compressed format. |
dE_logics ... I had read your initial message as suggesting you were using fusecompress for mail storeage, and that as you were now unable to compile it, were unable to access the data. That's somewhat the result of your presentation ... but I think what your actually asking is how you might setup a compressed fs, or whats on offer in that regard. Anyhow, squash_dir (sys-fs/squash_dir::mv) is one possible example. Perhaps, as has been suggested, using btrfs on a loop/file might also be a possible solution, but I'm not sure how much you'll save in terms of diskspace space.
| dE_logics wrote: | | But thunderbird needs plane/abstract rw access on the FS. |
Yes, but it wasn't altogether clear from your inital post what you we're doing or trying to achieve. I posted the mutt hooks meerly as an example of how deal with it.
best ... khay |
|
| Back to top |
|
|
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
| Posted: Wed Feb 20, 2013 5:30 pm Post subject: |
|
|
What I'm trying to compress here is the ~/.thunderbird; that's exactly not a 'mail directory' as you may be thinking; it's the place where thunderbird also stores settings and all, and among that are mails in mbox format.
Also squash_dir uses some kind of union FS which has issues with my setup (as stated in the main question). BUT the RDEPEND in the squash_dir ebuild has provided hints to aufs alternatives which I'll try. Thanks for that!
_________________
My blog |
|
| Back to top |
|
|
dE_logics
Advocate
Joined: 02 Jan 2009
Posts: 2336
Location: $TERM
|
| Posted: Thu Feb 21, 2013 4:41 pm Post subject: |
|
|
None of those work as per my expectations.
In the mean time I delete those mbox files if it takes too much space.
_________________
My blog |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
