Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 12, 13, 14, 15  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
lost+found
Guru
Guru


Joined: 15 Nov 2004
Posts: 429
Location: North~Sea~Coa~s~~t~~~

PostPosted: Tue Jul 29, 2014 6:24 am    Post subject: Reply with quote

I can recommend using unionfs-fuse with gzip, if file size doesn't matter. It has proven to be fast and reliable to me. I'm using squashmount mainly to speed up emerge/portage/kernel source operations on JFS filesystems. I think it prevents file system fragmentation too. A very useful tool indeed!
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Tue Jul 29, 2014 7:09 am    Post subject: Reply with quote

costel78 wrote:
It still failed to umount

But I hope, not because of complaining about a problem with /sbin/umount.aufs; it would be a bug in either squashmount or umount (or a typo in your configuration...) if this tool would have been called.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Tue Jul 29, 2014 7:21 am    Post subject: Reply with quote

lost+found wrote:
I can recommend using unionfs-fuse with gzip, if file size doesn't matter

If file size does not matter, you should use lz4: You get an enormous speed increase, see the file compression.txt in current versions of squashmount.
However, be aware that this cannot be used with <linux-3.16; I hope that it will be included in linux-3.16, finally.

When a kernel supporting it is out for quite while, it might become default method for squashmount. (Take this a warning for those who need to use older kernels: For new releases, be aware to read the ChangeLog if you need to set something)
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 208

PostPosted: Tue Jul 29, 2014 7:23 pm    Post subject: Reply with quote

mv wrote:
costel78 wrote:
It still failed to umount

But I hope, not because of complaining about a problem with /sbin/umount.aufs; it would be a bug in either squashmount or umount (or a typo in your configuration...) if this tool would have been called.

Yes, exactly same error. Maybe config file is wrong ?
Code:
#!/usr/bin/perl (this is only for editors)

# The tools which we have installed; if possible only the first in this list
# is used, but the others are a fallback if that fails.

@order = ('aufs', 'overlayfs', 'unionfs-fuse', 'unionfs', 'funionfs');

# Even if we define following is empty it is convenient to use
# this local variable throughout, so that we can simply change it:

my $defaults = {
   COMPRESSION => 'xz'
};

push(@mounts, {
   TAG => 'portage',
   DIR => '/usr/portage',
   FILE => '/usr/portage.sqfs',
   CHANGES => '/usr/portage.changes',
   READONLY => '/usr/portage.readonly',
   THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}, {
   TAG => 'db',
   DIR => '/var/db',
   FILE => '/var/db.sqfs',
   CHANGES => '/var/db.changes',
   READONLY => '/var/db.readonly',
   THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}

);

@umount = ('-i');


Anyway, manually using squashfs and aufs result in the same result with laters aufs-utils.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Wed Jul 30, 2014 8:28 am    Post subject: Reply with quote

There's a bug in this new feature of squashmount :oops:

@umount and @umount_ro are used oppositely as intended/documented in 7.8 and 7.9. This is fixed in 7.10.
Thus (temporarily for testing until you install 7.10) you should replace @umount by @umount_ro.
Quote:
Anyway, manually using squashfs and aufs result in the same result with laters aufs-utils

Did you use option -i with umount when doing things manually?
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 208

PostPosted: Wed Jul 30, 2014 11:28 am    Post subject: Reply with quote

Code:
[ebuild   R    ] sys-fs/aufs-headers-3.15_p20140728  0 kB
[ebuild   R    ] sys-fs/aufs-util-3.15_p20140728  0 kB
[ebuild   R    ] sys-fs/squashmount-7.10::added  0 kB


Everything is working just fine. :)
Manually, error occurred without -i. With it set, it worked.

Thank you very much!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Fri Aug 15, 2014 6:15 am    Post subject: Reply with quote

It feels like since the latest update
Code:
Wed Aug 13 12:04:32 2014 >>> sys-fs/aufs3-3_p20140811
the mounting takes very long. It does not fail but takes around 20 seconds. That makes booting up take very long for around 10 squash directories, while there are no errors in logfiles. Is there any verbose mode?
Code:
$ squash_dir start portage
Starting portage...
 * Mounting /usr/portage.sqfs as /usr/portage ...                                                   [ ok ]
Code:
$ mount |grep "/usr/portage"
/usr/portage.sqfs on /usr/portage.readonly type squashfs (ro,noatime)
aufs on /usr/portage type aufs (rw,noatime,si=782a68c3173c92b0)
Yes, I'm still using squash_dir.

While there was no failure now for a long time but I remember hassle when forced to use a boot-cd to re-emerge some things, is there some emergency start of squash_dir or your new squashmount, that usually is able to mount all directories on well stuffed live-cd environment from inside Gentoo-CHROOT?
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Fri Aug 15, 2014 7:48 am    Post subject: Reply with quote

Massimo B. wrote:
It feels like since the latest update
Code:
Wed Aug 13 12:04:32 2014 >>> sys-fs/aufs3-3_p20140811
the mounting takes very long.

Since aufs3 was lagging behind for a while and perhaps only overlayfs will go into upstream kernel, I have currently only kernels with overlayfs for testing.
(However, squash_dir currently does not support latest changes in overlayfs, and unless someone sends patches, I will not update it anymore - squash_dir is now really abandoned...)
How does it work if you send the commands manually?
Something like
Code:
modprobe squashfs
mount -t squashfs -o loop,ro,noatime -- /path/to/SQUASHFILE /path/to/READONLY
modprobe aufs3
mount -t aufs [your options from MOUNT_AUFS] -o noatime -o br:/path/to/CHANGES=rw:/path/to/READONLY=rr -- aufs /path/to/DIRECTORY

Probably, you want to use -v as an additional option to the last mount command.
If the latter command really just takes 20 seconds and writes nothing out (into systemlog or stdout/stderr), I am afraid that I cannot help: You have to ask the aufs author.
Quote:
Is there any verbose mode?

Not in squash_dir. In squashmount, there is: If you specify -v sufficiently often, you will see the above commands. But this won't help either, if these commands do not produce any usable output...
Quote:
is there some emergency start of squash_dir or your new squashmount, that usually is able to mount all directories on well stuffed live-cd environment from inside Gentoo-CHROOT?

No. If you hav a suggestion for a corresponding configuration file for squashmount (or perhaps need some functionality added for this), please send it per PM or report a bug on the squashmount bugtracker on github (or directly send a pull request if you have patches ready :wink: )
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Sat Aug 16, 2014 4:03 pm    Post subject: Reply with quote

Due to the recent discussion, squashmount has now obtained some enhanced configuration possibilities (e.g. suitable for non-permanent mount-points like for CDs). :wink:
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Wed Aug 20, 2014 11:21 am    Post subject: Reply with quote

Ok, I started migration to squashmount:
It seems /etc/squashmount.pl is the one and only configuration. Please make the config file configurable or distribute the example squashmount.pl additionally in /usr/share/doc/squashmount*.
Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes. You could also place some /etc/squashmount.pl.example beside. Now that I compressed my final version of config I lost the useful comments.
As you mention there "do not use it unchanged" is a but unusual, so it would better to start with a working minimal example and the rest commented... but ok, let's start:

My compressed version:
Code:
use Sys::Hostname;
my $hostname = ($ENV{'HOSTNAME'} // hostname());

@order = qw(overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);

my $defaults = {
   COMPRESSION => 'lz4',
   COMPOPT_LZ4 => '',
};
my $non_binary = {
   COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};

@mounts = (
   standard_mount('adobe',            '/opt/Adobe',         $defaults),
   standard_mount('firefox',         '/usr/lib/firefox',      $defaults),
   standard_mount('icedtea6',         '/usr/lib/icedtea6',   $defaults),
   standard_mount('icedtea7',         '/usr/lib/icedtea7',   $defaults),
   standard_mount('layman',         '/var/lib/layman',      $defaults),
   standard_mount('libreoffice',      '/usr/lib/libreoffice',   $defaults),
   standard_mount('local_portage',      '/usr/local/portage',   $defaults),
   standard_mount('vmware',         '/opt/vmware',         $defaults),
    standard_mount('db',            '/var/db',            $defaults),

   standard_mount('tex', '/usr/share/texmf-dist', $defaults, $non_binary, {
      DIFF => [
         qr{^ls-R$},
         qr{^tex(?:/generic(?:/config(?:/language(?:\.(?:dat(?:\.lua)?|def)))?)?)?$}
      ]
   }),
   standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
      UMOUNT => ((@umount) ? undef : '-i'),
      THRESHOLD => '80m',
      FILL => qr{^local/(?!(?:\.git|profiles|metadata)(?:/|$))}
   })
);

'EOF';

I disabled the old squash_dir setup by copying all squashed data back, so starting from scratch. But squashmount start fails for all items:
Code:
$ squashmount start -v
 * [adobe]:         It seems this is mounted for the first time:
                    The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
                    it will be initialized now from /opt/Adobe
Could not create destination file: No such file or directory
 * [adobe]:         error:   failed: /usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp lz4

PS.: It seems that app-arch/lz4 is there but Kernel support was missing. Now I have lz4 modules loaded but start is still failing for mksquashfs.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Wed Aug 20, 2014 5:50 pm    Post subject: Reply with quote

Massimo B. wrote:
Please make the config file configurable

I do not know what you mean by that. There are options to add additional config files, but they cannot be used reasonably by the default /etc/init.d files (or the default systemd units, respectively).
Quote:
Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes.

Yes, but this is normal for complex configuration files: You have similar issues with cups, wwwoffle, ssh, mtools, tor, privoxy, nvidia, ... even for some files in /etc/cron.*, /etc/modules.load.d, /etc/logrotate.d
Quote:
You could also place some /etc/squashmount.pl.example beside.

You can just make a copy for yourself.
There is now a USE=example which installs into /etc/squashmount-example.pl, but I am not sure whether this is a good idea. Maybe this will be removed, again.
Quote:
COMPRESSION => 'lz4'

Unless you patched the kernel manually, you will not be happy with that: Mounting will fail...
[quote]error: failed: /usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp lz4[/code]
Ah, I see: This command is "correct", but it fails because /opt/Adobe.mount does not exist. Of course, such things usually happen only on the first installation and thus are not found in a "normal" debugging procedure.
squashmount-8.4 now generates the corresponding parent files in advance.
Quote:
It seems that app-arch/lz4 is there but Kernel support was missing.

That's the problem: You can squash and (manually) unsquash with lz4, but you cannot use lz4 "regularly" unless you patch the kernel.
The kernel patch exists since almost a year (since lz4 is in the kernel), and it is only a few lines, but some kernel developer refused it with the stupid claim that it be not necessary that squashfs can compress quickly. I hope that the patch will be included, finally, but I am not very optimistic, meanwhile.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Thu Aug 21, 2014 9:11 am    Post subject: Reply with quote

mv wrote:
Massimo B. wrote:
Please make the config file configurable
I do not know what you mean by that.

I meant an option to choose the config file, maybe via /etc/conf.d/squashmount. But maybe this is not the best idea either. For now I need to copy a new /etc/squashmount.pl behind of etc-update to keep my config and still get some new examples.

mv wrote:
Quote:
Because lot of the example is enabled which I don't need, it is harder to keep the example and merging new changes.
Yes, but this is normal for complex configuration files: You have similar issues with cups, wwwoffle, ssh, mtools, tor, privoxy, nvidia, ... even for some files in /etc/cron.*, /etc/modules.load.d, /etc/logrotate.d

I know. But the thing with squashmount.pl is a bit different. The default and almost commented configurations like dnsmasq, privoxy etc. are working like this. But the squashmount.pl is absolutely required to be edited for not breaking things (it would start to squash otherwise..).
Then as I mentioned some time ago it feels a bit unusual to have real Perl code as config. Even some Perl colleagues on #perl have been surprised. But I see it is very powerful but also dangerous. No config will be comparable to others as Perl code can have very different styles. But leave it like this, it is powerful and I don't have a different idea..

mv wrote:
There is now a USE=example which installs into /etc/squashmount-example.pl, but I am not sure whether this is a good idea. Maybe this will be removed, again.

Maybe just leave it in /usr/share/doc/squashmount-*.

mv wrote:
Unless you patched the kernel manually, you will not be happy with that: Mounting will fail...

I don't prefer too much kernel patching as there already are some patches I'm using, like sys-fs/aufs3. Then I'm using sys-kernel/ck-sources. Is there an ebuild to do the patching? I'm quite interested in plain lz4 after reading your compress.txt benchmarks. Performance is more important than size. Even on recent platforms Portage is still very slow due to its architecture and physical drives (no SSD).
BTW, how did you measure the time, like 4 (lz4) vs. 3:50 (xz) for /var/db on Core2?

So for testing I switched all to old gzip now:
mv wrote:
Ah, I see: This command is "correct", but it fails because /opt/Adobe.mount does not exist. Of course, such things usually happen only on the first installation and thus are not found in a "normal" debugging procedure.
squashmount-8.4 now generates the corresponding parent files in advance.

Thanks for the bugfix, pulled and working better, but still not successful. Starting from scratch with only one item:
Code:
$ squashmount -vvv start
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]: It seems this is mounted for the first time:
            The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
            it will be initialized now from /opt/Adobe
/usr/bin/mksquashfs /opt/Adobe /opt/Adobe.mount/Adobe.sfs -noappend -quiet -comp gzip
[============================================================================================================================================/] 1285/1285 100%
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /root/tmp/EgZI8WpycJ
/bin/umount -- /root/tmp/EgZI8WpycJ
 * [adobe]: cleaning original DIR
 * [adobe]: cleaning /opt/Adobe
 * [adobe]: mounting...
 * [adobe]: error:   no directory /opt/Adobe.mount/readonly

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Thu Aug 21, 2014 12:09 pm    Post subject: Reply with quote

Massimo B. wrote:
I meant an option to choose the config file, maybe via /etc/conf.d/squashmount.

Installing a config-file to get the location of the config-file? This sounds very wrong. Moreover, for systemd you would have to modify the unit, if you want to have this (unless you make the unit very non-systemd-ish). But of course, nobody prevents you from adding the option to your path to the systemd unit or to the init-file; such customizations are nothing which should be covered by an ebuild.
Quote:
But the thing with squashmount.pl is a bit different. The default and almost commented configurations like dnsmasq, privoxy etc. are working like this. But the squashmount.pl is absolutely required to be edited for not breaking things (it would start to squash otherwise..).

No, dnsmasq, privoxy, tor etc will not do anything useful if you do not edit the config first (sshd will, but in a rather insecure way, so leaving the default is no good idea, either), mtools will even just print an error message (at least, it used to - not checked recently). Also squashmount will just print an error message and not start squashing. So I do not see much difference.
Quote:
Then as I mentioned some time ago it feels a bit unusual to have real Perl code as config. Even some Perl colleagues on #perl have been surprised.

Sourcing in a separate namespace is one of the official perl recommendations for config files for perl-projects. (I read it at several places, among them some FAQ about why there is no analogue for python's ConfigParser.)
Also, it seems very clean and not hackish to me. It would be different, if the sourcing would happen in the main:: namespace (or if you need compatibility with a non-perl-tool, of course).
Also /etc/conf.d is arbitrary shell code, and I consider this as one of the main advantages of openrc, so that you can have the identical configurations for several machines and different situations - the configuration itself being conditional with user-defined conditions. (Also systemd provides such conditions, but they are much more limited and cannot be used everywhere).
Just now when the wish for a runtime-defined path (for mounting a CD into a chroot) was mentioed, I see that this decision is absolutely correct.
Quote:
No config will be comparable to others as Perl code can have very different styles.

It is not necessary to compare configs. If currently chosen data must be mailed, you can use
Code:
squashmount -vv list

Quote:
Maybe just leave it in /usr/share/doc/squashmount-*.

At first, I also thought that this is the better idea, but then you would not see changes (not only etc-update will fail, but the previous config-file is not there at all).
Quote:
Is there an ebuild to do the patching?

I don't know, and I will not spent my time to manage one. However, once that patch is in upstream kernel (or at least gentoo-sources or hardened-souces), probably the defaults in squashmount will change to lz4; this was announced here already several times...
Quote:
Performance is more important than size.

Note that these benchmarks only involve the compression speed. Benchmarking decompression speed is not so easy, and this is what counts during using such a filesystem. I would guess that concerning decompression speed, the difference is not so much. Actually, xz might be superior here, since a shorter size means less disk acces and better caching possibilities.
Quote:
BTW, how did you measure the time, like 4 (lz4) vs. 3:50 (xz) for /var/db on Core2?

Not too much effort was spent to make it very reliable: After verifying that no cron job will happen, essentially just
Code:
time mksquash /var/db ...

on the corresponding system, usually averaged over some calls to eliminate effects of caching/daemons. It was mainly to get an idea which options to use as default and whether perhaps on some systems there is a different order of the algorithms concerning speed (it seems not). However, once the list existed, why not make it available for eveybody.
Quote:
Starting from scratch with only one item:

I did the same for testing, and it worked flawlessly. The only thing which I can imagine that might have happened in your case is that you once mounted (or almost mounted) and then removed the */readonly directory again: In this case, squashmount might have stored the existence of that directory for the mount-point adobe in /run/squashmount and thus did not check its existence again. Perhaps there should be a (usually redundant) check in squashmount for such a case.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Thu Aug 21, 2014 12:27 pm    Post subject: Reply with quote

mv wrote:
Also /etc/conf.d is arbitrary shell code, and I consider this as one of the main advantages of openrc, so that you can have the identical configurations for several machines and different situations - the configuration itself being conditional with user-defined conditions. (Also systemd provides such conditions, but they are much more limited and cannot be used everywhere).
OT: That I agree, I definitly like having the same configurations over my machines to be able to merge changes. Currently I only have that for my large /etc/vim/vimrc.local with hostname conditions. But how do you do these conditions for other conf.d stuff?

Hm, again I start from scratch. The caching of old config data should be reset by squashmount reset, no? Where are they stored and why? The .new is just my backup:
Code:
$ ls -ald /opt/Adobe*
drwxr-xr-x 3 root root 4,0K Aug 21 09:37 /opt/Adobe/
drwxr-xr-x 3 root root 4,0K Aug 21 09:37 /opt/Adobe.new/

$ squashmount -vv list                                  ## I like that
 * [adobe]: not mounted
            DIR: /opt/Adobe
            READONLY: /opt/Adobe.mount/readonly
            CHANGES: /opt/Adobe.mount/changes
            WORKDIR: /opt/Adobe.mount/workdir
            FILE: /opt/Adobe.mount/Adobe.sfs
            mksquashfs options: -noappend -quiet -comp gzip
            CHMOD: 0644
            CHOWN: unchanged (0:0)

$ squashmount reset
 * [adobe]: resetting configuration

$ squashmount start
 * [adobe]: It seems this is mounted for the first time:
            The squashed file /opt/Adobe.mount/Adobe.sfs does not exist yet;
            it will be initialized now from /opt/Adobe
[=======================================================================================/] 1285/1285 100%
 * [adobe]: cleaning original DIR
 * [adobe]: mounting...
 * [adobe]: error:   no directory /opt/Adobe.mount/readonly

What I'm doing wrong here?
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Thu Aug 21, 2014 12:59 pm    Post subject: Reply with quote

As for the benchmarks, here is a decompression only benchmark. However in times when there is enough CPU but the physical drives are still slow, it could be a completely different result in real-life. I guess I will check with some Portage scenarios and different compressions. Speeding up portage is highest priority. I still remember best results with /var/db on some ancient reiserfs.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Thu Aug 21, 2014 1:37 pm    Post subject: Reply with quote

Massimo B. wrote:
The caching of old config data should be reset by squashmount reset, no?

No: You are looking for "squashmount forget". "squashmount reset" essentially just "undo's" the effect of some "squsahmount set" - in your current stage, this is irrelevant.
Quote:
Where are they stored and why?

They are stored in /run/squashmount/adobe (in your case), but do not rely on that for future version of squashmount.
"squashmount forget" will remove this file if it is safe to do this.
They must be stored, because you might use a temporary directory (with a random, i.e. non-predictable name): once that directory is initialized, it should be reused, of course.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Thu Aug 21, 2014 3:08 pm    Post subject: Reply with quote

In squashmount-8.5 a nonexistent stored directory is now considered as non-stored, that is it becomes recreated (possibly under a new temporary name): This is secure and is probably what users expect to happen in such a case.
In particular, this should fix the previous issue where "invalid" directories were stored "by mistake".
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Thu Aug 21, 2014 7:31 pm    Post subject: Reply with quote

Oh, I missed this question:
Massimo B. wrote:
But how do you do these conditions for other conf.d stuff?

As I said, it is just usual (POSIX) shell code. For instance, I have in my net configuration stuff like
Code:
if ! test -r /etc/firewall.d/dhcp-client
then
   config_lan0="...."
fi
...
if test -f /etc/ppp/my-username
then
   read username_ppp0 </etc/ppp/my-username
fi
...

Here, I use special "magic" files to switch between certain configuration settings or to keep my "secrets" off the configuration, respectively.
Of course, you can also make conditions based on hostname or on the content of a single file or whatever is appropriate for you to distinguish your configurations.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Thu Aug 21, 2014 7:37 pm    Post subject: Reply with quote

Massimo B. wrote:
As for the benchmarks, here is a decompression only benchmark

Such benchmarks have to be read with care: They cannot measure the actual disk access needed for squashfs. In fact, the latter can heavily vary with the type of directory you are using with squashfs and, even more, depends on your RAM and cache size, especially if you use memory (due to cache) hungry tools like portage.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Fri Aug 22, 2014 8:38 am    Post subject: Reply with quote

mv wrote:
"squashmount forget" will remove this file if it is safe to do this.
They must be stored, because you might use a temporary directory (with a random, i.e. non-predictable name): once that directory is initialized, it should be reused, of course.

I don't understand that exactly, what is a temporary directory with a non-predictable name? Target dirs such as /opt/Adobe or /usr/portage won't change. Then I see that squashmount stop also runs a forget, so the cached information is only while squashmount is up, but with all squashfs mounted the directories won't change at all in the background, no?

It is still not working:
Code:
$ squashmount forget
 * [adobe]: forgetting settings

$ squashmount -vvv start
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]: mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly
/sbin/modprobe overlayfs
modprobe: FATAL: Module overlayfs not found.
/bin/mount -t overlayfs -o noatime -o 'upperdir=/opt/Adobe.mount/changes' -o 'lowerdir=/opt/Adobe.mount/readonly' -o 'workdir=/opt/Adobe.mount/workdir' -- overlayfs /opt/Adobe
mount: unknown filesystem type 'overlayfs'
 * [adobe]: warning: overlayfs failed
/sbin/modprobe aufs
/bin/mount -t aufs -o noatime -o 'br:/opt/Adobe.mount/changes=rw:/opt/Adobe.mount/readonly=rr' -- aufs /opt/Adobe

$ squashmount -vv list
 * [adobe]: aufs
            unmodified
            THRESHOLD: 0
            DIR: /opt/Adobe
            READONLY: /opt/Adobe.mount/readonly
            CHANGES: /opt/Adobe.mount/changes
            FILE: /opt/Adobe.mount/Adobe.sfs
            mksquashfs options: -noappend -quiet -comp gzip
            CHMOD: 0644
            CHOWN: unchanged (0:0)
It tries to use overlayfs (how do I get that, can't find in the kernel, some secret patches too?). But I still have aufs3 working. squashmount tries overlayfs anyway and fails.

BTW. sometime in the future we could start a squashmount only thread and link here. This thread is living since the original approach, then came squash_dir, and now squashmount. On a dedicated squashmount thread you could maintain the first #1 post to have a recent overwiew about the project.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme


Last edited by Massimo B. on Fri Aug 22, 2014 8:43 am; edited 1 time in total
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Fri Aug 22, 2014 8:40 am    Post subject: Reply with quote

mv wrote:
They cannot measure the actual disk access needed for squashfs. In fact, the latter can heavily vary with the type of directory you are using with squashfs and, even more, depends on your RAM and cache size, especially if you use memory (due to cache) hungry tools like portage.
Therefore I'm going to check time for some portage scenario. After re-compressing with a different COMPRESSION, disk caching should not affect the result.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4349

PostPosted: Fri Aug 22, 2014 10:15 am    Post subject: Reply with quote

Massimo B. wrote:
I don't understand that exactly, what is a temporary directory with a non-predictable name?

For example, you could have used File::Temp::newdir() to get the name originally. Or you could have used an option like -a to set the name once (see the details in the ADVANCED CONFIGURATION section): It is intentional that the name does not change "on the fly".
Quote:
Then I see that squashmount stop also runs a forget, so the cached information is only while squashmount is up

Yes, that's the idea. The content of /run is lost anyway once you boot new (which is normally the period in which squashmount is up; in most cases when you call squashmount manually you should use "squashmount mount/remount/umount" and not "squasmount start/restart/stop").
Since squashmount does not rely on the init system to manage the mounted directories, it manages them in /run/squashmount. It is necessary to manage also half-mounted states, e,g, if umount of DIR succeeds but the subsequent umount of READONLLY fails etc.

In particular, squashmount stores directories like READONLY or CHANGES in /run only after it created them.
So it seemed safe to rely that they exist when these directories are stored.
But you proved that this is not safe: What happened in your case was presumably that you called "squashmount start" which returned with some failure, but only after it created the READONLY and CHANGES directories and stored that information. Then you did remove this directory, making the stored information "false".
In squashmount-8.5 this is "fixed" by verifying that the stored direectories actually exist, and if they do not to consider the stored information as invalid (hence act, as if that information is not stored, that is, in your case, create the directory). In a sense, due to this fix, squashmount thus now wants to be smarter than the user, but I think in this case this can do no harm.
Quote:
squashmount tries overlayfs anyway and fails.

No, squashmount did not fail: It mounted succesfully with aufs and returned succesfully.

I suppose that IO-Uncompress-Gunzip is installed, but your /proc/config.gz is not readable, since otherwise squashmount would have scanned that file to find that overlayfs is not configured for your kernel. (If this is not the case, you have found a bug in squashmount, and in this case please inform me, whether /proc/config.gz is readable and/or IO-Uncompress-Gunzip is installed).

This is the documented beviour of the default @order=qw(overlayfs? aufs! ...):
It first modprobes overlayfs, and if that fails, it checks for overlay in /proc/config.gz, assuming a positive result if the latter check fails. If you want to assume a negative result if the latter check fails, you have to use ?? instead of ?, i.e.
Code:
$order[0]='overlayfs??';

(or e.g. shift @order if you want to skip completely all checks for overlayfs and immediately try aufs).
Quote:
sometime in the future we could start a squashmount only thread and link here.

A few users will miss the new thread, and then there will actually be two threads... :(
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Fri Aug 22, 2014 12:24 pm    Post subject: Reply with quote

mv wrote:
I suppose that IO-Uncompress-Gunzip is installed, but your /proc/config.gz is not readable, since otherwise squashmount would have scanned that file to find that overlayfs is not configured for your kernel. (If this is not the case, you have found a bug in squashmount, and in this case please inform me, whether /proc/config.gz is readable and/or IO-Uncompress-Gunzip is installed).
What is IO-Uncompress-Gunzip? Kernel option? However my config.gz is readable:
Code:
$ zgrep GZIP /proc/config.gz
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_KERNEL_GZIP=y
CONFIG_RD_GZIP=y
CONFIG_DECOMPRESS_GZIP=y

$ ls -al /proc/config.gz
-r--r--r-- 1 root root 19K Aug 22 14:24 /proc/config.gz
Maybe it is not up-to-date, because I added some modules later without restart?

Weird, squashfs is mounted quickly, but as I mentioned some time ago, aufs takes very long, ~20s. For 10 mounts this takes very long at boot.
What is the preferred backend setup for squashmount with the optimal that is only few kernel patches?
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1230
Location: PB, Germany

PostPosted: Fri Aug 22, 2014 1:41 pm    Post subject: Reply with quote

mv wrote:
With squashmount you can do it on the fly:
Code:
squashmount -fsx xz remount
BTW. I tried recompressing to test some COMPRESSIONs, but there is no recompression done:
Code:
$ squashmount -fsx xz remount
 * [adobe]: umounting...
 * [adobe]: cleaning changes...
 * [adobe]: mounting...

$ squashmount -fsx gzip remount
 * [adobe]: umounting...
 * [adobe]: cleaning changes...
 * [adobe]: mounting...
Same if changing the squashmount.pl config and forcing remount.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
Lomaxx
Tux's lil' helper
Tux's lil' helper


Joined: 11 Nov 2004
Posts: 112

PostPosted: Sat Aug 23, 2014 10:16 am    Post subject: Reply with quote

Hello.

I've been using squashmount for quite some while by now. Since i did a world-update (including kernel-update) yesterday, i first broke my aufs-support, then repaired it and now have got it working again, BUT with an annoying delay while booting that I didn't experience before. The delay is altogether roughly 5-10 seconds long for 3 squashed directories (portage/db/layman), i do not receive any errors while booting and i wonder what might be causing it. Before there was no delay at all.
If someone got a close guess it would help me somewhat. Otherwise I have to try to solve it by trial and error.

Some info on my system:

- sys-kernel/aufs-sources-3.16.1
- sys-fs/aufs-headers-3.16_p20140818:0
- sys-fs/aufs-util-3.16_p20140818:0
- sys-fs/squashfs-tools-4.2-r1:0
- sys-fs/squashmount-6.0a:0

Kernel-Config regarding aufs:

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
# CONFIG_AUFS_EXPORT is not set
# CONFIG_AUFS_FHSM is not set
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

extracts from squashmount.pl:

#!/usr/bin/perl (this is only for editors)
@order = qw(aufs! unionfs-fuse! unionfs??# funionfs??#);
my $defaults = {
COMPRESSION => 'xz', # We could omit this line as xz is default.
# However, this might change in the future
COMPOPT_LZ4 => '-Xhc', # We could omit this line as -Xhc is default
COMPOPT_XZ => ['-Xbcj', 'x86'] # Use this in case COMPRESSION => 'xz'
};
my $non_binary = {
COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};

@mounts = (
added_hash($defaults, $non_binary, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),
added_hash($defaults, $non_binary, {
TAG => 'layman',
DIR => '/var/lib/layman',
FILE => '/var/lib/layman.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/lib/layman.changes',
READONLY => '/var/lib/.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),

standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
THRESHOLD => '80m',
FILL => qr{^local/(?!(\.git|profiles|metadata)(/|$))}
}),
);

As a solution I thought about rebuilding the directories freshly or enabling the userspace-option in the kernel, but haven't tried that yet. Any help would be appreciated.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 12, 13, 14, 15  Next
Page 13 of 15

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum