Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compressing filesystems with squashfs and squashmount
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Fri Aug 22, 2014 8:40 am    Post subject: Reply with quote

mv wrote:
They cannot measure the actual disk access needed for squashfs. In fact, the latter can heavily vary with the type of directory you are using with squashfs and, even more, depends on your RAM and cache size, especially if you use memory (due to cache) hungry tools like portage.
Therefore I'm going to check time for some portage scenario. After re-compressing with a different COMPRESSION, disk caching should not affect the result.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Fri Aug 22, 2014 10:15 am    Post subject: Reply with quote

Massimo B. wrote:
I don't understand that exactly, what is a temporary directory with a non-predictable name?

For example, you could have used File::Temp::newdir() to get the name originally. Or you could have used an option like -a to set the name once (see the details in the ADVANCED CONFIGURATION section): It is intentional that the name does not change "on the fly".
Quote:
Then I see that squashmount stop also runs a forget, so the cached information is only while squashmount is up

Yes, that's the idea. The content of /run is lost anyway once you boot new (which is normally the period in which squashmount is up; in most cases when you call squashmount manually you should use "squashmount mount/remount/umount" and not "squasmount start/restart/stop").
Since squashmount does not rely on the init system to manage the mounted directories, it manages them in /run/squashmount. It is necessary to manage also half-mounted states, e,g, if umount of DIR succeeds but the subsequent umount of READONLLY fails etc.

In particular, squashmount stores directories like READONLY or CHANGES in /run only after it created them.
So it seemed safe to rely that they exist when these directories are stored.
But you proved that this is not safe: What happened in your case was presumably that you called "squashmount start" which returned with some failure, but only after it created the READONLY and CHANGES directories and stored that information. Then you did remove this directory, making the stored information "false".
In squashmount-8.5 this is "fixed" by verifying that the stored direectories actually exist, and if they do not to consider the stored information as invalid (hence act, as if that information is not stored, that is, in your case, create the directory). In a sense, due to this fix, squashmount thus now wants to be smarter than the user, but I think in this case this can do no harm.
Quote:
squashmount tries overlayfs anyway and fails.

No, squashmount did not fail: It mounted succesfully with aufs and returned succesfully.

I suppose that IO-Uncompress-Gunzip is installed, but your /proc/config.gz is not readable, since otherwise squashmount would have scanned that file to find that overlayfs is not configured for your kernel. (If this is not the case, you have found a bug in squashmount, and in this case please inform me, whether /proc/config.gz is readable and/or IO-Uncompress-Gunzip is installed).

This is the documented beviour of the default @order=qw(overlayfs? aufs! ...):
It first modprobes overlayfs, and if that fails, it checks for overlay in /proc/config.gz, assuming a positive result if the latter check fails. If you want to assume a negative result if the latter check fails, you have to use ?? instead of ?, i.e.
Code:
$order[0]='overlayfs??';

(or e.g. shift @order if you want to skip completely all checks for overlayfs and immediately try aufs).
Quote:
sometime in the future we could start a squashmount only thread and link here.

A few users will miss the new thread, and then there will actually be two threads... :(
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Fri Aug 22, 2014 12:24 pm    Post subject: Reply with quote

mv wrote:
I suppose that IO-Uncompress-Gunzip is installed, but your /proc/config.gz is not readable, since otherwise squashmount would have scanned that file to find that overlayfs is not configured for your kernel. (If this is not the case, you have found a bug in squashmount, and in this case please inform me, whether /proc/config.gz is readable and/or IO-Uncompress-Gunzip is installed).
What is IO-Uncompress-Gunzip? Kernel option? However my config.gz is readable:
Code:
$ zgrep GZIP /proc/config.gz
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_KERNEL_GZIP=y
CONFIG_RD_GZIP=y
CONFIG_DECOMPRESS_GZIP=y

$ ls -al /proc/config.gz
-r--r--r-- 1 root root 19K Aug 22 14:24 /proc/config.gz
Maybe it is not up-to-date, because I added some modules later without restart?

Weird, squashfs is mounted quickly, but as I mentioned some time ago, aufs takes very long, ~20s. For 10 mounts this takes very long at boot.
What is the preferred backend setup for squashmount with the optimal that is only few kernel patches?
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Fri Aug 22, 2014 1:41 pm    Post subject: Reply with quote

mv wrote:
With squashmount you can do it on the fly:
Code:
squashmount -fsx xz remount
BTW. I tried recompressing to test some COMPRESSIONs, but there is no recompression done:
Code:
$ squashmount -fsx xz remount
 * [adobe]: umounting...
 * [adobe]: cleaning changes...
 * [adobe]: mounting...

$ squashmount -fsx gzip remount
 * [adobe]: umounting...
 * [adobe]: cleaning changes...
 * [adobe]: mounting...
Same if changing the squashmount.pl config and forcing remount.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Lomaxx
Tux's lil' helper
Tux's lil' helper


Joined: 11 Nov 2004
Posts: 112

PostPosted: Sat Aug 23, 2014 10:16 am    Post subject: Reply with quote

Hello.

I've been using squashmount for quite some while by now. Since i did a world-update (including kernel-update) yesterday, i first broke my aufs-support, then repaired it and now have got it working again, BUT with an annoying delay while booting that I didn't experience before. The delay is altogether roughly 5-10 seconds long for 3 squashed directories (portage/db/layman), i do not receive any errors while booting and i wonder what might be causing it. Before there was no delay at all.
If someone got a close guess it would help me somewhat. Otherwise I have to try to solve it by trial and error.

Some info on my system:

- sys-kernel/aufs-sources-3.16.1
- sys-fs/aufs-headers-3.16_p20140818:0
- sys-fs/aufs-util-3.16_p20140818:0
- sys-fs/squashfs-tools-4.2-r1:0
- sys-fs/squashmount-6.0a:0

Kernel-Config regarding aufs:

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
# CONFIG_AUFS_EXPORT is not set
# CONFIG_AUFS_FHSM is not set
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

extracts from squashmount.pl:

#!/usr/bin/perl (this is only for editors)
@order = qw(aufs! unionfs-fuse! unionfs??# funionfs??#);
my $defaults = {
COMPRESSION => 'xz', # We could omit this line as xz is default.
# However, this might change in the future
COMPOPT_LZ4 => '-Xhc', # We could omit this line as -Xhc is default
COMPOPT_XZ => ['-Xbcj', 'x86'] # Use this in case COMPRESSION => 'xz'
};
my $non_binary = {
COMPOPT_XZ => undef # "-Xbcj x86" is slower for pure text archives
};

@mounts = (
added_hash($defaults, $non_binary, {
TAG => 'db',
DIR => '/var/db',
FILE => '/var/db.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/db.changes',
READONLY => '/var/db.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),
added_hash($defaults, $non_binary, {
TAG => 'layman',
DIR => '/var/lib/layman',
FILE => '/var/lib/layman.sqfs',
BACKUP => '.bak', # keep a backup in /var/db.sqfs.bak
# For an absolute path, we could have written:
# BACKUP => '/backup-disk/db.sqfs'
CHANGES => '/var/lib/layman.changes',
READONLY => '/var/lib/.readonly',
THRESHOLD => '30m',
BLOCKSIZE => 65536
}),

standard_mount('portage', '/usr/portage', $defaults, $non_binary, {
THRESHOLD => '80m',
FILL => qr{^local/(?!(\.git|profiles|metadata)(/|$))}
}),
);

As a solution I thought about rebuilding the directories freshly or enabling the userspace-option in the kernel, but haven't tried that yet. Any help would be appreciated.
Back to top
View user's profile Send private message
Lomaxx
Tux's lil' helper
Tux's lil' helper


Joined: 11 Nov 2004
Posts: 112

PostPosted: Sat Aug 23, 2014 10:27 am    Post subject: Reply with quote

Out of curiosity i measured the times and noticed that my 5-10-seconds-guess was a bit underestimated:

Each directory takes 15 seconds to mount although their sizes differ.
So in total it takes 45 seconds.

$ ls -l /usr/portage.sqfs
-rw-r--r-- 1 root root 76918784 22. Aug 15:49 /usr/portage.sqfs
$ ls -l /var/db.sqfs
-rw-r--r-- 1 root root 31797248 22. Aug 15:48 /var/db.sqfs
$ ls -l /var/lib/layman.sqfs
-rw-r--r-- 1 root root 11046912 29. Mai 14:19 /var/lib/layman.sqfs

CPU is a Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz with 16GB ram.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Sat Aug 23, 2014 11:20 am    Post subject: Reply with quote

Massimo B., thanks for testing. There was a regression in an apparently trivial code change in v7.6 (a negation was missing) so that the detection of IO::Uncompress::Gunzip was inverted.
Massimo B. wrote:
What is IO-Uncompress-Gunzip?

A perl module. It is part of virtual/perl-IO-Compress in gentoo and contained in >perl-5.9
Quote:
BTW. I tried recompressing to test some COMPRESSIONs, but there is no recompression done

It is not possible to force recompression if CHANGES is empty: --force only forces a remount, and -s only ignores threasholds which might apply. I think it is not necessary to have a special option which is useful only for "benchmarking".
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Sat Aug 23, 2014 11:24 am    Post subject: Reply with quote

Lomaxx wrote:
BUT with an annoying delay while booting

You are not the first one who reports a delay with aufs. It seems to be a problem of recent aufs versions.
Back to top
View user's profile Send private message
Lomaxx
Tux's lil' helper
Tux's lil' helper


Joined: 11 Nov 2004
Posts: 112

PostPosted: Sat Aug 23, 2014 2:33 pm    Post subject: Reply with quote

@mv: Good to know, bad to suffer. ;) Thanks for the feedback. If no cure is known, then I will fiddle around and report back in case i find a solution.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Mon Aug 25, 2014 2:37 pm    Post subject: Reply with quote

Beside all the patching around compression like LZ4 and overlayfs/aufs, can you give a short overview what is supported by main line kernels, what is easy to patch / add as userspace tools, and what is known to be broken or fiddly.

BTW. looking for backup strategies I got to know one more time btrfs, supporting transparent compression, incrementing snapshots, subvolumes etc. This would ease a lot of single mounts with squashmount, however btrfs is said to be still experimental although some enterprise linux has enabled this officially.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Mon Aug 25, 2014 3:24 pm    Post subject: Reply with quote

Massimo B. wrote:
can you give a short overview what is supported by main line kernels

The current state is easy:
Concerning squashfs, everything is supported except lz4.
Concerning making it read-writable: The only supported solution is unionfs-fue (or funionfs, but that does not seem to be maintained anymore). Neither aufs nor overlayfs are supported.
Of course, there are patches, but they are not mainline, and I cannot predict whether some will go mainline eventually. I would expect that squashfs-lz4 and overalyfs are the first to become mainline, but this is a wild guess.
Quote:
BTW. looking for backup strategies I got to know one more time btrfs, supporting transparent compression, incrementing snapshots, subvolumes etc.

I would not risk to use it now. Moreover, I heard it behaves not well on full disks.
What is even more important: All "online" compressions are necessarily rather suboptimal since you need a huge amount of data before you can compress well. For typical implementations this means that compression ratio is extremely poor for many small files. So for PORTDIR and /var/db, there will hardly be anything better than squashfs.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Tue Aug 26, 2014 8:21 am    Post subject: Reply with quote

Thanks for your opinion about btrfs making things clearer.

Adding new squashes still does not work:
Code:
$ squashmount -vvvvv umount
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]:    umounting...
/bin/umount -v -- /opt/Adobe
umount: /opt/Adobe (unionfs) unmounted
/bin/umount -v -- /opt/Adobe.mount/readonly
umount: /opt/Adobe.mount/readonly (/dev/loop0) unmounted
 * [adobe]:    cleaning changes...
 * [adobe]:    cleaning /opt/Adobe.mount/changes
 * [icedtea6]: error:   not mounted

$ squashmount forget
 * [adobe]:    forgetting settings
 * [icedtea6]: forgetting settings

$ squashmount reset
 * [adobe]:    resetting configuration
 * [icedtea6]: resetting configuration

$ squashmount -vvvvv mount
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]:    mounting...
/sbin/modprobe -v squashfs
/bin/mount -t squashfs -o loop,ro,noatime -v -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly
mount: /dev/loop0 mounted on /opt/Adobe.mount/readonly.
/sbin/modprobe -v fuse
/usr/bin/unionfs -o cow -o allow_other -o use_ino -o nonempty -o noatime -o hide_meta_files '/opt/Adobe.mount/changes=RW:/opt/Adobe.mount/readonly=RO' /opt/Adobe
 * [icedtea6]: error:   squashfile /usr/lib64/icedtea6.mount/icedtea6.sfs not found

So I go for sys-fs/unionfs-fuse now, eventhough aufs3 seems to be the better choice due to Knoppix. I changed to
Code:
#@order = qw(overlayfs aufs! unionfs-fuse! unionfs??# funionfs??#);
#@order = qw(aufs);
@order = qw(unionfs-fuse);

Switching to unionfs-fuse worked by remount, but the mounting still takes ~30s right after the squashfs:
Code:
$ squashmount -fs -vvv remount
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]:    umounting...
/bin/umount -- /opt/Adobe
/bin/umount -- /opt/Adobe.mount/readonly
 * [adobe]:    cleaning changes...
 * [adobe]:    cleaning /opt/Adobe.mount/changes
 * [adobe]:    mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly

### LONG TIME DELAY ###

/sbin/modprobe fuse
/usr/bin/unionfs -o cow -o allow_other -o use_ino -o nonempty -o noatime -o hide_meta_files '/opt/Adobe.mount/changes=RW:/opt/Adobe.mount/readonly=RO' /opt/Adobe
 * [icedtea6]: error:   not mounted

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Tue Aug 26, 2014 9:00 am    Post subject: Reply with quote

Massimo B. wrote:
Adding new squashes still does not work

If you want to create a new squashfile, you have to use "create" (or "start"), not "mount": Mount intentionally performs only "safe" operations (deleting directories recursively is always somewhat dangerous, no matter how many sanity checks are done first).
Quote:
eventhough aufs3 seems to be the better choice

It certainly is, but the disadvantage are the kernel patches. Personally, I use overlayfs and unionfs-fuse as a fallback. But sometimes I also use aufs and unionfs-fuse as a fallback. It depends which kernel patches I get earlier and install without problems. aufs always has problems with hardened-souces since it wants to hack a function pointer list which is readonly in hardened.
Quote:
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly

### LONG TIME DELAY ###

So in your case it is probably squashfs which takes so long to mount. I never had a long delay here. Is this file paricularly large? Or maybe it is too much "spread" on your harddisk. Maybe it helps to run a defragmentation utility for the underlying filesystem.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Tue Aug 26, 2014 10:08 am    Post subject: Reply with quote

mv wrote:
So in your case it is probably squashfs which takes so long to mount. I never had a long delay here. Is this file paricularly large? Or maybe it is too much "spread" on your harddisk. Maybe it helps to run a defragmentation utility for the underlying filesystem.

I don't think so. underlying filesystem is a some months old ext4:
Code:
$ ls -alh /opt/Adobe.mount/Adobe.sfs
-rw-r--r-- 1 root root 58M Aug 21 16:15 /opt/Adobe.mount/Adobe.sfs
$ filefrag /opt/Adobe.mount/Adobe.sfs
/opt/Adobe.mount/Adobe.sfs: 5 extents found

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Tue Aug 26, 2014 12:22 pm    Post subject: Reply with quote

Feature request: Is there a way to have THRESHOLD as relative levels like "0.2" (20%) or something? For binary only like /usr/lib/icedtea7 a change usually means recompression, so the default 0 is ok. But for dirs that are often changing like /var/db I usually like to have some relative level of difference instead of hard thresholds like 20 MB, no?
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Tue Aug 26, 2014 12:41 pm    Post subject: Reply with quote

Massimo B. wrote:
Feature request: Is there a way to have THRESHOLD as relative levels like "0.2" (20%)

The problem of this is: 20% of "what"? squashmount would have to calculate the size of all files, and this would take a lot of time. Usually, one changes mount-points not very frequenty, and mount-points usually do not change their size dramatically. For instance, for /var/db, you will probably always have roughly the same number of packages installed - nothing which makes it worth to calculate the size for almost every run of squashmount.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Tue Aug 26, 2014 1:42 pm    Post subject: Reply with quote

Massimo B. wrote:

/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly

### LONG TIME DELAY ###
I have no clue about this and squashmount is quite unusable with more than 1 or 2 mounts, so I opened a bug report. Please append information if you have more.
I also removed the modules and loaded again. I just wonder that it only dependy on xz_dec while I have all compressions enabled in the kernel config:
Code:
$ modinfo squashfs
filename:       /lib/modules/3.14.4-ck/kernel/fs/squashfs/squashfs.ko
license:        GPL
author:         Phillip Lougher <phillip@squashfs.org.uk>
description:    squashfs 4.0, a compressed read-only filesystem
alias:          fs-squashfs
depends:        xz_dec
intree:         Y
vermagic:       3.14.4-ck SMP preempt mod_unload

$ zgrep SQUASHFS /proc/config.gz |grep -v "^#"
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_DIRECT=y
CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Tue Aug 26, 2014 2:18 pm    Post subject: Reply with quote

Massimo B. wrote:
just wonder that it only dependy on xz_dec

ZLIB_DEFLATE and LZO_DECOMPRESS are probably hard compiled into the kernel (not as modules).
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Tue Aug 26, 2014 2:57 pm    Post subject: Reply with quote

True, those were hard compiled.
Weird, I got the mount delay problem fixed now , be resquashing everything, and using unionfs-fuse only. This should have been some aufs3 related issue though it did not look like...
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Tue Aug 26, 2014 3:13 pm    Post subject: Reply with quote

Massimo B. wrote:
This should have been some aufs3 related issue though it did not look like...

I doubt it: It was squashfs which took so long too mount.
Perhaps the file order was unfortunate for you, but it is hard to believe that this makes such a huge difference.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Wed Aug 27, 2014 7:18 am    Post subject: Reply with quote

The delay at mount -t squashfs is back again. It appeared again when switching around with COMPRESSION lzo and gzip and remounting.
I also got to know that some "forget" and forced create also may drop the existing squashfs and loosing all data. So I should be careful with important data such as /var/db.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Wed Aug 27, 2014 9:33 am    Post subject: Reply with quote

Massimo B. wrote:
I also got to know that some "forget" and forced create

Yes, this is an example where the sanity checks for the deletion in the "create" action are possibly insufficient. The "create" will create a non-empty squashfile, but the deletion afterwards might mark the data as deleted in CHANGES, so that the doubly-overmounted DIR becomes empty, and the next umount will store that empty DIR in the squashfile. Doubly overmounting just is a mess... this should not be able to happen if you do not use --force (there is a reason why there is a warning for it...)
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Wed Aug 27, 2014 3:53 pm    Post subject: Reply with quote

How do I preserve permissions in the squashfs? There is something wrong. I did chmod -R texlive:texlive /usr/local/texlive. Then after resquashing, the permissions are reset to root, and some updates I did before are lost. I guess the resquashing is failing somehow and some old squashfs is mounted. This time the mounting did take more than 10 minutes.
Code:
$ du -sh /usr/local/texlive /usr/local/texlive.mount/*
1,1G   /usr/local/texlive
4,0K   /usr/local/texlive.mount/changes
1,1G   /usr/local/texlive.mount/readonly
752M   /usr/local/texlive.mount/texlive.sfs

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        24G   19G  4,6G  81% /

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6242

PostPosted: Wed Aug 27, 2014 4:30 pm    Post subject: Reply with quote

Massimo B. wrote:
How do I preserve permissions in the squashfs?

Should be preserved automatically. (Of course, permissions of DIR and READONLY itself might change between mounted and umounted state).
Quote:
Then after resquashing, the permissions are reset to root, and some updates I did before are lost

Did you perhaps use different tools? I mean: If you once mounted with aufs, did not resquash, and next time mount with unionfs-fuse there might be problems, because the format in CHANGES for these two tools are probably not compatible with each other.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1530
Location: Germany

PostPosted: Thu Aug 28, 2014 7:13 am    Post subject: Reply with quote

mv wrote:
Did you perhaps use different tools? I mean: If you once mounted with aufs, did not resquash, and next time mount with unionfs-fuse there might be problems, because the format in CHANGES for these two tools are probably not compatible with each other.
Could be. But at least after resquashing that conflict should be gone, no? Then the mount delay should only appear for some of the squashmounts, not for the already re-squashed.

Let's try with a really small /usr/share/texmf-dist (because I switched to a local /usr/local/texlive):
Code:
$ squashmount list -vvv texmf
 * squashmount: reading config file /etc/squashmount.pl
 * [texmf]:         unionfs-fuse
                    unmodified
                    THRESHOLD: 0
                    DIR: /usr/share/texmf-dist
                    READONLY: /usr/share/texmf-dist.mount/readonly
                    CHANGES: /usr/share/texmf-dist.mount/changes
                    FILE: /usr/share/texmf-dist.mount/texmf-dist.sfs
                    mksquashfs options: -noappend -quiet -comp gzip
                    CHMOD: 0644
                    CHOWN: unchanged (0:0)

$ squashmount status -vvv texmf
 * squashmount: reading config file /etc/squashmount.pl
 * [texmf]:         not mounted
                    DIR: /usr/share/texmf-dist
                    READONLY: /usr/share/texmf-dist.mount/readonly
                    CHANGES: /usr/share/texmf-dist.mount/changes
                    WORKDIR: /usr/share/texmf-dist.mount/workdir
                    FILE: /usr/share/texmf-dist.mount/texmf-dist.sfs
                    mksquashfs options: -noappend -quiet -comp gzip
                    CHMOD: 0644
                    CHOWN: unchanged (0:0)

$ du -sh /usr/share/texmf-dist /usr/share/texmf-dist.mount/*
4,0K   /usr/share/texmf-dist
4,0K   /usr/share/texmf-dist.mount/changes
4,0K   /usr/share/texmf-dist.mount/readonly
396K   /usr/share/texmf-dist.mount/texmf-dist.sfs
4,0K   /usr/share/texmf-dist.mount/workdir

$ squashmount mount -vvv texmf
 * squashmount: reading config file /etc/squashmount.pl
 * [texmf]:         mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /usr/share/texmf-dist.mount/texmf-dist.sfs /usr/share/texmf-dist.mount/readonly
### DELAY ###
/sbin/modprobe fuse
/usr/bin/unionfs -o cow -o allow_other -o use_ino -o nonempty -o noatime -o hide_meta_files '/usr/share/texmf-dist.mount/changes=RW:/usr/share/texmf-dist.mount/readonly=RO' /usr/share/texmf-dist

$ du -sh /usr/share/texmf-dist /usr/share/texmf-dist.mount/*
4,1M   /usr/share/texmf-dist
4,0K   /usr/share/texmf-dist.mount/changes
4,1M   /usr/share/texmf-dist.mount/readonly
396K   /usr/share/texmf-dist.mount/texmf-dist.sfs
4,0K   /usr/share/texmf-dist.mount/workdir

$ touch /usr/share/texmf-dist/test && time squashmount remount -vvv texmf
 * squashmount: reading config file /etc/squashmount.pl
 * [texmf]:         squashing (this may take a while)
/usr/bin/mksquashfs /usr/share/texmf-dist /root/tmp/yb_MMvEReH -noappend -quiet -comp gzip
[==============================================================|] 114/114 100%
 * [texmf]:         umounting...
/bin/umount -- /usr/share/texmf-dist
/bin/umount -- /usr/share/texmf-dist.mount/readonly
 * [texmf]:         tempfile (/root/tmp/yb_MMvEReH) -> /usr/share/texmf-dist.mount/texmf-dist.sfs
 * [texmf]:         cleaning changes...
 * [texmf]:         cleaning /usr/share/texmf-dist.mount/changes
 * [texmf]:         mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /usr/share/texmf-dist.mount/texmf-dist.sfs /usr/share/texmf-dist.mount/readonly
### DELAY ###
/sbin/modprobe fuse
/usr/bin/unionfs -o cow -o allow_other -o use_ino -o nonempty -o noatime -o hide_meta_files '/usr/share/texmf-dist.mount/changes=RW:/usr/share/texmf-dist.mount/readonly=RO' /usr/share/texmf-dist

real   0m39.822s
user   0m0.532s
sys   0m0.017s

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 4 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum