Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Compressing filesystems with squashfs and squashmount
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Thu Aug 28, 2014 9:13 am    Post subject: Reply with quote

Massimo B. wrote:
But at least after resquashing that conflict should be gone, no?

But at resquash only the information is squashed which you currently can see; if there was a conflict, you might have squashed the wrong data.
Quote:
mount -t squashfs -o loop,ro,noatime -- /usr/share/texmf-dist.mount/texmf-dist.sfs /usr/share/texmf-dist.mount/readonly

For testing, you could mount manually with the above command (and umount manually) - this should be the cause for the delay for whatever reason. Another possibility is that it is not related with squashfs but with the mount command: I have util-linux-2.24.2 with the USE-flags "caps fdformat ncurses tty-helpers unicode" enabled. I suppose you have also pam and maybe something else. I never checked what mount actually does, but maybe with these additional useflags it might do something which waits for a timeout? Perhaps running strace with the above command gives a hint where the time is lost?
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Fri Aug 29, 2014 12:59 pm    Post subject: Reply with quote

Really, strace discovered, there is some blocking davfs2 on a slow link...
Code:
lstat("/mnt/sd2dav.massimo.b",

I just forgot about that since the delay was also last time I restarted and that davfs2 is only mounted temporarilly on user space. First I need to get rid of that davfs2 before claiming about squashfs.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Stolz
Moderator
Moderator


Joined: 19 Oct 2003
Posts: 3007
Location: Hong Kong

PostPosted: Sun Oct 26, 2014 11:52 am    Post subject: Reply with quote

I have been using squash_dir successfully for years but last updated has introduced a long delay every time I mount the squashed file systems. The funny thing is if you switch VT while the delay is still on you can access the file system that is supposed to be getting mounted. I saw there are a few reports of this issue in the last months but I didn't see any solution.

Right now I'm using sys-fs/squash_dir-13.6:mv, sys-fs/squashfs-tools-4.3:mv, sys-kernel/aufs-sources-3.17.1, aufs and LZO compression to squash /usr/portage and /usr/src.

Is there any workaround so far?
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Sun Nov 09, 2014 5:53 pm    Post subject: Reply with quote

On one machine I have a squashfs problem:
Code:
$ squashmount -vvv start
 * squashmount: reading config file /etc/squashmount.pl
 * [adobe]:         mounting...
/sbin/modprobe squashfs
/bin/mount -t squashfs -o loop,ro,noatime -- /opt/Adobe.mount/Adobe.sfs /opt/Adobe.mount/readonly
mount: /opt/Adobe.mount/readonly: mount failed: Unknown error -1
 * [adobe]:         error:   failed to mount squashfile
 * [firefox]:       mounting...
/bin/mount -t squashfs -o loop,ro,noatime -- /usr/lib64/firefox.mount/firefox.sfs /usr/lib64/firefox.mount/readonly
mount: /usr/lib64/firefox.mount/readonly: mount failed: Unknown error -1
 * [firefox]:       error:   failed to mount squashfile
Even mounting squashfs manually give the same error.
I even tried creating some new squashfs with default gzip and mounting of that fails the same.
Code:
$ lsmod |grep squash
squashfs               23672  0
xz_dec                 10928  1 squashfs

$ zgrep "SQUASH" /proc/config.gz |grep -v "^#"
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_DIRECT=y
CONFIG_SQUASHFS_DECOMP_MULTI=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3


Now I can't even emerge without db and portage. Is there a workaround? What is wrong with the kernel? Squashmount is using LZO for all mounts.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Sun Nov 09, 2014 6:38 pm    Post subject: Reply with quote

I have no solution to your problem: Perhaps it is bug of the kernel or the squashfs driver in the kernel.

What you can do is to use e.g.
Code:
cd /var/db.mount
unsquashfs db.sfs
mv /var/db /var/db.dup
mv squashfs-root /var/db
and similarly for your portage tree.

If unsquashfs fails, you have a problem: In this case your squashfile is corrupted and your data is lost unless you have a backup.
Might it be that you were running out of disk space when you were squashing? (squashmount checks the error codes, but maybe some modern filesystems do not report correctly when they are running out of disk space).
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Sun Nov 09, 2014 7:10 pm    Post subject: Reply with quote

unsquashfs works, but as even new created squashfs are not possible to be mounted, there must be something wrong with the kernel. I already did some make clean and built it again, but the problem remains.
-> topic 1003918
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Mon Nov 10, 2014 10:46 pm    Post subject: Reply with quote

According to mentioned topic, squashmount-8.7.6 now calls modprobe loop if $modprobe_loop is set (which defaults to true).
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Thu Dec 11, 2014 7:05 am    Post subject: Reply with quote

Has anyone tried yet squashmount with in-kernel built overlayfs (gentoo-sources-3.18.0) ?
For me it's not working, squashmount list show me mounts points as "bind" and everything it's mounted as read-only.
Config is: @order = ('overlayfs', 'aufs', 'unionfs-fuse', 'unionfs', 'funionfs');
I presume it's because cat /proc/filesystems show nodev overlay instead overlayfs, or other tools are required.
Aufs was working great, but, generally, patches are a little behind latest kernel, since some time now umount times are longer than usual (not due to resquash), and as overlayfs is in mainline I prefer to use it.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Thu Dec 11, 2014 2:45 pm    Post subject: Reply with quote

costel78 wrote:
Has anyone tried yet squashmount with in-kernel built overlayfs (gentoo-sources-3.18.0)?

Originally, I wanted to wait with this until hardened-sources-3.18.* appeared. If I should find time earlier, I will do earlier.
Quote:
I presume it's because cat /proc/filesystems show nodev overlay instead overlayfs, or other tools are required.

From what you say it can mean that they require now mount -t overlay ... instead of mount -t overlayfs ...
If this is the case, squasmount needs to be updated, first. It is not clear to me yet how to do this without breaking compatibility with earlier overlayfs.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Thu Dec 11, 2014 7:40 pm    Post subject: Reply with quote

mv wrote:
If this is the case, squasmount needs to be updated, first. It is not clear to me yet how to do this without breaking compatibility with earlier overlayfs.

Thank for answering. Overlay/overlayfs nomenclature seems to be problematic, indeed. I don't know how fast or reliable questioning /proc/filesystem is, but it was the quickest way for me.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Thu Dec 11, 2014 11:13 pm    Post subject: Reply with quote

squashmount-9.0.0 should now support "overlay" of kernel-3.18.

Note that the name "overlay" is now used in squashmount (e.g. in @order or in MOUNT_OVERLAY) to distinguish from "overlayfs" of earlier kernel versions.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Fri Dec 12, 2014 8:12 am    Post subject: Reply with quote

That was fast. :)
Unfortunately, it seems that something is wrong with mount options: workdir and upperdir must reside under the same mount
Code:
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: mounting...
Dec 12 10:07:51 gentoo kernel: overlayfs: workdir and upperdir must reside under the same mount
Dec 12 10:07:51 gentoo squashmount[1251]: mount: wrong fs type, bad option, bad superblock on overlay,
Dec 12 10:07:51 gentoo squashmount[1251]: missing codepage or helper program, or other error
Dec 12 10:07:51 gentoo squashmount[1251]: In some cases useful info is found in syslog - try
Dec 12 10:07:51 gentoo squashmount[1251]: dmesg | tail or so.
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: warning: overlay failed
Dec 12 10:07:51 gentoo squashmount[1251]: /usr/bin/which: no unionfs in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: error:   unionfs not found in $PATH
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: warning: unionfs-fuse failed
Dec 12 10:07:51 gentoo squashmount[1251]: /usr/bin/which: no funionfs in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: error:   funionfs not found in $PATH
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: warning: funionfs failed
Dec 12 10:07:51 gentoo squashmount[1251]: * [portage]: error:   fallback to mount --bind
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      mounting...
Dec 12 10:07:51 gentoo squashmount[1251]: mount: wrong fs type, bad option, bad superblock on overlay,
Dec 12 10:07:51 gentoo squashmount[1251]: missing codepage or helper program, or other error
Dec 12 10:07:51 gentoo squashmount[1251]: In some cases useful info is found in syslog - try
Dec 12 10:07:51 gentoo squashmount[1251]: dmesg | tail or so.
Dec 12 10:07:51 gentoo kernel: overlayfs: workdir and upperdir must reside under the same mount
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      warning: overlay failed
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      error:   unionfs not found in $PATH
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      warning: unionfs-fuse failed
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      error:   funionfs not found in $PATH
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      warning: funionfs failed
Dec 12 10:07:51 gentoo squashmount[1251]: * [db]:      error:   fallback to mount --bind

_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Fri Dec 12, 2014 12:22 pm    Post subject: Reply with quote

costel78 wrote:
it seems that something is wrong with mount options: workdir and upperdir must reside under the same mount

That's a restriction of "overlay": The error message means that WORKDIR and CHANGES must belong to the same partition ("upperdir" is what squashmount calls CHANGES).
If you use the default paths this should be the case (becaues they both belong to DIR.mount/...)
Maybe this error is caused because you tried to mount first with the old version of squashmount and could not properly umount?
I cannot reproduce this error here.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Fri Dec 12, 2014 1:18 pm    Post subject: Reply with quote

mv wrote:
That's a restriction of "overlay": The error message means that WORKDIR and CHANGES must belong to the same partition ("upperdir" is what squashmount calls CHANGES).
If you use the default paths this should be the case (becaues they both belong to DIR.mount/...)
Maybe this error is caused because you tried to mount first with the old version of squashmount and could not properly umount?


Previous umount was successfully. Even with overlay not working umount is always successful.
To be sure, I test with no previous squashmount, on brand new portage and db dir on disk.
I do not know perl, but I make a backup of squashmount script and tried some "classic" debugging with print and overlay is right about the error:
Code:
Dec 12 15:04:11 mini squashmount[10325]: Workdir: /tmp/ra3n7kRDDc
Dec 12 15:04:11 mini squashmount[10325]: Changes: /var/db.changes, Read-Only: /var/db.readonly, WorkDir: /tmp/ra3n7kRDDc
Dec 12 15:04:11 mini squashmount[10325]: mount: wrong fs type, bad option, bad superblock on overlay,
Dec 12 15:04:11 mini squashmount[10325]: missing codepage or helper program, or other error
Dec 12 15:04:11 mini squashmount[10325]: In some cases useful info is found in syslog - try
Dec 12 15:04:11 mini squashmount[10325]: dmesg | tail or so.


In my configuration /tmp is mount on RAM as tmpfs - afaik that is default for systemd - and /var/db, /var/db.changes and /var/db.readonly are on /dev/sda2
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Fri Dec 12, 2014 1:29 pm    Post subject: Reply with quote

costel78 wrote:
and /var/db, /var/db.changes and /var/db.readonly are on /dev/sda2

So you are using the "obsolete" defaults, not the current ones:
The defaults have changed to use /var/db /var/db.mount/changes /var/db.mount/readonly and /var/db.mount/workdir, respectively:
This change happened exactly, because overlayfs had this requirement concerning WORKDIR being on the same partition as CHANGES.
If you want to use overlay[fs], you should better specify WORKDIR explicitly.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Fri Dec 12, 2014 2:24 pm    Post subject: Reply with quote

You were right. It is working now. My bad by ignoring etc-update.
Comparing to aufs there is no delay at mount or umount.

Thank you very much!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Thu Jan 08, 2015 6:10 pm    Post subject: Reply with quote

Hi, now that overlayfs is part of the 3.18 kernel and supported by squashmount as "overlay", is that now the preferred choice? I had unionfs-fuse before and keep it as fallback, but that itself claims to have some performance issues due to userspace fuse compared to kernel solutions.

Some other issue: Back on some machine behind a http proxy I realised that http://overlays.gentoo.org/ is down for a while now and no ETA (look at http://infra-status.gentoo.org/). With your mv overlay being the only important overlay currently for my basic installation I'm lost without that overlay. Is there no other http mirror at least for that single overlay I can add manually to layman? On the github there are no ebuilds. For now I just added the script manually via wget, but like to keep updated by the overlay as usual.
Bad enough that the overlay mirror is down, is there no alternative? There is http://gpo.zugaina.org/ with snapshots but it does not seem to be appropriate for layman or as overlay in general.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Thu Jan 08, 2015 7:42 pm    Post subject: Reply with quote

Massimo B. wrote:
Hi, now that overlayfs is part of the 3.18 kernel and supported by squashmount as "overlay", is that now the preferred choice?

I guess so, at least until perhaps someday aufs will arrive in mainline kernel: In the moment overlay is taken by default if you have it and do not have modified @order.
Quote:
I realised that http://overlays.gentoo.org/ is down for a while

AFAIK, only the web interface is down. You can still use the overlay with git.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Thu Jan 08, 2015 7:52 pm    Post subject: Reply with quote

mv wrote:
AFAIK, only the web interface is down. You can still use the overlay with git.
I have some hosts in http restricted proxied networks, which is not that unusual. There also emerge-delta-webrsync is doing a good job, but always when it comes to git:// it is hard to solve (like Vim Addon Manager also relying mostly on git:// sources even when there are http:// alternatives around).
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1512
Location: Germany

PostPosted: Wed Feb 04, 2015 11:55 am    Post subject: Reply with quote

mv wrote:
Massimo B. wrote:
I realised that http://overlays.gentoo.org/ is down for a while
AFAIK, only the web interface is down. You can still use the overlay with git.
mrueg told me he has mirrored his Overlay on github for that issue. Since I can't access your overlay on some machines, looking out for a http mirror, I found http://cgit.gentooexperimental.org/user/mv.git/ supposed to be maintained by xiaomiao, a quite up-to-date cgit interface. If this is reliable it could be an idea to ask layman if he would add this mirror as http-fallback for all overlays until overlays.gentoo.org is back.
_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G|amd64:HP EliteBook 8560w,i7-2620M,16G|amd64:Acer Z5610 (Core2QuadQ8200),8G|amd64-prefix:OpenSuse
Lila-Theme
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Fri Feb 20, 2015 7:10 pm    Post subject: Reply with quote

With systemd it happens sometimes that filesystems are mounted but actually do not become mounted, at least if some --make-shared is in effect, see e.g. the problem described in the thread concerning systemd-216: mount.

If this happens, squashmount claims to have mounted successfully, but when shutting down squashes the (unmounted, hence usually empty!) directory and fails to umount: Thus, you get horrible failures with data loss in such a case!

With systemd-219 the problem has increased, since systemd now uses the concept that every mounting operation is superpvised by systemd and automatically acts like the starting of an internal unit (e.g. being displayed with systemctl). This is complete inappropriate behaviour for any init system: systemd attempts to play the role of the operating system which it should not do, and fails miserably in this attempt.

Due to all these bugs and idiocies, systemd support in squashmount is now officially dropped: It would not be sane to write dozens of hacks to work around all the breakage which systemd causes by its completely broken concepts.

You may be lucky and squashmount works with systemd for you (it would not be a problem if systemd would not change the behaviour of "mount" itself), but don't expect to get support or that squashmount works around the systemd bugs (including the conceptional bugs, which are the worst ones, of course).

Use a sane init system like openrc!

Edit: Reformulated according to most current information.
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3792

PostPosted: Wed Feb 25, 2015 9:41 am    Post subject: Reply with quote

Quote:
Due to all these bugs and idiocies, systemd support in squashmount is now officially dropped: It would not be sane to write dozens of hacks to work around all the breakage which systemd causes by its completely broken concepts.


sorry to hear that but it is of course understandable.

Before i start messing around, what is actually the safest way to undo this setup ?

TIA !!
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Wed Feb 25, 2015 9:57 am    Post subject: Reply with quote

gringo wrote:
Before i start messing around, what is actually the safest way to undo this setup?

Do you mean undoing squashmount or systemd?
For squashmount, you just quit using it.
For systemd, you don't need to do much: You can install openrc and systemd together and decide just to boot with openrc. This is what I am doing.
In fact, I am still happy to have systemd as an emergency boot system, but after I had severe data loss of squsahmount data with it (and it was not the fault of squsahmount), I am careful to avoid resquashing when booting with systemd.

On the other hand, if you are not using a special setup (like e.g. with mount --make-shared for some partitions), chances are good that you can use squashmount with systemd without any issues, provided you have linux-utils emerged with USE=systemd.
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3792

PostPosted: Wed Feb 25, 2015 10:51 am    Post subject: Reply with quote

thanks !

Quote:
Do you mean undoing squashmount or systemd?


sorry for not being clear enough : i meant squashmount.

i have just stopped squashmount and have now empty target directories, i guess something else has to be done to have the contents back ?
Just to be clear enough, one of the squashed dirs i have is /var/db, after stopping squashmount i end up with an empty /var/db directory.

TIA !
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6182

PostPosted: Wed Feb 25, 2015 11:34 am    Post subject: Reply with quote

Once more: If you do not have a special setup, the combination squashmount+systemd will probably just work.
What is not going to happen is that squashmount will add hacks to workaround the bugs of systemd which cause "mount" to fail in certain situations (e.g. sometimes when mount --make-shared is in effect) without reporting a failure: This is what I meant by "support is dropped".
Quote:
i have just stopped squashmount and have now empty target directories

If they are empty while squashmount+systemd are running, you have hit the mentioned systemd bug.
If they were correct while squashmount was running and you called squashmount stop, they are supposed to be empty, since the content of the directory is now in the squashed file. You can unsquash the file to get the content without mounting anything. However, in this case you do not have a reason to do so, since apparently squashmount+systemd then just works for you.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7, 8, 9, 10  Next
Page 5 of 10

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum