Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 11, 12, 13, 14  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Tue Oct 22, 2013 5:54 am    Post subject: Reply with quote

slycordinator wrote:
error parsing /etc/squashmount.pl: Undefined subroutine &CFG::getpwname called at /etc/squashmount.pl line 40

Typo in config: should be getpwnam (not getpwname).
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Tue Oct 29, 2013 11:40 pm    Post subject: Reply with quote

i'd like to try this.

where's the updated version that works with copy paste?

The script has a lot of variables and symbols and lines.

I wonder what's the difference between the script method and manually mounting a temp folder in ram then 7z'ed?

simple steps:

1- create a dir in ram:
Code:
mkdir /tmp/portage

2- mount ram:
Code:
mount -t tmpfs,size=1G /tmp/portage

3- rsync your portage dir unto it.
Code:
rsync -Pvhr --exclude=distfiles /usr/portage/ /tmp/portage

4- edit make.conf:
Code:
PORTDIR=/tmp/portage

5- emerge sync, and/or work with portage as normal:
emerge --sync
6- compress it with pigz back to the hard drive, with multiple thread options should be awsome!!:
Code:
time tar cf - portdir/ | pigz -c -9 > portage.tar.gz

real    0m3.708s
user    0m26.297s
sys     0m1.406s


Code:
ls -sh portage.tar.gz
81M portage.tar.gz


7- when rebooting de-compress after step 2 above.
Code:
to do



This method no need to unmount as rebooting destroys /tmp/portage.


emerge -pv system
in ram (type tmpfs):

Code:
real    0m16.902s
user    0m16.556s
sys     0m0.251s


in hard drive with xfs:

Code:
real    0m36.389s
user    0m20.462s
sys     0m0.462s


Haven't done a full cache flash but did the test over and over to check is not one time numbers. They are consistent.

And my ram isn't the fastest out there.

ps: got confused with best compression/decompression method. 7z uses ONE core!!!?? tried this (ultra settings found in man page):

Code:
7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on portage.7z portdir


had to ctrl-c, because using one core and taking too long.

Didn't find a way to completely unpigz since I was left with portage.tar :/
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Wed Oct 30, 2013 9:57 am    Post subject: Reply with quote

_______0 wrote:
i'd like to try this.

where's the updated version that works with copy paste?

If you mean synss' original script, this is not maintained since ages. The actively maintained version is squashmount from the mv overlay, but except for some variable names and the basic idea this has nothing in common anymore with the original script.
Quote:
I wonder what's the difference between the script method and manually mounting a temp folder in ram then 7z'ed?

Here are the main differences:
  • You loose a lot of ram for no gain. For directories like texmf-dist for which you do it more for size than speed, this is completely indiscussible. Only for the portage tree it might be acceptable if you have lot of ram to waste, but why waste it unnecessarily?
  • You must recompress after every change - no possibilities to compress only when a threshold is reached.
  • On unclean shutdown (e.g. crash) all changes are lost.
  • You must unpack after every reboot.
  • You do not have a readonly version of the portage tree to compare changes with easily.
  • You do not have a directory in which you can see quickly which files actually have changed. (This is rather convenient e.g. after syncing the portage tree, although you can of course also use "find" and friends with timestamps).
  • After all these disadvantages, what is the advantage? Maybe a slightly better compression (hower, I didn't realize a significant increase in compression size for the squashfs format; lzma should be about as small as 7z, although it is certainly possible to construct some theoretical examples in which one of the two wins clearly). Another advantage is clearly that no kernel patches are needed. This advantage will only vanish when linux upstream finally includes aufs/overlayfs/something else. That things work since many years but are not included upstream is really a sad story for years which makes me more and more doubt aobut the idea of free software. (Of course, e2compr is another such sad story...)
Back to top
View user's profile Send private message
_______0
Guru
Guru


Joined: 15 Oct 2012
Posts: 521

PostPosted: Wed Oct 30, 2013 1:17 pm    Post subject: Reply with quote

7z is:

Code:
time 7z a -t7z -mmt=on -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on portage.7z portdir/

real    1m57.155s
user    5m51.693s
sys     0m7.447s

54M portage.7z




But it takes way longer than pigz. Cores are not at full and at the end of the compression process there's only one working at 100%. And p7zip -d[/d] doesn't decompress using multiple core.

With [b]pigz


Code:
81M portage.tar.gz


But again, decompressing leaves a 400MB portage.tar :/

Is there a way to compress portage dir with pigz without tar? I want to check fully threaded de-compression.

Ok so your script looks nice, where to get it? Link?

I didn't get your last part:

Code:
This advantage will only vanish when linux upstream finally includes aufs/overlayfs/something else. That things work since many years but are not included upstream is really a sad story for years which makes me more and more doubt aobut the idea of free software. (Of course, e2compr is another such sad story...)


What will be included and what will vanish? squashfs will vanish?

e2compr is an interesting development.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Wed Oct 30, 2013 2:23 pm    Post subject: Reply with quote

_______0 wrote:
What will be included and what will vanish? squashfs will vanish?

I hope that some day aufs or overlayfs or something similar will be included. Then the disadvantage that you need a patched kernel will vanish.
Quote:
e2compr is an interesting development.

Yes, it was excellent and maintained over many kernel versions. The problem is that upstream rejected it like aufs for no serious reasons (a few technical arguments were invented like theoretical problems with unmodified e2fsck in some corner cases, but it is obvious that the real reason is only NIH syndrome). Eventually the developer of e2compr gave up.
This is the main problem with open source, that upstream practically has full power and is not forced to decide on technical reason. In theory everybody can fork the kernel (or gnome, kde, systemd etc.) but in practice nobody has the resources to do that so that we all have to take what is shoved down our throats. It is not much different to proprietary software.
Back to top
View user's profile Send private message
emc
Guru
Guru


Joined: 02 Jul 2004
Posts: 390
Location: Cracow, Poland

PostPosted: Tue Nov 26, 2013 11:41 pm    Post subject: Reply with quote

Problem with overlay?
Code:
gopher emc # eix -I squashmount
-- Invalid line 1 in /var/lib/layman/mv/profiles/package.mask: '<<<<<<< ...'
    Can't read category.

-- Invalid line 3 in /var/lib/layman/mv/profiles/package.mask: '======= ...'
    Can't read category.

-- Invalid line 5 in /var/lib/layman/mv/profiles/package.mask: '>>>>>>> ...'
    Can't read category.

-- Invalid line 1 in /var/lib/layman/mv/profiles/package.mask: '<<<<<<< ...'
    Can't read category.

-- Invalid line 3 in /var/lib/layman/mv/profiles/package.mask: '======= ...'
    Can't read category.

-- Invalid line 5 in /var/lib/layman/mv/profiles/package.mask: '>>>>>>> ...'
    Can't read category.

[I] sys-fs/squashmount [1]
     Available versions:  (~)3.7-r1^m (~)3.8^m
     Installed versions:  3.8^m(01:06:44 AM 11/23/2013)
     Homepage:            http://forums.gentoo.org/viewtopic-t-465367.html
     Description:         Keep directories compressed with squashfs. Useful for portage tree, texmf-dist

[1] "mv" /var/lib/layman/mv
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Wed Nov 27, 2013 9:23 am    Post subject: Reply with quote

emc wrote:
Problem with overlay?

The overlay (and also many git packages) were rebased to update the email backward in history. layman cannot deal with rebased repositories. "Solution":
Code:
layman --delete mv; layman --add mv

For packages it might be necessary to remove the git source from your DISTDIR manually before reemerging.
Back to top
View user's profile Send private message
emc
Guru
Guru


Joined: 02 Jul 2004
Posts: 390
Location: Cracow, Poland

PostPosted: Fri Dec 13, 2013 11:02 pm    Post subject: Reply with quote

mv:
I have to say I'm a bit lost, what I really need to start squashmount with?
Currentlly I'm using geek-sources with aufs USE flag (beside other patches)
Code:
# grep AUFS .config213_4
CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
# CONFIG_AUFS_RDU is not set
# CONFIG_AUFS_SP_IATTR is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
# CONFIG_AUFS_DEBUG is not set


I have as well:
Code:
# eix -I aufs
[I] sys-fs/aufs-headers
     Available versions:  (~)3.11_p20130915
     Installed versions:  3.11_p20130915(10:10:36 AM 10/06/2013)
     Homepage:            http://aufs.sourceforge.net/
     Description:         User space headers for aufs3

[I] sys-fs/aufs-util
     Available versions:  (~)3.9_p20130915 **99999999(0/3.9)*l[1]
     Installed versions:  3.9_p20130915(10:12:39 AM 10/06/2013)
     Homepage:            http://aufs.sourceforge.net/
     Description:         Userspace tools for aufs

[U] sys-fs/aufs3
     Available versions:  (~)3_p20131007 (~)3_p20131014 (~)3_p20131104-r1 (~)3_p20131111-r1 {debug doc fuse hfs inotify kernel-patch nfs pax_kernel ramfs KERNEL="linux"}
     Installed versions:  3_p20130928(09:54:13 PM 10/07/2013)(-debug -doc -fuse -hfs -inotify -kernel-patch -nfs -pax_kernel -ramfs KERNEL="linux")
     Homepage:            http://aufs.sourceforge.net/
     Description:         An entirely re-designed and re-implemented Unionfs

[1] "mv" /var/lib/layman/mv

and upgrade aufs3 failed because CONFIG_AUFS_FS=y was set, but I i was succesfully instal in first place.

Anyway the question is what I need to start using squashmount. Kernel with aufs support but why aufs3 package failed, should I set CONFIG_AUFS_FS=m?
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Fri Dec 13, 2013 11:57 pm    Post subject: Reply with quote

If you do have a kernel with aufs patches (like probably geek sources), you need nothing else: You must enable squashfs and aufs in the kernel ("m" is probably more tested, but "y" should work as well) and setup the appropriate /etc/squashmount.pl
Back to top
View user's profile Send private message
emc
Guru
Guru


Joined: 02 Jul 2004
Posts: 390
Location: Cracow, Poland

PostPosted: Fri Jan 17, 2014 4:08 pm    Post subject: Reply with quote

Can you do anything regarding last crash of overlays.gentoo.org:

Code:
# layman -a mv

 * Adding overlay,...
 * Running Git... # ( cd /var/lib/layman  && /usr/bin/git clone git://git.overlays.gentoo.org/user/mv.git /var/lib/layman/mv )
Cloning into '/var/lib/layman/mv'...
fatal: read error: Connection reset by peer
 * Failure result returned from Git
 * Running Git... # ( cd /var/lib/layman/mv  && /usr/bin/git config user.name "layman" )
 * [Errno 2] No such file or directory: '/var/lib/layman/mv'
 *
 * Trying next source of listed sources...
 * Running Git... # ( cd /var/lib/layman  && /usr/bin/git clone http://git.overlays.gentoo.org/gitroot/user/mv.git/ /var/lib/layman/mv )
Cloning into '/var/lib/layman/mv'...
error: File 8c89b13b1966abb6f1695254fe497968b03fe6a4 (http://git.overlays.gentoo.org/gitroot/user/mv.git/objects/8c/89b13b1966abb6f1695254fe497968b03fe6a4) corrupt
error: Unable to find 8c89b13b1966abb6f1695254fe497968b03fe6a4 under http://git.overlays.gentoo.org/gitroot/user/mv.git
Cannot obtain needed object 8c89b13b1966abb6f1695254fe497968b03fe6a4
error: Fetch failed.
 * Failure result returned from Git
 * Running Git... # ( cd /var/lib/layman/mv  && /usr/bin/git config user.name "layman" )
 * [Errno 2] No such file or directory: '/var/lib/layman/mv'
 *
 * Trying next source of listed sources...
 * Running Git... # ( cd /var/lib/layman  && /usr/bin/git clone git+ssh://git@git.overlays.gentoo.org/user/mv.git /var/lib/layman/mv )
Cloning into '/var/lib/layman/mv'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
 * Failure result returned from Git
 * Running Git... # ( cd /var/lib/layman/mv  && /usr/bin/git config user.name "layman" )
 * [Errno 2] No such file or directory: '/var/lib/layman/mv'
 * Adding repository "mv" failed!

 * CLI: Errors occurred processing action add
 * Adding repository "mv" failed!
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Fri Jan 17, 2014 6:45 pm    Post subject: Reply with quote

emc wrote:
Can you do anything regarding last crash of overlays.gentoo.org

Edit: Somebody is working on this
Back to top
View user's profile Send private message
qsuscs
n00b
n00b


Joined: 11 Jan 2014
Posts: 6

PostPosted: Tue Jan 21, 2014 8:29 pm    Post subject: Reply with quote

I have this configuration for squashmount (okay, it's actually the "default"):
Code:
        standard_mount('portage', '/usr/portage', $defaults, {
                THRESHOLD => '80m',
                # Any change in the local/ subdirectory (except in .git,
                # profiles, metadata) should lead to a resquash, even if
                # the threshold is not reached:
                FILL => qr{^local/(?!(\.git|profiles|metadata)(/|$))}
        })

So I have the directories /usr/{portage|portage.readonly|portage.changes}. According to mount(8), /usr/portage.sqfs is mounted on /usr/portage.readonly, and "aufs" is mounted on /usr/portage. [s]I’d guess that the overlay consists of /usr/portage.readonly and /usr/portage.changes (is there a way to find out? mount(8) isn’t particularly helpful: aufs on /usr/portage type aufs (rw,noatime,si=d3ae66ea856a49d)).[/s] EDIT: Yep, /sys/fs/aufs/si_d3ae66ea856a49d contains all that info. (and why does cross-out not work)
If there is no according entry in mount(8), does that mean that /usr/portage.changes is actually written on disk?
Is /usr/portage.readonly respectively /usr/portage then read from /usr/portage.sqfs directly or from RAM? Can I also put that and /usr/portage.changes into the RAM? I mean, if any changes get lost, another emerge --sync will do, and since I’ve usually ~10G of RAM free, I’ll most likely not even notice that 700M from the portage tree.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Tue Jan 21, 2014 9:51 pm    Post subject: Reply with quote

qsuscs wrote:
I’d guess that the overlay consists of /usr/portage.readonly and /usr/portage.changes

Yes; this is how the standard_mount function works
Quote:
is there a way to find out? mount(8) isn’t particularly helpful

For overlayfs, mount prints also the parameters upperdir and lowerdir, but for aufs this may be different, of course. It might also depens on your aufs/kernel/mount version. To find out what squashmount thinks about it, you can use
Code:
squashmount print-changes

Quote:
If there is no according entry in mount(8), does that mean that /usr/portage.changes is actually written on disk?

Yes, this intentional: If you reboot without resquashing you should not have lost the changes. Moreover, you need not resquash at every reboot after only a few changes if you set THRESHOLD.
Quote:
Can I also put that and /usr/portage.changes into the RAM

Of course. An example is on the "manpage" (I just realize that there is a typo: => is meant here):
Code:
CHANGES => sub { return File::Temp::newdir(undef, DIR => '/path/to/ramdisk') }

Alternatively, if you have /tmp as a ramdisk, just do not specify CHANGES: Then CHANGES becomes a temporary directory in /tmp. (However, note that if you use the standard_mount function to define your mountpoint then CHANGES is automatically set; therefore you must use something like
Code:
CHANGES => undef

to unset it explicitly in this case.) However, I would consider it a bad idea to waste valuable RAM permanently just to store changes in the portage tree: If e.g. an eclass is changed, a lot of files are stored here, and you will not regain this RAM. In the best case, it is then swapped to disk...
Quote:
Is /usr/portage.readonly respectively /usr/portage then read from /usr/portage.sqfs directly or from RAM?

/usr/portage.readonly is read from /usr/portage.sqfs, but the kernel/squashfs has some mechanisms to cache the data. In the forthcoming kernel 3.13 there are even some improvements of these mechanisms. /usr/portage is just the "merge" of /usr/portage.readonly and /usr/portage.changes, so it must read from both directories. Of course, also in this case some caching mechanisms hold.
Back to top
View user's profile Send private message
qsuscs
n00b
n00b


Joined: 11 Jan 2014
Posts: 6

PostPosted: Wed Jan 22, 2014 4:01 pm    Post subject: Reply with quote

Thanks for that answer, now everything works fine.

I eventually figured out that I’d have to unset CHANGES, but I don’t know perl at all, unfortunately.
I did see the documentation to overlayfs, but I couldn’t figure out how to use it, so I just used aufs.
And as I said, I don’t use too much of my RAM, most of it is free. I build everything there, even libreoffice, and even without swap, I never ran into memory problems. After today’s sync, the portage changes dir is only 3.2M of size, but I doubt that I’d notice 500M.
Back to top
View user's profile Send private message
Massimo B.
Veteran
Veteran


Joined: 09 Feb 2005
Posts: 1209
Location: PB, Germany

PostPosted: Mon Jan 27, 2014 8:10 am    Post subject: Reply with quote

On a new i5 CPU I like to benchmark the different COMPRESSION. How can I force re-squash with a different cipher set with old squash_dir-13.4-r1? I already tried -s but it does not re-squash:
Code:
$ squash_dir -s restart portage
/etc/init.d/squash_portage stop    # is unmodified

_________________
ppc:PowerBook5,8 15"(1440)-G4/1.67,2G | amd64:Acer Z5610 (Core2QuadQ8200),i5-3470 | amd64-prefix:OpenSuse | Lila-Theme
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Mon Jan 27, 2014 12:44 pm    Post subject: Reply with quote

Massimo B. wrote:
On a new i5 CPU I like to benchmark the different COMPRESSION. How can I force re-squash with a different cipher set with old squash_dir-13.4-r1?

Force resquashing:
Code:
squash_dir -fs restart
However, with squash_dir you are not able to change COMPRESSION on the fly: You must edit the configuration. This is the fault of squash_dir relying in openrc: It is not possible to pass environment variables through openrc (unless you mess around with openrc configuration itself). With squashmount you can do it on the fly:
Code:
squashmount -fsx xz remount
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3734

PostPosted: Tue May 20, 2014 6:41 pm    Post subject: Reply with quote

hi all,

when i booted today i realized /var/db wasnt mounted and restarted the service to see why :

Code:
may 20 20:22:43 mountain systemd[1]: Stopping mount/umount all squashmount configured mountpoints...
may 20 20:22:43 mountain squashmount[1406]: * [db]:      error:   not mounted
may 20 20:22:43 mountain squashmount[1406]: * [db]:      forgetting settings
may 20 20:22:45 mountain squashmount[1406]: * [portage]: umounting...
may 20 20:22:45 mountain squashmount[1406]: * [portage]: forgetting settings
may 20 20:22:45 mountain systemd[1]: Starting mount/umount all squashmount configured mountpoints...
may 20 20:22:45 mountain squashmount[1414]: * [db]:      mounting...
may 20 20:22:45 mountain squashmount[1414]: mount: wrong fs type, bad option, bad superblock on /dev/loop0,
may 20 20:22:45 mountain squashmount[1414]: missing codepage or helper program, or other error
may 20 20:22:45 mountain squashmount[1414]: In some cases useful info is found in syslog - try
may 20 20:22:45 mountain squashmount[1414]: dmesg | tail or so.
may 20 20:22:45 mountain kernel: SQUASHFS error: Filesystem uses "unknown" compression. This is not supported
may 20 20:22:45 mountain squashmount[1414]: * [db]:      error:   failed to mount squashfile
may 20 20:22:45 mountain squashmount[1414]: * [portage]: mounting...
may 20 20:22:45 mountain systemd[1]: Started mount/umount all squashmount configured mountpoints.


havent touched this since setup :

Code:
#!/usr/bin/perl

@order = ( 'unionfs-fuse' );

push(@mounts, {
   TAG => 'db',
   DIR => '/var/db',
   FILE => '/var/db.sqfs',
   BACKUP => '/var/db.sqfs.bak',
   CHANGES => '/var/db.changes',
   READONLY => '/var/db.readonly',
   THRESHOLD => '30m',
   BLOCKSIZE => 65536

   }, {
   TAG => 'portage',
   DIR => '/var/repos/portage',
   FILE => '/var/repos/portage.sqfs',
   CHANGES => '/var/repos/portage.changes',
   READONLY => '/var/repos/portage.readonly',
   THRESHOLD => '40m'
   });



this is with 6.0, i think im missing sth. obvious here and dont want to mess around before i hear someone elses thoughts.

cheers guys
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Tue May 20, 2014 7:39 pm    Post subject: Reply with quote

The default COMPRESSION value has changed from 'xz' to 'lz4'.

Unfortunately, it seems that lz4 support for squashfs is not yet in the mainstream kernel (although lz4 support is in the kernel since quite a while, a brief harmless patch was submitted a year ago, and mksquashfs supports lz4).

Not sure what to do now: Does anybody know whether squashfs with lz4 can be expected in a reasonable time in the kernel?

(If not, maybe it is necessary to release a squashmount version with different defaults).

To avoid the problem, set the COMPRESSION value explicitly.
Now that you have already compressed in lz4 format, you must resquash: Use
Code:
unsquash /path/to/your/squashfile # to unpack
mksquash /path/of/unpackad/data /path/to/squashfile -noappend -comp xz
rm -rf  /path/of/unpackad/data

(if you have customized the MKSQUASH variable append its value also to the second command.)
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Tue May 20, 2014 10:36 pm    Post subject: Reply with quote

In the new squashmount-6.0a, it is reverted to the COMPRESSION=xz default.

When mainstream kernel supports squashfs with lz4, this release will probably be considered obsolete, but perhaps this will never happen.

@gringo: This means when you update you can keep your original configuration. However, the archives which are now in lz4 format have to be transferred back to xz format in the way described in the previous posting.
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3734

PostPosted: Wed May 21, 2014 8:27 am    Post subject: Reply with quote

thanks @mv : everything worked as you described.

cheers guys
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 199

PostPosted: Sun Jul 27, 2014 8:45 pm    Post subject: Reply with quote

When I try to stop squashmount service I get:
Code:
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error:   non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error:   lazy umount failed: /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      error:   non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /var/db
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      error:   lazy umount failed: /var/db


Packages version and use flags:
Code:
[ebuild   R    ] sys-fs/aufs-headers-3.15_p20140728  0 kB
[ebuild   R    ] sys-fs/aufs-util-3.15_p20140728  0 kB
[ebuild   R    ] sys-kernel/aufs-sources-3.15.6:3.15.6  USE="experimental symlink -build -deblob -module -vanilla" 0 kB
[ebuild   R    ] sys-fs/squashmount-7.7::added  0 kB
[ebuild   R    ] sys-fs/squashfs-tools-4.3::added  USE="xz -lz4 -lzma -lzo -xattr" 0 kB


Config file:
Code:
#!/usr/bin/perl (this is only for editors)

# The tools which we have installed; if possible only the first in this list
# is used, but the others are a fallback if that fails.

@order = ('aufs', 'overlayfs', 'unionfs-fuse', 'unionfs', 'funionfs');

# Even if we define following is empty it is convenient to use
# this local variable throughout, so that we can simply change it:

my $defaults = {
   COMPRESSION => 'xz'
};

push(@mounts, {
   TAG => 'portage',
   DIR => '/usr/portage',
   FILE => '/usr/portage.sqfs',
   CHANGES => '/usr/portage.changes',
   READONLY => '/usr/portage.readonly',
   THRESHOLD => '50m' # resquash on umount if 40 megabytes changed
}, {
   TAG => 'db',
   DIR => '/var/db',
   FILE => '/var/db.sqfs',
   CHANGES => '/var/db.changes',
   READONLY => '/var/db.readonly',
   THRESHOLD => '30m' # resquash on umount if 40 megabytes changed
}

);


Obviously I use systemd and I didn't notice the error but during shutdown/reboot system take unusually long and culprit seems to be squashmount. It recompress despite the *.changes are well bellow threshold.
I tried to start from scratch with same results: with squashmount started compress portage and db directories, disable squashmount service, reboot, put data back and start squashmount. It does not recompress the whole thing unless threshold is reached but systemctl stop squashmount fail with above errors. Kernel config didn't change recently.

Any help is really appreciated. Thank you!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Mon Jul 28, 2014 6:16 am    Post subject: Reply with quote

[quote="costel78"]When I try to stop squashmount service I get:
Code:
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error:   non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /usr/portage: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [portage]: error:   lazy umount failed: /usr/portage
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      umounting...
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      error:   non-lazy umount failed,
iul 27 23:32:16 gentoo squashmount[999]: using lazy umount of /var/db
iul 27 23:32:16 gentoo squashmount[999]: /sbin/umount.aufs:br.c:47: internal error, /var/db: Inappropriate ioctl for device
iul 27 23:32:16 gentoo squashmount[999]: * [db]:      error:   lazy umount failed: /var/db
/quote]
This is an aufs problem in your installation; perhaps recompiling aufs-util helps.
I had some problems without /sbin/umount.aufs, too, and have decided to remove it (Some hardlink information on the filesystems might get lost, but this is not needed for the portage tree). Perhaps, a variable should be added to squashmount by which one can add umount options like -I for each mountpoint separately.
Note that squashmount has no sane way to recover from this error: Removing the data in CHANGES might even crash the system after such a bug, and keeping it, leads to inconsistent states.
Quote:
It recompress despite the *.changes are well bellow threshold.

Did you take into account that if you set a huge BLOCKSIZE the threshold is reached much earlier if you have many small files (as in the portage directory?)
Does the recompression happen even if
Code:
squashmount list
shows that it should not happen? (This would be rather strange)
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 199

PostPosted: Mon Jul 28, 2014 8:11 am    Post subject: Reply with quote

Recompiling sys-fs/aufs-util-3.15_p20140728 did not help.
Recompressing on every stop did not occurred since starting from scratch.
gentoo costel # squashmount list
* [portage]: aufs (50m), modified, but will not resquash
* [db]: aufs (30m), modified, but will not resquash
To be on the safe side I'll stop using squashmount for a while (yes, it's clear now for me that IT IS NOT his fault) on portage and db. I'll keep a small directory on squashmount to monitor when problem will disappear.

Thank for support!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4116

PostPosted: Mon Jul 28, 2014 2:37 pm    Post subject: Reply with quote

costel78 wrote:
Recompiling sys-fs/aufs-util-3.15_p20140728 did not help.

I cannot help here.

In squashmount-7.8 there is support added for umount options (one of the few things squashmount was missing compared to squash_dir).

In your situation, it might help to add to your
/etc/squashmount.pl wrote:
@umount = ('-i');
(with squashmount-7.8).
This is somewhat a hack, but AFAIK the worst thing which can happen by this is that some hardlink information gets lost, i.e. files are stored in duplicate instead of once. Except for very special directories this is not an issue, and for the /usr/portage and /var/db directories, hardlinks are not used, anyway.
Back to top
View user's profile Send private message
costel78
Apprentice
Apprentice


Joined: 20 Apr 2007
Posts: 199

PostPosted: Mon Jul 28, 2014 8:32 pm    Post subject: Reply with quote

I had this evening more time to investigate. The error appear with >=sys-fs/aufs-util-3.15_p20140721. I reported bug 518418.
Unfortunately, umount options didn't change the behaviour. It still failed to umount, but I masked
Code:
>=sys-fs/aufs-headers-3.15_p20140721
>=sys-fs/aufs-util-3.15_p20140721
and squashmount it running flawless again :D
I put a squashmount restart on my update script, so will be more easy to discover if something goes wrong from now on.

Thank you for working on squashmount! Nice piece of software.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 11, 12, 13, 14  Next
Page 12 of 14

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum