TIP: Compressing portage using squashfs: initscript method

[mv]

If umount fails, squash_dir-10.6 will now sleep for 1s and retry (and repeat this procedure). Moreover, it attempts now to check whether the directory is already umounted and skip the umount if this is the case. I guess, in most cases this is the desired behavior although perhaps some strange error situations are not cought now.

[js08]

Are you sure its 0.24 and not 0.25alpha? 0.24 is mostly a bugfix release to 0.23. But 0.25 simplifies path building (unionfs internal) and handles MAX_PATH_LEN better. But this change might have introduced bugs.
However, I'm a Debian user and only found that thread here, as I had been curious what Gentoo is using unionfs-fuse for. So that means I am not familiar with what you are doing.

With both, first I got the empty /usr/portage directory with 0.25-xxx, then I switrched back to 0.24, same effect several times and now I'm using 0.23 again... I think gentoo's /usr/portage-directory is really good for unionfs-fuse stress-testing. I contains >100k small files so a rsync with the original /usr/portage-tree is from the network point of view normally really fast and the unionfs-fuse has a lot to do. And during such rsyncs /usr/portage becomes inaccessible and a "ls /usr/portage" shows an empty directory...

There "busy" problem on umount?

The busy umount problem has nothing to do with the 0.24/0.25alpha empty /usr/portage directory problem.

[aakef]

Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse? I assume the kernel is definitely the same? Are you able to create a test case for me? So I already know the unionfs-fuse command and now I know that ?it is not only the umount issue. But I still do not have the slightest idea, what is actually your problem. Saying that 0.23 works, but that 0.24/0.25 fail is not sufficient, I'm afraid.

1) How does it fail, what are the symptoms?

2) If it fails, can you check using 'ps ax | grep unionfs' if unionfs is still running?

3) Depending on 2)

a) If it is running, does it show in 'top' a high cpu usage?

b) If it is not running anymore, you should get

#define ENOTCONN 107 /* Transport endpoint is not connected */
bernd@bathl Ph.D>errno 108
#define ESHUTDOWN 108 /* Cannot send after transport endpoint shutdown */


on accessing the union path

4) What do I need to do to reproduce it?

4.1) If it is not easy to reproduce, are you willing to run debug commands yourself? So recompile it with debug symbols (-g) and -DDebug (using cmake that should be easy). And then runing it either in valgrind or gdb and provide me the output.


Thanks,
Bernd

[js08]

If umount fails, squash_dir-10.6 will now sleep for 1s and retry (and repeat this procedure). Moreover, it attempts now to check whether the directory is already umounted and skip the umount if this is the case. I guess, in most cases this is the desired behavior although perhaps some strange error situations are not cought now.

works!

[mv]

Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse?

fuse is not slotted in gentoo, so the answer will almost surely be "yes". Moreover, libfuse has not been upgraded for a while in gentoo: The latest stable (and simultaneously latest unstable) version of fuse in the portage tree is 2.8.1 which is therefore almost surely what js08 is using.

[js08]

Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse? I assume the kernel is definitely the same? Are you able to create a test case for me? So I already know the unionfs-fuse command and now I know that ?it is not only the umount issue. But I still do not have the slightest idea, what is actually your problem. Saying that 0.23 works, but that 0.24/0.25 fail is not sufficient, I'm afraid.

1) How does it fail, what are the symptoms?

2) If it fails, can you check using 'ps ax | grep unionfs' if unionfs is still running?

3) Depending on 2)

a) If it is running, does it show in 'top' a high cpu usage?

b) If it is not running anymore, you should get

#define ENOTCONN 107 /* Transport endpoint is not connected */
bernd@bathl Ph.D>errno 108
#define ESHUTDOWN 108 /* Cannot send after transport endpoint shutdown */


on accessing the union path

4) What do I need to do to reproduce it?

4.1) If it is not easy to reproduce, are you willing to run debug commands yourself? So recompile it with debug symbols (-g) and -DDebug (using cmake that should be easy). And then runing it either in valgrind or gdb and provide me the output.


Thanks,
Bernd

on one host if have now re-installed the 0.25alpha:

0) /usr/portage mounted - ls shows the contents - fine.
1) emerge --sync
result:

Code: Select all

>>> Starting rsync with rsync://81.91.242.10/gentoo-portage...
>>> Checking server timestamp ...
----------------------------------------------------------------
|               .d88888b.           888                        |
|              d88P" "Y88b          888                        |
|              888     888          888                        |
|              888     888 888  888 88888b.   .d88b.           |
|              888     888 888  888 888 "88b d8P  Y8b          |
|              888 Y8b 888 888  888 888  888 88888888          |
|              Y88b.Y8b88P Y88b 888 888 d88P Y8b.              |
|               "Y888888"   "Y88888 88888P"   "Y8888           |
|                     Y8b                                      |
|==============================================================|
|                :: QUBE MANAGED SERVICES LIMITED ::           |
|==============================================================|
|                     http://www.qubenet.net/                  |
----------------------------------------------------------------
|ipv4             : 81.91.242.10                               |
|ipv6             : [NOT AVAILABLE AT PRESENT - coming soon..] |
|servername       : mirror.qubenet.net.                        |
|bandwidth        : 1 Gbit/s                                   |
|server specs     : VMWare vSphere, 2GB RAM, Gentoo Linux OS   |
|server location  : QUBE LN1 :: London, United Kingdom         |
|contact          : Qube Support :: mirror-admin@qubenet.net   |
|phone            : +44-207-150-3810 (NOC)                     |
|--------------------------------------------------------------|
|Please do not abuse this mirror.                              |
|It is provided as a free service, thank you!                  |
|--------------------------------------------------------------|

receiving incremental file list
timestamp.chk
          32 100%   31.25kB/s    0:00:00 (xfer#1, to-check=0/1)

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 113
Total bytes received: 1890

sent 113 bytes  received 1890 bytes  1335.33 bytes/sec
total size is 32  speedup is 0.02
----------------------------------------------------------------
|               .d88888b.           888                        |
|              d88P" "Y88b          888                        |
|              888     888          888                        |
|              888     888 888  888 88888b.   .d88b.           |
|              888     888 888  888 888 "88b d8P  Y8b          |
|              888 Y8b 888 888  888 888  888 88888888          |
|              Y88b.Y8b88P Y88b 888 888 d88P Y8b.              |
|               "Y888888"   "Y88888 88888P"   "Y8888           |
|                     Y8b                                      |
|==============================================================|
|                :: QUBE MANAGED SERVICES LIMITED ::           |
|==============================================================|
|                     http://www.qubenet.net/                  |
----------------------------------------------------------------
|ipv4             : 81.91.242.10                               |
|ipv6             : [NOT AVAILABLE AT PRESENT - coming soon..] |
|servername       : mirror.qubenet.net.                        |
|bandwidth        : 1 Gbit/s                                   |
|server specs     : VMWare vSphere, 2GB RAM, Gentoo Linux OS   |
|server location  : QUBE LN1 :: London, United Kingdom         |
|contact          : Qube Support :: mirror-admin@qubenet.net   |
|phone            : +44-207-150-3810 (NOC)                     |
|--------------------------------------------------------------|
|Please do not abuse this mirror.                              |
|It is provided as a free service, thank you!                  |
|--------------------------------------------------------------|

receiving incremental file list
rsync: failed to set times on "/usr/portage/app-accessibility": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/dasher": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/emacspeak": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/espeak": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/festival-freebsoft-utils": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/festival-it": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/gnome-speech": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/gnopernicus": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/mbrola": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/morseall": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/orca": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/perlbox-voice": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/pidgin-festival": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/sound-icons": No such file or directory (2)
rsync: failed to set times on "/usr/portage/app-accessibility/speakup-utils": No such file or directory (2)
...

2)
ps ax | grep unionfs

Code: Select all

2539 pts/5    S+     0:00 grep --colour=auto unionfs
25419 ?        Ssl    0:01 /usr/sbin/unionfs -o cow -o allow_other -o use_ino -o nonempty /usr/portage.changes=RW:/usr/portage.readonly=RO /usr/portage

3a)
top shows ca. 40% processor-usage, also when try to rsync again

4) I think you don't need gentoo linux. Get a snapshot of the portage tree from http://gentoo.inode.at/snapshots/, extract it, make a squashfile... and then execute a rsync with one of the gentoo mirrors. I dont know exactly what rsync options emerge --sync, eix-sync use but I think it's more or less this command line...
rsync -av --exclude=/.unionfs rsync://rsync.europe.gentoo.org/gentoo-portage /usr/portage

[js08]

Are you sure 0.23 vs. 0.24/0.25 are linked against the very same libfuse?

fuse is not slotted in gentoo, so the answer will almost surely be "yes". Moreover, libfuse has not been upgraded for a while in gentoo: The latest stable (and simultaneously latest unstable) version of fuse in the portage tree is 2.8.1 which is therefore almost surely what js08 is using.

fuse - 2.8.1,
kernel (at the moment 2.6.34-gentoo, but before a vanilla 2.6.34)

[aakef]

Hmm, can't reproduce it

/dev/loop0 on /tmpa/gentoo/portage.ro type squashfs (rw)
unionfs on /tmpa/gentoo/portage type fuse.unionfs (rw,nosuid,nodev,allow_other,default_permissions)


/home/bernd/src/unionfs-fuse/unionfs/BUILD/src/unionfs -d -o cow -o allow_other -o use_ino -o nonempty -o hide_meta_dir /tmpa/gent/portage.rw=RW:/tmpa/gentoo/portage.ro=RO /tmpa/gentoo/portage

bathl:/tmpa/gentoo# /home/bernd/src/unionfs-fuse/unionfs/BUILD/src/unionfs --version
Debug mode, log will be written to ./unionfs_debug.log
unionfs-fuse version: 0.25
FUSE library version: 2.7.4
fusermount version: 2.7.4
using FUSE kernel interface version 7.8

bathl:~# rsync -a rsync://rsync.europe.gentoo.org/gentoo-portage /tmpa/gentoo/portage/

-========== B Y T E M A R K H O S T I N G M I R R O R ==========-

bathl:~# rsync -a rsync://rsync.europe.gentoo.org/gentoo-portage /tmpa/gentoo/portage/
Server name: tux.rainside.sk
IP address: 212.89.225.155
System: Intel(R) Pentium(R) D CPU 2.80GHz, 4GB RAM
Bandwidth: 1 Gbit/s
Server location: Bratislava, Slovakia
Contact: kominek@rainside.sk

Please note: common gentoo-netiquette says you should not sync
more than once a day. Users who abuse the rsync.gentoo.org
rotation may be added to a temporary ban list.


I think I need to update libfuse and to check what then happens.

[mv]

I think it is related with the way rsync accesses files - I suppose it keeps some filehandle open which collides with cow or something similiar. Here is a script for which rsync reports a nonexistent dir, although it exists. I tried the same without the squashfs, and the error did no occur. However, with aufs2 there is also no such error, so it is somewhat the interplay between squashfs, unionfs-fuse, and rsync which matters:

Code: Select all

#! /bin/sh
T="`mktemp -d /tmp/XXXXXXXX`"
cd -- "$T"
mkdir readonly readonly/a changes union
echo 1 >readonly/a/1
cp -a readonly reference
mksquashfs readonly sqfs >/dev/null
rm -rf readonly/a
mount -t squashfs -o loop,ro -- "$T/sqfs" "$T/readonly"
# The touch is only to force a different time so that rsync wants to set times:
touch -d 10:30 reference/a
unionfs -o cow -o allow_other -o use_ino -o nonempty -o hide_meta_dir \
	"$T/changes=RW:$T/readonly=RO" "$T/union"
cd "$T/union"
set -x
/usr/bin/rsync -a -- "$T/reference/" .
set +x
cd /tmp
umount -i -- "$T/union"
sleep 1
umount -d -i -- "$T/readonly"
rm -rf -- "$T"

(I am sorry, but this week I am too busy and will not have time to discuss further).

Edit: Made the example slightly more "minimal". Note that if "a" is an empty directory, the problem does not occur either, so this is really the minimal test case.

[js08]

Hmm, can't reproduce it

I think I need to update libfuse and to check what then happens.

strange.
- with unionfs-fuse versions >0.23 it is so reproducable that I'm not able update the portage tree - every rync fails.
- I tried also the next older version in the portage tree libfuse 2.7.4 - same rsync issue

I still think that my environment is not very special. It's a simple 64bit pc linux environment which runs on a pc (amd) and on a laptop (intel)

so my next steps is to compile a debug version and let's see what happens

[aakef]

Martin, thanks a bunch for your reproducer script. While running it in debug mode, I noticed a few new bugs in 0.25. I could only reproduce it with the script after I added this line

ln -s readonly/a/1 readonly/a/link_1

I think I will add the script to our regression test script.

So now the issue: http://podgorny.cz/~bernd/hg/hgwebdir.c ... 2f284a0fff

I have not the slightest idea how that could ever work in 0.23, but this is a kernel (or glibc) bug and not unionfs related.

Please check out the recent 0.25alpha version and please report if it works or not.

Thanks,
Bernd

[js08]

Is it possible to upload/send you the output of ddd/gdb and the concerning console output?

I have created at least one which ends in an empty /usr/portage directory.

EDIT:
Since I didn't find an upload page here or on radek's website I uploaded the tar-file to uploaded.to

http://ul.to/7j52bl/unionfs-fuse.100525.out.tgz

[aakef]

Thanks for the logs, I think you used Martins previously compiled version, right? The bug will be most probably gone, if you rebuild using the current hg-version (0.25 branch). For example here:

utimens /app-dicts/stardict-freedict-eng-swe 1274740792.000000000
1274738977.000000000
unique: 16404, error: -2 (No such file or directory), outsize: 16
unique: 16405, opcode: OPENDIR (27), nodeid: 4868, insize: 48

Definitely 'fixed' in 0.25-hg. As it is not a unionfs bug, but a kernel/glibc issue, I simply disabled the error for now. If the issue still exists, I would appreciate further debug logs. However, debugging must be enabled for the build, that way unionfs will provide lots of information what it does internally. Maybe it will be a good idea, if I simply also check for "-d" in unionfs (so far libfuse reads it) and enable debug information without the need to recompile...

Thanks,
Bernd

[js08]

Thanks for the logs, I think you used Martins previously compiled version, right?

exactly !

[aakef]

Hi all,

just would like to ask if you have got a chance to try out the recent 0.25-hg version already?

Thanks,
Bernd

[mv]

just would like to ask if you have got a chance to try out the recent 0.25-hg version already?

Last time I re-emerged, I got the version from June 1, ChangeSet 462. It worked fine.

[js08]

works fine.

[aakef]

Great, thanks for testing it. Btw, the 'official' to hide the .unionfs directory is "-o hide_meta_files". The reason are .fuse... files, which are created, if an open file gets deleted. A ubuntu user who also works with squashfs run into those files and so we need to blacklist those as well. "-o hide_meta_dir" still works, but is deprecated.


Cheers,
Bernd

[mv]

the 'official' to hide the .unionfs directory is "-o hide_meta_files".

This is known

[msalerno]

sys-fs/squash_dir-10.8
sys-fs/squashfs-tools-4.0
sys-fs/fuse-2.8.1
sys-fs/unionfs-fuse-0.24

I'm currently using these scripts to compress portage. I had no issue with the install, but the majority of the time when I do an emerge --sync or an eix-sync, I get lots of "failed: Too many open files (24)"

"rsync: mkstemp "/usr/portage/xfce-base/xfce4-meta/.Manifest.gr8fut" failed: Too many open files (24)"

"lsof -u root / | wc -l" currently shows >1700 open files. I have not messed with my limits.

[aakef]

Hmm, too many open files by rsync? That's a bit weird. When you get this, could you please check /proc/`pidof unionfs`/fd if there are open filedescriptors left over? If not, or if those reduce within a few seconds, then it is linux cache and the gentoo script will need to specify the option "-o max_files=number" (e.g number=16384").
But if those FDs stay, then it is bug in unionfs-fuse. If that is true, could you please tell me what kind of file those are (the links /proc/fs/fd will tell you).

Yesterday I also pushed an new debug patches into the 0.25alpha branch, if you would enable debugging with that "-o /tmp/debug_file", we could check why those files are not closed.


Thanks,
Bernd

[samonli]

sys-fs/squash_dir-10.8
sys-fs/squashfs-tools-4.0
sys-fs/fuse-2.8.1
sys-fs/unionfs-fuse-0.25-9999

/etc/conf.d/squash_portage:
DIRECTORY="/usr/portage"
DIR_CHANGE="${DIRECTORY}.changes"
DIR_SQUASH="${DIRECTORY}.readonly"
THRESHOLD=40000

/etc/init.d/squash_portage start:

Code: Select all

squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...
squash_portage       |mount: unknown filesystem type 'aufs'
squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

mount:

Code: Select all

/dev/loop0 on /usr/portage.readonly type squashfs (ro)
unionfs on /usr/portage type fuse.unionfs (rw,nosuid,nodev,allow_other,default_permissions)

after set "ORDER=unionfs-fuse unionfs funionfs aufs" in /etc/conf.d/squash_portage

/etc/init.d/squash_portage start:

Code: Select all

squash_portage       |Failed to open //funionfs/: No such file or directory. Aborting!
squash_portage       |
squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...
squash_portage       |mount: unknown filesystem type 'aufs'
squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

Does this ok?
thanks

[mv]

/etc/init.d/squash_portage start:

Code: Select all

squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...
squash_portage       |mount: unknown filesystem type 'aufs'
squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

This is ok: It is first attempted to mount with aufs (which fails, since apparently you do not have the kernel patched), so it falls back to the second choice in the default order which apparently succeeds (so quash_portage has probably succesfully started).

after set "ORDER=unionfs-fuse unionfs funionfs aufs" in /etc/conf.d/squash_portage

/etc/init.d/squash_portage start:

Code: Select all

squash_portage       |Failed to open //funionfs/: No such file or directory. Aborting!
squash_portage       |
squash_portage       | * Mounting /usr/portage.sqfs as /usr/portage ...
squash_portage       |mount: unknown filesystem type 'aufs'
squash_portage       | * Failed mounting /usr/portage.changes with aufs [exit with 32]

However, this I cannot reproduce. Are you sure that you did not start with ORDER="funionfs aufs ..."? In my test, after successfully mounting with unionfs-fuse, it exits as it should (and if unionfs-fuse fails,l it prints an error message before trying unionfs, then also failing and printing an error, and only then trying unionfs).

[samonli]

Sorry,my mistake,i miss the "" for ORDER in /etc/conf.d/squash_portage

Correct:
ORDER="unionfs-fuse unionfs funionfs aufs"

my:
ORDER=unionfs-fuse unionfs funionfs aufs

Thanks very much!

[msalerno]

Currently getting too many open files.

/proc/`pidof unionfs`/fd
ls -1 | wc -l
1024