TIP: Compressing portage using squashfs: initscript method

[gringo]

The patch should be fixed now to build in all cases.

nice, thanks a lot !

cheers

[costel78]

Is it possible to port the init script to systemd ?
I looked over actual openrc script, but it is too far complex to simple migrating the commands

[mv]

Is it possible to port the init script to systemd?

I don't know. AFAIK systemd does not have proper dependencies and probably also not a standard route during shutdown which is crucial for the script.
Actually, I consider the whole systemd a very bad idea (running permanently daemons which take resources only to save 1-2 seconds during booting is not sane), so I hope that I will never be forced to use it, and I am not at all interested in porting something to a system with a bad concept.

I looked over actual openrc script, but it is too far complex to simple migrating the commands

Essentially, it is the "start" and "stop" which would have to be called at the proper place. The other provided command are only for wrapper scripts to check the current state (e.g. whether it would be saved). Probably one could put a part in a "library" which is sourced.

[costel78]

Thank you for your answer!
Yes, I already tried to simulate the logics behind and, so far, I found no way to start tar and bzip compress of the portage tree BEFORE /usr/portage was umounted.
What was the simulation:
- a separate partition for /usr/portage (without distfiles, of course) mounted automatically at boot via initscript (noauto in fstab)
- during shutdown compress it (tar -cjpf ....) and umount it
As you said, I found no way to prevent systemd to try to umount the partition and kill tar process.
It seems that all thing it's to risky.

[mv]

As you said, I found no way to prevent systemd to try to umount the partition and kill tar process.

At least the mount and umount commands from sys-apps/util-linux provide a way to call helper scripts for mount and umount of certain filesystem types. I have never dealt with them and do not know how compatible this setting is. But it might be a possible to invent a "new" filesystem type and to do the actual action in these helper scripts. However, this needs to be well-tested: e.g. aufs also uses such helper scripts, so "recursive" callings of such scripts must be possible.
The advantage of this approach would be that it works with any init-system, the disadvantage is that it probably works only with util-linux.
Currently, I have no time to implement such a thing.

[slycordinator]

I'm having a problem with mv's squash_dir initscript.

My portage squashfs file is at /squashed_dirs/portage.sqfs and the portage.readonly and portage.changes are placed inside that same directory. And I named the portage initscript squash_portage

My portage.changes directory has a whole bunch of subdirs in it currently, since I did an "emerge --sync" earlier, but if I do "/etc/init.d/squash_portage restart" I simply get:

Code: Select all

/etc/init.d/squash_portage restart
 * Unmounting /usr/portage ...                                                                   [ ok ]
 * Mounting /squashed_dirs/portage.sqfs as /usr/portage ...                                      [ ok ]

And the portage.sqfs is not recompressed with the portage.changes directory containing the same stuff it did previously.

[mv]

And the portage.sqfs is not recompressed

This is strange. I cannot reproduce it here. Are you sure that you do not have a .no-save file and not set any threshold?
If yes, please help me debugging:

1. Which version of squash_dir are you using? To simplify debugging, please try with the latest from the mv overlay (from today).
2. Is the output of

   Code: Select all

   /etc/init.d/squash_portage print_dir_change

   indeed your portage.changes which contains data? In particular, does

   Code: Select all

   ls $(/etc/init.d/squash_portage print_dir_change)

   show that content?
3. Does some of the following two commands output "1" to stderr?

   Code: Select all

   /etc/init.d/squash_portage need_squash
   /etc/init.d/squash_portage will_squash

[slycordinator]

Note: This happens sporadically. Like when I shutdown the computer last night, the portage squashfs file got recompressed and the ones I've got for /var/db and /usr/share/texmf-dist didn't, but at least now (using version 11.6) doing a "/etc/init.d/squash_db restart" made it recompress on the first try. And the need_squash and will_squash variables showed that the tex directory will be recompressed. Guess it's fixed but not sure, given how sporadic it would happen.

edit: PEBKAC. Think it was the threshold setting.

[js08]

Hi,

I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1


emerge --sync produces the following error:

Code: Select all

app-backup/backup-manager/files/
app-backup/backupninja/
app-backup/backuppc/
app-backup/backuppc/files/
app-backup/backuppc/files/3.2.0/
app-backup/bacula/
app-backup/bacula/ChangeLog
      40.52K 100%   59.51kB/s    0:00:00 (xfer#24, to-check=1056/4230)
app-backup/bacula/Manifest
       8.54K 100%   12.22kB/s    0:00:00 (xfer#25, to-check=1055/4230)
app-backup/bacula/bacula-5.2.5.ebuild
      10.09K 100%   14.42kB/s    0:00:00 (xfer#26, to-check=1048/4230)
app-backup/bacula/bacula-5.2.6.ebuild
      10.09K 100%   14.09kB/s    0:00:00 (xfer#27, to-check=1047/4230)
rsync: rename "/usr/portage/app-backup/bacula/.bacula-5.2.6.ebuild.rJDBEZ" -> "app-backup/bacula/bacula-5.2.6.ebuild": Software caused connection abort (103)
app-backup/bacula/files/
app-backup/bacula/files/5.0.2/
app-backup/bacula/files/5.0.3/
app-backup/bacula/files/5.2.3/
app-backup/boxbackup/
app-backup/boxbackup/files/
app-backup/ccollect/
app-backup/cdbackup/
app-backup/cdbkup/
app-backup/cpdup/
app-backup/cpdup/files/
app-backup/dar/
app-backup/dar/files/
rsync: failed to set times on "/usr/portage/app-crypt": Software caused connection abort (103)
rsync: opendir "/usr/portage/app-crypt" failed: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat "/usr/portage/app-crypt/metadata.xml": Transport endpoint is not connected (107)
rsync: recv_generator: mkdir "/usr/portage/app-crypt/WiRouterKeyRec" failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/usr/portage/app-crypt/acr38u" failed: Transport endpoint is not connected (107)
...

[mv]

Hi,

I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1


emerge --sync produces the following error:

And it worked before (i.e. with other kernels)? Then it looks to me like a race condition problem in unionfs-fuse or an incompatibility of fuse with the newest kernels.
If the start of the error message is not reproducible, I would guess on the former and suggest that you report the problem upstream at unionfs-fuse.

[js08]

Hi,

I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1


emerge --sync produces the following error:

And it worked before (i.e. with other kernels)? Then it looks to me like a race condition problem in unionfs-fuse or an incompatibility of fuse with the newest kernels.
If the start of the error message is not reproducible, I would guess on the former and suggest that you report the problem upstream at unionfs-fuse.

yes, everything worked fine until 3-21-2012.
and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine.

[mv]

and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine.

As I mentioned, please report upstream at unionfs-fuse - this needs help by the corresponding specialist (it probably is some race issue).

[Massimo B.]

Hi, since Portage gets too slow on my old ppc Notebook I found this Tip...
Where is the recent version of these init scripts maintained and who does it currently? I just found http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree, there are already newer versions than the original author wrote in the first post of this thread.

Usually I avoid using non-official approaches which tend to be un-maintained after a while, but since removing the SquashFS approach for the default Portage is just easy and the benefits of this tweak are just too nice.
Wouldn't it be nice to make the Howto a bit more official and centrally maintained by creating an ebuild with the init script in some of the overlays, so I don't need to copy/paste the init-script code from the browser into the editor?

[mv]

The project (squash_dir, consisting not only of the init-scripts but also of a shell-wrapper with zsh-completion and an extensive README) is currently maintained on github.
The corresponding ebuild is in the mv overlay (available by layman).

[Massimo B.]

Sorry that I overlooked the middle of this thread. I'm also using sys-fs/squash_dir now from your mv overlay. Thank you very much for maintaining these scripts, nice piece of work, especially the very good README.
The only issue for now is that aufs3 sources don't build on [topic=935052]PPC[/topic]. On amd64 its working fine.
Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance (say slow 1core cpu and slow drive)? This is more important for me than space for some slow old machines and its the main reason that made me come to this thread, after reading Portage_tips.

[mv]

The only issue for now is that aufs3 sources don't build on [topic=935052]PPC[/topic].

You might try the git sources (aufs-99999999.3, also available from the mv overlay). Let us hope that the kernel developers come to their mind and finally include aufs or at least overlayfs, although the latter has several drawbacks compared to aufs.

Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance

For compression, lzo is undoubtfully the fastest (and least effective) and xz the slowest (and most effective). However, except for the time on shutdown only decompression time plays a role. Concerning decompression, lzo is also the fastest, but surprisingly xz is not much slower. Since xz needs to access less data (because it compresses much better), it can be that on some systems (especially those with not much memory or huge cache) xz is even the winner for decompression, speedwise. Your really have to try.
On the other hand, if you have a really slow system and a huge directory, it can be that compression time of xz is really inacceptable - on some systems, I end up with gtip for this reason...

[Massimo B.]

Because I can't afford long startup or stop time, I restart the squashing on daily base:

Code: Select all

$ cat /etc/cron.daily/1squashfs_flush
#!/usr/bin/env bash
for rc in /etc/runlevels/boot/squash_*; do
    if [[ 0 != "$($rc will_squash 2>&1 >/dev/null)" ]]; then
        $rc restart
    fi
done

Synchronizing a 1 week old Portage made this difference:

Code: Select all

du -sh /usr/portage*
311M	/usr/portage
52M	/usr/portage.changes
280M	/usr/portage.readonly
67M	/usr/portage.sqfs

...which already exceeded my threshold of 40M.
Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only?

I have 3 squash_dir setups now: /usr/portage, /var/db/ and /var/lib/layman.

Because I need to use webrsync on some hosts, I've seen that after a new webrsync of the same snapshot it seems that all has been re-written:

Code: Select all

$ du -sh /usr/portage*
711M	/usr/portage
711M	/usr/portage.changes
280M	/usr/portage.readonly
67M	/usr/portage.sqfs

Let's see if app-portage/emerge-delta-webrsync is different here.

[mv]

Because I can't afford long startup or stop time, I restart the squashing on daily base:

Are you aware that there is a squash_dir wrapper for this?

squash_dir restart

should do the same as your script (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.

Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only?

Exactly.

I've seen that after a new webrsync of the same snapshot it seems that all has been re-written.

This is because of the COW mechanism of aufs: If something changes in a file, even if it is only the filestamp or some attributes, the file must be copied into the .changes directory. I guess webrsync changes all filestamps, at least temporarily. With other union-type filesystem the necessary COW might be even more intensive, e.g. (IIRC this was the case in some versions of unionfs-fuse) if only the directory filestamp changes then the whole directory might need to be written to *.changes - it really depends which mechanisms are used to merge the directories.

[Massimo B.]

Are you aware that there is a squash_dir wrapper for this?

squash_dir restart should do the same as your s

cript (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.

No I wasn't, thanks. squash_dir restart really simplifies that task or cronjob.

[Massimo B.]

With some manual work to get aufs3 building on ppc as described in the bug [1], your squash_dir is working fine. You could add ~ppc keywords to squash_dir and runtitle.
[1] [bug=433497]Bug 433497 - sys-fs/aufs3-3_p20120813: unrecognized command line option[/bug].

[Massimo B.]

When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts. This is ok.
After I got the aufs module built and loaded, squash_dir restart or start do not work.

Code: Select all

$ squash_dir restart
/etc/init.d/squash_db is not running
/etc/init.d/squash_layman is not running
/etc/init.d/squash_local_portage is not running
/etc/init.d/squash_portage is not running

$ squash_dir start
Starting db...
squash_db           | * Mounting /var/db.sqfs as /var/db ...
squash_db           |mount: /var/db.sqfs is already mounted
squash_db           | * Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]
squash_db           |rmdir: failed to remove `/var/db.readonly': Device or resource busy                   [ !! ]_db           |
squash_db           | * ERROR: squash_db failed to start
Skipping /etc/init.d/squash_layman due to previous error
Skipping /etc/init.d/squash_local_portage due to previous error
Skipping /etc/init.d/squash_portage due to previous error

I first needed to umount all read.only stuff to get it starting. Maybe you can add some error safety here.

[mv]

When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts.

The problem is that it is not clear what to do:
One possibility is to add an "umount" option which tries to umount even if the start failed. However, this can have horrible undesired effects if the start failed for a different reason or if directories with the temporary name feature are involved.
The other possibility is to start anyway successfully in such a situation, because the directory is mounted but just not writable.
The disadvantage of the latter is that you will not realize that something is broken unless you study the logs (or actually attempt to write).
The real problem is that it is not possible to report to openrc a "partial success".

[Massimo B.]

I would like to improve performance, throughput and disk usage on a WebDAV server, which I mount via net-fs/davfs2. Now I think about if it is a good idea to store the content with squash_dir there.
Disadvantage would be of course that I can't view the files from the web-interface anymore. But I use the WebDAV only as central synchronisation for different sites using net-misc/unison.
Now the problem would be that maybe any change on the re-compressed squashfs would lead to a complete upload of the whole file again instead of sending delta only. Maybe this reason makes squash_dir the wrong solution for that, what do you think?

[mv]

Maybe this reason makes squash_dir the wrong solution for that, what do you think?

Yes, I guess this is the wrong solution. Maybe you can use a VCS like git instead?

[Gordex]

tried it today but didnt get it to work properly :/
Compression-var seems to be ignored..I got gzip compression all the way
an error message indicated that it tried to umount the readonly before the aufs when shutdown
openrc doesnt seem to launch it altho added to default runlevel..as there was no output in the rc.log and no proper mounts after reboot
I guess the script is a bit outdated
anyway nice idea man!