TIP: Compressing portage using squashfs: initscript method

[msalerno]

After I had that issue, I had to restart the sqash_portage service since it had been unmounted. Now it's holding steady at 34 open files.

[ernov]

Is this guide up to date? I am not asking because I don't have that few hunderd mb on my hardisk, it's just Im planning to move my system to ssd. Is this howto a good way of treating ssd?

[mv]

Is this howto a good way of treating ssd?

You mean concerning reducing of numbers of writes? It depends: It will reduce the numbers of writes if you put TMPDIR and DIR_CHANGE to something else than ssd. To put TMPDIR to a ramdisk is reasonable safe, but putting DIR_CHANGE to ramdisk means that after a power loss all changes are lost.

[mattst88]

So I tried this last night, using the ebuilds from mv's overlay.

I'm getting errors during emerge --sync like

rsync: failed to set times on "/usr/portage/app-benchmarks/gtkperf": No such file or directory (2)

which is quite disappointing since I see that these were reported a few months ago. And indeed, I see 623 files open in /proc/`pidof unionfs`/fd.

What am I supposed to do? Use the unionfs-fuse-0.25-9999 ebuild? I'd rather not install mercurial just for a single package. Is there a snapshot ebuild somewhere?

[mv]

Use the unionfs-fuse-0.25-9999 ebuild?

Yes; the bug is not fixed in earlier versions, and untils unionfs-fuse-0.25 is officially released, you will probably not find it in portage.
Another way is to use aufs2 (e.g. with the live-ebuild from the mv overlay).

[mattst88]

I hope newer versions fix a lot more than just that.

Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

This simply isn't worth it.

Also, I don't have any idea what's going on here, or what's causing it.

Code: Select all

Thinkpad pkg # pwd
/var/db.changes/pkg
Thinkpad pkg # ls -lh
total 8.0K
drwxr-xr-x 3 root  root        4.0K Jan 11 16:35 app-emulation
crw-rwsr-- 1 32518 32808 1930, 1607 Feb 18  1970 dev-perlArchive-Zip-1.30
crw-rwsr-- 1 32518 32808 2063, 1567 Feb 18  1970 media-libsa52dec-0.7.4-r6
drwxr-xr-x 3 root  root        4.0K Jan 11 13:59 media-video
b-ws-ws--- 1 32518 32808 2190, 1583 Feb 18  1970 x11-libscairo-1.8.10

[mv]

Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

Did you add least add the line

Code: Select all

PORTAGE_RSYNC_EXTRA_OPTS="${PORTAGE_RSYNC_EXTRA_OPTS} --exclude=/.unionfs --exclude=.fuse_hidden*"

to /etc/make.conf? It is not optimal, but better than nothing, and without it you are screwed.

Also, I don't have any idea what's going on here, or what's causing it.

Indeed, something is very broken here - of course, this causes your problems with emerge --prune. I never had such problems (on the other hand, I normally use aufs2). Probably, this broken setting is now already stored in your archive, so I see no chance that we can reproduce how this happened. This is very serious, since /var/db/pkg contains valuable data - you have to reemerge the broken packages to fix it (and probably this re-emerging will cause file collisions and may leave orphaned files). I just can recommend to not use unionfs-fuse for a valuable directory like /var/db/pkg any more, if it does not work reliably on your machine...

[mattst88]

I gave up on this, for now at least.

Yeah, I had that line in /etc/make.conf.

You mention this bug being fixed between 0.24 and 0.25, but I don't see any commits after 0.24. http://hg.podgorny.cz/unionfs-fuse/shortlog

So, I don't understand.

[mv]

but I don't see any commits after 0.24.

It seems that unionfs-fuse is now developed in mercurial, only. There is a live ebuild in the mv overlay.

[mattst88]

So I gave this another try with a unionfs-fuse snapshot from mercurial.

I'm using it on my laptop where it seems to work fine, but I'm also using it on my portage NFS server.

There, I see lots of strange 'Manifest file not found: '/usr/portage/sys-fs/udev/Manifest'' and stale NFS file handle errors. Any ideas what's going on here?

Usually, unmounting and remounting /usr/portage/ on the clients temporarily resolves the problem.

[p04ty]

What about this message:

Code: Select all

/etc/init.d/squash_portage sync
squash_portage     | * Use of the opts variable is deprecated and will be
squash_portage     | * removed in the future.
squash_portage     | * Please use extra_commands or extra_started_commands.
squash_portage     | * Syncing portage tree ...

[mv]

What about this message ...

This should be fixed in squash_dir-11.0.

[p04ty]

What's that? How would it compare to in-kernel squashfs?

[mv]

What's that? How would it compare to in-kernel squashfs?

I do not understand your question.
squash_dir is essentially only an init-script which in the simplest configuration (when started) uses the in-kernel squashfs (together with aufs or unionfs-fuse or others) to mount a squashed directory rewritable and which (when stopped) recompresses the new directory. Of course, it expects the in-kernel squashfs (and aufs or unionfs-fuse or others) to work, although there are some fallbacks for emergency cases. (But I am not sure whether this was your question...)

[F1r31c3r]

What about this message:

Code: Select all

/etc/init.d/squash_portage sync
squash_portage     | * Use of the opts variable is deprecated and will be
squash_portage     | * removed in the future.
squash_portage     | * Please use extra_commands or extra_started_commands.
squash_portage     | * Syncing portage tree ...

Hey people this has become global across all init.d scripts for OpenRC. If you use baselayout 2 it is advisable as the warning suggests to change the "opts" to "extra_commands" but in doing this you make backward compatibility a problem. Here is the original comments on this.

http://git.overlays.gentoo.org/gitweb/? ... 8051522b4b

This means that its a good idea to update on one hand but it means this warning is not critical.
Here is how i upgraded the commands:

# cd /etc/init.d

# grep -r "opts" *

This will display all files using the opts as statement. WARNING the opts that dont hove = on eg "$fsck_opts -R" are not to be changed only them that show up at the start of the scripts saying opts=

grep -r "opts=" *

The above command shows the real opts that need changing.

Here is the quick dirty way to change them to extra_commands NOTE the = sign after the words dont leave it out or it will break your system.

# grep -lr -e "opts=" * | xargs sed -i "s/opts=/extra_commands=/g"

Now double check it change them correctly:

# grep -r "opts=" *

it should find nothing

# grep -r "extra_commands=" *

All the files should now be listed with the extra_commands and your good to go

Pick yourself a init.d script and restart it see if it works i chose iptables and sysklogd:

# /etc/init.d/iptables restart

# /etc/init.d/sysklogd restart

They should all restart without the above named warnings and or any other warning or errors.
You have successfully migrated the opts variable on your baselayout 2 OpenRC gentoo box.

[mv]

but in doing this you make backward compatibility a problem.

Since baselayout-1 is now gone from the tree and meanwhile even the oldest openrc in the tree supports extra_commands, I would not worry about backward compatibility. However, with patching init-scripts from the tree, I would wait until they are officially bumped (which does not necessarily mean a revision bump of the corresponding package - you have to check manually).

[kyoshiro]

What about this message:

# cd /etc/init.d

# grep -r "opts" *

This will display all files using the opts as statement. WARNING the opts that dont hove = on eg "$fsck_opts -R" are not to be changed only them that show up at the start of the scripts saying opts=

grep -r "opts=" *

The above command shows the real opts that need changing.

Here is the quick dirty way to change them to extra_commands NOTE the = sign after the words dont leave it out or it will break your system.

# grep -lr -e "opts=" * | xargs sed -i "s/opts=/extra_commands=/g"

Now double check it change them correctly:

# grep -r "opts=" *

it should find nothing

# grep -r "extra_commands=" *

All the files should now be listed with the extra_commands and your good to go

Please have a special look at files containing ${opts} references, e.g. /etc/init.d/reboot.sh aso.
You'll have to fix them afterwards:

# grep -lr -e "{opts}" * | xargs sed -i "s/{opts}/{extra_commands}/g"

[SlashBeast]

Code: Select all

sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i

[lpapa]

Code: Select all

sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i

Still doesn't work for some scripts, for example, sshd script uses variable "myopts" and then refers to it as ${myopts}.

I fix it adding -e 's:${myopts}:${myextra_options}:g' but I don't know if this may affect other packages which I didn't installed on my system.

[Stolz]

In order to reduce the writes of my future SSD disk I've successfully installed squash_dir via layman and configured it to squash both /urs/portage and /usr/src, saving all the files in my rotational disk. Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.

[Dont Panic]

Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.

I was going to suggest /var/db/pkg

However, that directory doesn't meet your criteria for "not of vital importance". Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered.

[Stolz]

I was going to suggest /var/db/pkg

However, that directory doesn't meet your criteria for "not of vital importance". Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered.

Thanks Dont Panic, I'm aware of it, the squash_dir README mentions it, but since it's dangerous to play with it and my raid disks already have some S.M.A.R.T errors I prefer to leave it alone

[mv]

However, that directory doesn't meet your criteria for "not of vital importance".

Actually, I have good experience with squashing it with keeping a backup (setting FILE_SQFS_OLD). This backup even saved my installation once when a kernel panic caused damage in /var/db.

/usr/share/doc or even /usr/share is a good candidate for squashing from the viewpoint of compression ratio, but there is an enormous drawback: These directories are huge and change with every package upgrade. So you should put a huge THRESHOLD and then need sufficient disk space to have the compressed directory twice on disk together with the THRESHOLD (since, of course, it will be first compresssed and the old squash removed only afterwards). Also, many things in /usr/share are somewhat vital.

Other good candidates: /usr/share/games /usr/share/texmf-dist /usr/lib/libreoffice (the first two are mentioned in the README).

Note that if you compress a lot of directories you must increase the number of loopback-devices:
Put e.g.

Code: Select all

options loop max_loop=16

into /etc/modprobe.d/loop.conf

[gringo]

Stolz pointed me to this thread and i have it now running in one of my systems, thanks a lot !

Only one thing though : if you enable the kernel-patch USE in aufs from mv overlay it will also apply the grsec one and if the kernel isnt grsec enabled it will fail to build here.
Maybe a grsec USE is needed or sth. ?

oh, and thanks also for the zram-init stuff, ive been using it for some time already

cheers

[mv]

if the kernel isnt grsec enabled it will fail to build here.

The patch should be fixed now to build in all cases.