Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
TIP: Compressing portage using squashfs: initscript method
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 7, 8, 9, 10, 11  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
msalerno
Veteran
Veteran


Joined: 17 Dec 2002
Posts: 1338
Location: Sweating in South Florida

PostPosted: Wed Jul 07, 2010 1:33 am    Post subject: Reply with quote

After I had that issue, I had to restart the sqash_portage service since it had been unmounted. Now it's holding steady at 34 open files.
Back to top
View user's profile Send private message
ernov
n00b
n00b


Joined: 28 May 2010
Posts: 55

PostPosted: Thu Aug 26, 2010 7:30 pm    Post subject: Reply with quote

Is this guide up to date? I am not asking because I don't have that few hunderd mb on my hardisk, it's just Im planning to move my system to ssd. Is this howto a good way of treating ssd?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Fri Aug 27, 2010 8:24 pm    Post subject: Reply with quote

ernov wrote:
Is this howto a good way of treating ssd?

You mean concerning reducing of numbers of writes? It depends: It will reduce the numbers of writes if you put TMPDIR and DIR_CHANGE to something else than ssd. To put TMPDIR to a ramdisk is reasonable safe, but putting DIR_CHANGE to ramdisk means that after a power loss all changes are lost.
Back to top
View user's profile Send private message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 422

PostPosted: Sun Jan 09, 2011 5:47 pm    Post subject: Reply with quote

So I tried this last night, using the ebuilds from mv's overlay.

I'm getting errors during emerge --sync like

rsync: failed to set times on "/usr/portage/app-benchmarks/gtkperf": No such file or directory (2)

which is quite disappointing since I see that these were reported a few months ago. And indeed, I see 623 files open in /proc/`pidof unionfs`/fd.

What am I supposed to do? Use the unionfs-fuse-0.25-9999 ebuild? I'd rather not install mercurial just for a single package. Is there a snapshot ebuild somewhere?
_________________
My Wiki page
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jan 09, 2011 6:10 pm    Post subject: Reply with quote

mattst88 wrote:
Use the unionfs-fuse-0.25-9999 ebuild?

Yes; the bug is not fixed in earlier versions, and untils unionfs-fuse-0.25 is officially released, you will probably not find it in portage.
Another way is to use aufs2 (e.g. with the live-ebuild from the mv overlay).
Back to top
View user's profile Send private message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 422

PostPosted: Tue Jan 11, 2011 10:40 pm    Post subject: Reply with quote

I hope newer versions fix a lot more than just that.

Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

This simply isn't worth it.

Also, I don't have any idea what's going on here, or what's causing it.

Code:
Thinkpad pkg # pwd
/var/db.changes/pkg
Thinkpad pkg # ls -lh
total 8.0K
drwxr-xr-x 3 root  root        4.0K Jan 11 16:35 app-emulation
crw-rwsr-- 1 32518 32808 1930, 1607 Feb 18  1970 dev-perlArchive-Zip-1.30
crw-rwsr-- 1 32518 32808 2063, 1567 Feb 18  1970 media-libsa52dec-0.7.4-r6
drwxr-xr-x 3 root  root        4.0K Jan 11 13:59 media-video
b-ws-ws--- 1 32518 32808 2190, 1583 Feb 18  1970 x11-libscairo-1.8.10

_________________
My Wiki page
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Jan 12, 2011 2:46 pm    Post subject: Reply with quote

mattst88 wrote:
Ignoring that I can't emerge --sync, every time I do an emerge --prune, it somehow then thinks tons of other software isn't installed anymore.

Did you add least add the line
Code:
PORTAGE_RSYNC_EXTRA_OPTS="${PORTAGE_RSYNC_EXTRA_OPTS} --exclude=/.unionfs --exclude=.fuse_hidden*"

to /etc/make.conf? It is not optimal, but better than nothing, and without it you are screwed.
Quote:
Also, I don't have any idea what's going on here, or what's causing it.

Indeed, something is very broken here - of course, this causes your problems with emerge --prune. I never had such problems (on the other hand, I normally use aufs2). Probably, this broken setting is now already stored in your archive, so I see no chance that we can reproduce how this happened. This is very serious, since /var/db/pkg contains valuable data - you have to reemerge the broken packages to fix it (and probably this re-emerging will cause file collisions and may leave orphaned files). I just can recommend to not use unionfs-fuse for a valuable directory like /var/db/pkg any more, if it does not work reliably on your machine...
Back to top
View user's profile Send private message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 422

PostPosted: Fri Feb 04, 2011 9:29 pm    Post subject: Reply with quote

I gave up on this, for now at least.

Yeah, I had that line in /etc/make.conf.

You mention this bug being fixed between 0.24 and 0.25, but I don't see any commits after 0.24. http://hg.podgorny.cz/unionfs-fuse/shortlog

So, I don't understand.
_________________
My Wiki page
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Feb 05, 2011 12:13 pm    Post subject: Reply with quote

mattst88 wrote:
but I don't see any commits after 0.24.

It seems that unionfs-fuse is now developed in mercurial, only. There is a live ebuild in the mv overlay.
Back to top
View user's profile Send private message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 422

PostPosted: Sun Jul 10, 2011 4:31 am    Post subject: Reply with quote

So I gave this another try with a unionfs-fuse snapshot from mercurial.

I'm using it on my laptop where it seems to work fine, but I'm also using it on my portage NFS server.

There, I see lots of strange 'Manifest file not found: '/usr/portage/sys-fs/udev/Manifest'' and stale NFS file handle errors. Any ideas what's going on here?

Usually, unmounting and remounting /usr/portage/ on the clients temporarily resolves the problem.
_________________
My Wiki page
Back to top
View user's profile Send private message
p04ty
n00b
n00b


Joined: 09 Apr 2011
Posts: 55

PostPosted: Sat Sep 03, 2011 9:11 am    Post subject: Reply with quote

What about this message:
Code:
/etc/init.d/squash_portage sync
squash_portage     | * Use of the opts variable is deprecated and will be
squash_portage     | * removed in the future.
squash_portage     | * Please use extra_commands or extra_started_commands.
squash_portage     | * Syncing portage tree ...
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Sep 03, 2011 5:05 pm    Post subject: Reply with quote

p04ty wrote:
What about this message ...

This should be fixed in squash_dir-11.0.
Back to top
View user's profile Send private message
p04ty
n00b
n00b


Joined: 09 Apr 2011
Posts: 55

PostPosted: Sun Sep 04, 2011 6:36 am    Post subject: Reply with quote

What's that? How would it compare to in-kernel squashfs?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Sep 04, 2011 7:32 am    Post subject: Reply with quote

p04ty wrote:
What's that? How would it compare to in-kernel squashfs?

I do not understand your question.
squash_dir is essentially only an init-script which in the simplest configuration (when started) uses the in-kernel squashfs (together with aufs or unionfs-fuse or others) to mount a squashed directory rewritable and which (when stopped) recompresses the new directory. Of course, it expects the in-kernel squashfs (and aufs or unionfs-fuse or others) to work, although there are some fallbacks for emergency cases. (But I am not sure whether this was your question...)
Back to top
View user's profile Send private message
F1r31c3r
Tux's lil' helper
Tux's lil' helper


Joined: 31 Aug 2007
Posts: 107
Location: UK

PostPosted: Sat Sep 10, 2011 12:02 am    Post subject: OpenRC Moving away from opts= to extra_commands= Reply with quote

p04ty wrote:
What about this message:
Code:
/etc/init.d/squash_portage sync
squash_portage     | * Use of the opts variable is deprecated and will be
squash_portage     | * removed in the future.
squash_portage     | * Please use extra_commands or extra_started_commands.
squash_portage     | * Syncing portage tree ...




Hey people this has become global across all init.d scripts for OpenRC. If you use baselayout 2 it is advisable as the warning suggests to change the "opts" to "extra_commands" but in doing this you make backward compatibility a problem. Here is the original comments on this.

http://git.overlays.gentoo.org/gitweb/?p=proj/openrc.git;a=commitdiff;h=df1f02ac848a010092df2d3d40b8828051522b4b

This means that its a good idea to update on one hand but it means this warning is not critical.
Here is how i upgraded the commands:

Quote:

# cd /etc/init.d

# grep -r "opts" *


This will display all files using the opts as statement. WARNING the opts that dont hove = on eg "$fsck_opts -R" are not to be changed only them that show up at the start of the scripts saying opts=

Quote:

grep -r "opts=" *


The above command shows the real opts that need changing.

Here is the quick dirty way to change them to extra_commands NOTE the = sign after the words dont leave it out or it will break your system.


Quote:

# grep -lr -e "opts=" * | xargs sed -i "s/opts=/extra_commands=/g"


Now double check it change them correctly:

Quote:

# grep -r "opts=" *

it should find nothing

# grep -r "extra_commands=" *

All the files should now be listed with the extra_commands and your good to go



Pick yourself a init.d script and restart it see if it works i chose iptables and sysklogd:

Quote:

# /etc/init.d/iptables restart

# /etc/init.d/sysklogd restart


They should all restart without the above named warnings and or any other warning or errors.
You have successfully migrated the opts variable on your baselayout 2 OpenRC gentoo box.

:D
_________________
A WikI, A collection of mass misinformation based on opinion and manipulation by a deception of freedom.
If we know the truth, then we should be free from deception (John 8:42-47 )
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Sep 10, 2011 6:35 am    Post subject: Re: OpenRC Moving away from opts= to extra_commands= Reply with quote

F1r31c3r wrote:
but in doing this you make backward compatibility a problem.

Since baselayout-1 is now gone from the tree and meanwhile even the oldest openrc in the tree supports extra_commands, I would not worry about backward compatibility. However, with patching init-scripts from the tree, I would wait until they are officially bumped (which does not necessarily mean a revision bump of the corresponding package - you have to check manually).
Back to top
View user's profile Send private message
kyoshiro
n00b
n00b


Joined: 23 Jun 2011
Posts: 1
Location: Beck'sTown

PostPosted: Sun Sep 11, 2011 11:32 am    Post subject: Re: OpenRC Moving away from opts= to extra_commands= Reply with quote

F1r31c3r wrote:
p04ty wrote:
What about this message:

Quote:

# cd /etc/init.d

# grep -r "opts" *


This will display all files using the opts as statement. WARNING the opts that dont hove = on eg "$fsck_opts -R" are not to be changed only them that show up at the start of the scripts saying opts=

Quote:

grep -r "opts=" *


The above command shows the real opts that need changing.

Here is the quick dirty way to change them to extra_commands NOTE the = sign after the words dont leave it out or it will break your system.


Quote:

# grep -lr -e "opts=" * | xargs sed -i "s/opts=/extra_commands=/g"


Now double check it change them correctly:

Quote:

# grep -r "opts=" *

it should find nothing

# grep -r "extra_commands=" *

All the files should now be listed with the extra_commands and your good to go



Please have a special look at files containing ${opts} references, e.g. /etc/init.d/reboot.sh aso.
You'll have to fix them afterwards:

# grep -lr -e "{opts}" * | xargs sed -i "s/{opts}/{extra_commands}/g"
Back to top
View user's profile Send private message
SlashBeast
Retired Dev
Retired Dev


Joined: 23 May 2006
Posts: 2922

PostPosted: Sun Sep 11, 2011 2:28 pm    Post subject: Reply with quote

Code:
sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i
Back to top
View user's profile Send private message
lpapa
n00b
n00b


Joined: 19 Sep 2011
Posts: 1

PostPosted: Mon Sep 19, 2011 1:48 am    Post subject: Reply with quote

SlashBeast wrote:
Code:
sed -e 's:opts=:extra_commands=:g' -e 's:${opts}:${extra_commands}:g' -e 's:$opts:$extra_commands:g' /etc/init.d/* -i


Still doesn't work for some scripts, for example, sshd script uses variable "myopts" and then refers to it as ${myopts}.

I fix it adding -e 's:${myopts}:${myextra_options}:g' but I don't know if this may affect other packages which I didn't installed on my system.
Back to top
View user's profile Send private message
Stolz
Moderator
Moderator


Joined: 19 Oct 2003
Posts: 3028
Location: Hong Kong

PostPosted: Tue Oct 11, 2011 11:00 pm    Post subject: Reply with quote

In order to reduce the writes of my future SSD disk I've successfully installed squash_dir via layman and configured it to squash both /urs/portage and /usr/src, saving all the files in my rotational disk. Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.
Back to top
View user's profile Send private message
Dont Panic
Guru
Guru


Joined: 20 Jun 2007
Posts: 322
Location: SouthEast U.S.A.

PostPosted: Wed Oct 12, 2011 3:19 pm    Post subject: Reply with quote

Stolz wrote:
Are there any other directories that are good candidates to be squashed? may be /usr/share/doc?. With "good candidates" I mean they use have a lot of small text files and use to involve lot of writes to the disk but also they are not of vital importance.

I was going to suggest /var/db/pkg

However, that directory doesn't meet your criteria for "not of vital importance". Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered. :)
Back to top
View user's profile Send private message
Stolz
Moderator
Moderator


Joined: 19 Oct 2003
Posts: 3028
Location: Hong Kong

PostPosted: Wed Oct 12, 2011 3:54 pm    Post subject: Reply with quote

Dont Panic wrote:
I was going to suggest /var/db/pkg

However, that directory doesn't meet your criteria for "not of vital importance". Bad things will happen if the information in /var/db/pkg gets out of sync or is clobbered. :)


Thanks Dont Panic, I'm aware of it, the squash_dir README mentions it, but since it's dangerous to play with it and my raid disks already have some S.M.A.R.T errors I prefer to leave it alone :)
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Wed Oct 12, 2011 4:23 pm    Post subject: Reply with quote

Stolz wrote:
However, that directory doesn't meet your criteria for "not of vital importance".

Actually, I have good experience with squashing it with keeping a backup (setting FILE_SQFS_OLD). This backup even saved my installation once when a kernel panic caused damage in /var/db.

/usr/share/doc or even /usr/share is a good candidate for squashing from the viewpoint of compression ratio, but there is an enormous drawback: These directories are huge and change with every package upgrade. So you should put a huge THRESHOLD and then need sufficient disk space to have the compressed directory twice on disk together with the THRESHOLD (since, of course, it will be first compresssed and the old squash removed only afterwards). Also, many things in /usr/share are somewhat vital.

Other good candidates: /usr/share/games /usr/share/texmf-dist /usr/lib/libreoffice (the first two are mentioned in the README).

Note that if you compress a lot of directories you must increase the number of loopback-devices:
Put e.g.
Code:
options loop max_loop=16
into /etc/modprobe.d/loop.conf
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3793

PostPosted: Fri Oct 14, 2011 1:21 pm    Post subject: Reply with quote

Stolz pointed me to this thread and i have it now running in one of my systems, thanks a lot !

Only one thing though : if you enable the kernel-patch USE in aufs from mv overlay it will also apply the grsec one and if the kernel isnt grsec enabled it will fail to build here.
Maybe a grsec USE is needed or sth. ?

oh, and thanks also for the zram-init stuff, ive been using it for some time already :-)

cheers
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Oct 15, 2011 3:55 pm    Post subject: Reply with quote

gringo wrote:
if the kernel isnt grsec enabled it will fail to build here.

The patch should be fixed now to build in all cases.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 7, 8, 9, 10, 11  Next
Page 8 of 11

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum