[tool]portage-bashrc-ng - modular portage's bashrc

[Ferdinando]

Hello everyone.

This tool was first born as a spin-off of one of Fonderiadigitale's projects, portage-bashrc, a bashrc which is imported by portage and adds it the capability to compile in a dedicated tmpfs partition (using ram + swap instead of the live filesystem).
Actually it's become something more, and the bashrc now just calls some functions in a group of modules, which can serve multiple purposes.
The project right now is on sourceforge, and you can read its news with a RSS reader (e.g. thunderbird, firefox, straw or akregator) from this RSS feed.

Warnings
As you can guess, this tool is experimental, so, please, don't test it on mission-critical systems, and anyway don't hold me responsible for portage corruptions and/or other damages: use at your own risk. Probably in the worst case you'd only have a failed emerge, but I can't be sure about that.
Also, I'm in the ~x86 branch, and I can't test for other archs; on the italian forum, however, there's already a little testing community, so that shouldn't be a problem.
BIG FAT WARNING: before reporting any problem to [bug=]Gentoo's bugzilla[/bug], deactivate completely the bashrc with 'mv /etc/portage/bashrc /etc/portage/bashrc.bk' and retry. Obviously if the devs see any output which is not from portage, your bug will probably be discarded as INVALID.

How to use
To install it, download the ebuild, put it in your portage overlay (e.g. /usr/local/portage/app-portage/portage-bashrc-ng/portage-bashrc-ng-0.13_beta.ebuild) and create its digest ("ebuild portage-bashrc-ng-0.13_beta.ebuild digest), then emerge portage-bashrc-ng. To activate the modules, follow the ebuild's instructions, or just copy /etc/portage/bashrc-ng/bashrc-ng.example to /etc/portage/bashrc-ng/bashrc-ng and then use "eselect bashrc-ng".
»»»Fetch the most recent ebuild with layman: first of all emerge layman, activate it (put the line "source /usr/portage/local/layman/make.conf" in your /etc/make.conf), then fetch the overlay (with "layman -f -o http://wtk-overlay.sf.net/layman.xml -a WTK-Testing"), and when you want to check if there's a new version just synchronize it ("layman -s WTK-Testing").
Some of the following modules use additional configuration files in /etc/portage; for those files, the syntax is similar to those of portage, i.e. a portage atom, then a space or a tab, and then the optional configuration for that atom. Currently I'm supporting only the following atoms (I'm using an example to clarify, you can use any category, any package name and any version): sys-apps/* (that applies to all packages of the category sys-apps/), sys-apps/portage (that applies to all versions of sys-apps/portage), =sys-apps/portage-2.1.1_pre1-r1 (that applies only to that version of sys-apps/portage).

Tmpfs module
This module provides the capability to compile in ram + swap instead of the live filesystem; to activate it you need to set the compilation temporary dir's dimension:

• You can set it writing in /etc/portage/bashrc-ng/bashrc-ng.conf the line "PORTAGE_MEMSIZE=dim" where dim is one of the values accepted by mount for the size option: valid examples are 500M and 1G (500 megabytes, one gigabyte). Naturally, being a tmpfs partition, if the ram isn't enough to contain all the files, it uses the swap, and if it isn't enough too mount gives an error.
• You can modify that value on a per-package basis in /etc/portage/package.mem, specifying package and dimension: a valid example is "sys-devel/gcc 800M". The value in package.mem has priority over that in /etc/portage/bashrc-ng/bashrc-ng.conf.
• If you need to override the dimension value for a single emerge, and you don't want to modify the configuration file, you can set the OVERRIDE_MEMSIZE var: e.g. "OVERRIDE_MEMSIZE=100M emerge ...".
• If the chosen dimension is "0", it means to compile on disk.
• You can (if you really want to) run more than one emerge in ram simultaneously, if the overall dimension doesn't exceed the dimension of ram + swap.
• If when the emerge starts the $DEBUG variable is set (e.g. "DEBUG=on emerge ...") it prints more info: particularly, the command line with which mount is called, and the indication of the phase portage's in (in the form "step $EBUILD_PHASE").
• If in FEATURES there's "mantain" the tmpfs dir isn't unmounted; use with caution, or it will fill your ram.
• Problems: with very small packages it can be slower than an emerge on disk.

Per-package module
This module permits to modify $CFLAGS, $CXXFLAGS $LDFLAGS and $FEATURES on a per-package basis, first written by Ned Ludd (solar) and then modified by Ryan McIntosh (thebigslide): the original version is here, I modified it again, and then comio modified it again, so it's actually a bit different from the original now.
To activate it:

• CFLAGS: add a line with atom and CFLAGS to /etc/portage/package.cflags. You can use GLOBALCFLAGS and GLOBALCXXFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.nocflags file to filter out flags you don't want.
• CXXFLAGS: add a line with atom and CXXFLAGS to /etc/portage/package.cxxflags. You can use GLOBALCFLAGS and GLOBALCXXFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.nocxxflags file to filter out flags you don't want.
• LDFLAGS: add a line with atom and LDFLAGS to /etc/portage/package.ldflags. You can use GLOBALLDFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.noldflags file to filter out flags you don't want.
• FEATURES: add a line with atom and FEATURES to /etc/portage/package.features. You can use GLOBALFEATURES as a flag to reintroduce the global features. You can also use the /etc/portage/package.nofeatures file to filter out flags you don't want.

Localepurge module
This module simply calls localepurge, if present, after each installation: original idea by Diego Pettenò (flameeyes). To activate it, then, emerge app-admin/localepurge.

Autopatch module
This module automatically applies any patch the user choses after each installation; I simply adapted the original script by solar. To activate it set the PATCH_OVERLAY variable in /etc/portage/bashrc-ng/bashrc-ng.conf (e.g. PATCH_OVERLAY=/usr/portage/local/patches/), and inside the directory to which that points recreate the overlay structure (thus a patch for unzip will be for instance /usr/portage/local/patches/app-arch/unzip/mypatch.patch).

ResumeMerge module
Attempts to resume a merge. Especially useful in combination with the tmpfs module, since if the tmpfs partition's size is underestimated it can make the emerge fail; when this module is active the next compilation will start from the last phase, without unpacking again if that was already done, or without recompiling if the ebuild failed in the install phase.

Custom modules
If you want to write any custom module for the bashrc, you need to create a bash script and name it /etc/portage/bashrc-ng/mymodule.module, respecting the following rules. Feel free to submit any module you write if you think it could be useful to others.

• Each function which has the name of a portage phase with the prefix "on_" will be called when that phase begins; for instance, you could write on_compile() and it will get called when the compile phase begins.
• The additional functions on_BEGIN() and on_END() will be called, as the name implies, respectively before and after the phase function.
• There shouldn't be any code outside of a function, or it'll be called a lot of times in each phase.
• To make your module work, you need to set to on its name in /etc/portage/bashrc-ng/bashrc-ng.conf, e.g. "mymodule=on".
• Please don't make your module print output outside of a function, or it'll get way too verbose, since the module is called several times.
• Please implement each new functionality in a different module.
• You can use the bashrc's functions
  • debuginfo(), which prints its arguments when DEBUG has a non-null content
  • isfeature(), which sets the variable $itsafeature to 1 if its first argument is in FEATURES
  • parseconffile(), which checks if the current package matches an atom in the file which is given as the first parameter, and sets the value (if any) in $configval; it implements the syntax described in the "How to use" section.
• You shouldn't expect any variable to stay the same between calls.
• You shouldn't expect the order in which the modules are called to stay the same; however, right now it's alphabetical.
• You shouldn't write to any files, but you can remove them; the reason being that a mischievous user could create a link to an important system file, and use your code to overwrite it. Thanks to solar for pointing this out.

Final notes
The tool is still in testing, as you can see in the ChangeLog, so while I have time I'm still updating it and fixing its bugs; I hope that someone else would like to test it and maybe contribute to its development, and I'm open to any suggestion. Thanks to veonline and thewally for the precious ideas, fraido, fabius, comio and mziab for the patches and/or new modules, and Dr.Dran, !equilibrium and Philantrop for joining the project.
If you're interested, to know asap when a bug is discovered and fixed I suggest you to watch this topic by clicking on the link at the bottom of the page.

Thank you for your attention
Bye

[gringo]

thanks for this, finally an all-in-one bashrc
Will try for sure asap !

cheers

[Philantrop]

This tool was first born as a spin-off of one of Fonderiadigitale's projects,

Check your mail (s233...). :-)

[Ferdinando]

Check your mail (s233...).

Thank you Philantrop, I've tested your ebuild and it seems perfect; tomorrow I'll add it to sourceforge, hopefully for both the 0.9 and the 0.10 beta which I'm writing right now.

Bye

[Ferdinando]

As some of you have obviously noticed, there was a misspelling in /etc/portage/bashrc-ng/bashrc-ng.conf.example. I've made some other fixes, and in particular now the custom ebuilds can use one more function, parseconffile(), to parse a configuration file with the same syntax as /etc/portage/package.*; also this way the modules can share a common syntax for managing the configuration files. This and more in portage-bashrc-ng-v0.10.

Btw, Philantrop has officially joined the project on sourceforge, so you can bother him if you think your post is important and I don't reply in a reasonable time (but give me at least a day); also, this evening he'll upload an ebuild for the tool, and we can start distributing that too.

Bye

[Philantrop]

The updated ebuild for version 0.10 is now in the CVS repository. You can download it directly without using CVS here.

[enderandrew]

What about package.ldflags filtering?

[Ferdinando]

What about package.ldflags filtering?

Yeah, that's useful; I'll add that too to the per-package module (it's just a few more lines of code).
Thanks for the suggestion.

Bye

[Ferdinando]

I've uploaded the new version; the ebuild is now the only supported installation method, but be careful that the ebuild is only ~x86 (I'll fix that asap). Also, now you can use "eselect bashrc-ng" to activate the modules, and there's some additional documentation.
The per-package.module now can handle LDFLAGS too, but its name has now changed to perpackage.module.

Bye

[Philantrop]

To improve the ebuild, I'd like to get some reliable user feedback on the oldest versions of the following packages successfully used with portage-bashrc-ng:

sys-apps/portage
sys-apps/coreutils
sys-devel/patch
app-admin/localepurge

Only versions in the portage tree as of today are counted. "Reliable" refers to "I have actually used it." :-)

[gringo]

don´t know if i got you right Philantrop, if been using this since i know about it and no problems here with an up to date gentoo.

Ferdinando, would you be interested in adding this as a module http://forums.gentoo.org/viewtopic-t-472244.html ?

TIA

[Philantrop]

don't know if i got you right Philantrop, if been using this since i know about it and no problems here with an up to date gentoo.

You got me right but I need versions. :-) I can currently test on ~x86 only but I need feedback for stable (x86).

[gringo]

I can currently test on ~x86 only but I need feedback for stable (x86).

oops, sorry then, i´m on ~amd64 here
Anyways, i don´t see why this wouldn´t run on arch ...

cheers

[Ferdinando]

Anyways, i don´t see why this wouldn´t run on arch ...

Me neither, but he wants to be sure (a bit paranoid maybe?).
Btw, thanks for the link, it seems an interesting functionality; as soon as I can I´ll make a module out of it and notify the author.

Bye

[Onip]

Code: Select all

[I] sys-apps/coreutils (5.94-r1)
[I] sys-apps/portage (2.1-r1)
[I] sys-devel/patch (2.5.9):

I'm actually using only per-package module and it seems to work right

Code: Select all

Hal9000 ~ # emerge yafray
Calculating dependencies... done!
>>> Emerging (1 of 1) media-gfx/yafray-0.0.8 to /
 * Found CFLAGS '-O3 -march=pentium4 -mtune=pentium4 -pipe -fomit-frame-pointer -funroll-loops -msse -msse2 -mfpmath=sse' in //etc/portage/package.cflags: altering CFLAGS ...                                                            [ ok ]
...cut...

Byez

P.S. thanks for your work

[fabius]

Here same Onip's versions (x86 stable). I'm using only the module tmpfs.

If bashrc-ng requires portage-2.1 it's better to specify that version in the deps because portage is not stable for all archs (http://packages.gentoo.org/packages/?ca ... me=portage)

[Philantrop]

Thanks for the feedback! I've lowered the required versions in the dependencies.


Ferdinando, I wouldn't call it paranoia but professionally I'm in quality management and I've seen things break in the most obscure ways so I've become very, very cautious about such things. :)

There's no reason not to lower the requirements, though, after more testing. I'm currently in the process of setting up another testing environment which will run stable software only so that we should get over this really soon. :-)

[!equilibrium]

whatching now the CVS:

Code: Select all

>=app-shells/bash-3.0

can be safely removed now, because bash-3.0+ is a dependency for sys-apps/portage and sys-apps/baselayout (stable branch), and the lastest stable release is v3.1, so it's redundand.

[Philantrop]

Thanks for the information. I've reviewed the changes and I'm going to remove that dependency right now.

[Dr.Dran]

Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!!
Next days we release a new version of that module that clean the worst problem

Stay Tuned

Cheers

Franco

[EAD]

Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!!
Next days we release a new version of that module that clean the worst problem

Stay Tuned

Cheers

Franco

When can I use this script?
Can I use if after Stage3 before I put any thing on my system?
Can I use it to make emerge -e system && emerge -e world ?

[Philantrop]

Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!!

Do you have some details? I've tested the tmpfs module extensively and never found a problem like that.

[Ferdinando]

Do you have some details? I've tested the tmpfs module extensively and never found a problem like that.

It can only happen with poorly written ebuilds which overwrite $PN, no ebuild in the portage tree does that but still it's legal for portage so we've got to handle it. !Equilibrium said he's going to rewrite the module, so let's wait for the final product .

@EAD: you can use it whenever you want, it should work with every portage version I know of (since a couple of years ago at least), but keep in mind that it's still an experimental version, so perhaps you'd want your system up and running first, just to make sure that any error you encounter isn't related to the bashrc.

Bye

[EAD]

So, is it safe to use this after I just install Gentoo ?
After stage 3?
and, if I have 768 mem (256x3) , Should I really get some speedup?
How many Swap mem should I put? twice as much as my memmory will be good enough? (1500) ?

[Ferdinando]

So, is it safe to use this after I just install Gentoo ?
After stage 3?
and, if I have 768 mem (256x3) , Should I really get some speedup?
How many Swap mem should I put? twice as much as my memmory will be good enough? (1500) ?

It should be safe afaik, but I still haven't tested that configuration; anyway if you're using the 2006.0 livecd you shouldn't have any problem. That said, the ebuild pulls some dependencies in, so if you've sync'd it will probably try to update system first.
The speedup depends on the package, with small packages it could be unnoticeable, and the same with packages too large as it has to use the swap so basically it relies on the disk; with 768MB you should be fine, since I can compile almost everything in 500MB.
That of the swap is an old problem, no one really knows how much you need; the recommendation to use twice the dimension of the RAM is still valid, but in some cases useless and I know a lot of people who don't use a swap partition at all. That said, there are some programs which make an extensive use of the RAM (like the tmpfs ) so it's better to have some, and if you use demanding programs like matlab the more the better.

Bye