View previous topic :: View next topic |
Author |
Message |
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Sun Jun 18, 2006 9:34 am Post subject: [tool]portage-bashrc-ng - modular portage's bashrc |
|
|
Hello everyone.
This tool was first born as a spin-off of one of Fonderiadigitale's projects, portage-bashrc, a bashrc which is imported by portage and adds it the capability to compile in a dedicated tmpfs partition (using ram + swap instead of the live filesystem).
Actually it's become something more, and the bashrc now just calls some functions in a group of modules, which can serve multiple purposes.
The project right now is on sourceforge, and you can read its news with a RSS reader (e.g. thunderbird, firefox, straw or akregator) from this RSS feed.
Warnings
As you can guess, this tool is experimental, so, please, don't test it on mission-critical systems, and anyway don't hold me responsible for portage corruptions and/or other damages: use at your own risk. Probably in the worst case you'd only have a failed emerge, but I can't be sure about that.
Also, I'm in the ~x86 branch, and I can't test for other archs; on the italian forum, however, there's already a little testing community, so that shouldn't be a problem.
BIG FAT WARNING: before reporting any problem to Gentoo's bugzilla, deactivate completely the bashrc with 'mv /etc/portage/bashrc /etc/portage/bashrc.bk' and retry. Obviously if the devs see any output which is not from portage, your bug will probably be discarded as INVALID.
How to use
To install it, download the ebuild, put it in your portage overlay (e.g. /usr/local/portage/app-portage/portage-bashrc-ng/portage-bashrc-ng-0.13_beta.ebuild) and create its digest ("ebuild portage-bashrc-ng-0.13_beta.ebuild digest), then emerge portage-bashrc-ng. To activate the modules, follow the ebuild's instructions, or just copy /etc/portage/bashrc-ng/bashrc-ng.example to /etc/portage/bashrc-ng/bashrc-ng and then use "eselect bashrc-ng".
»»»Fetch the most recent ebuild with layman: first of all emerge layman, activate it (put the line "source /usr/portage/local/layman/make.conf" in your /etc/make.conf), then fetch the overlay (with "layman -f -o http://wtk-overlay.sf.net/layman.xml -a WTK-Testing"), and when you want to check if there's a new version just synchronize it ("layman -s WTK-Testing").
Some of the following modules use additional configuration files in /etc/portage; for those files, the syntax is similar to those of portage, i.e. a portage atom, then a space or a tab, and then the optional configuration for that atom. Currently I'm supporting only the following atoms (I'm using an example to clarify, you can use any category, any package name and any version): sys-apps/* (that applies to all packages of the category sys-apps/), sys-apps/portage (that applies to all versions of sys-apps/portage), =sys-apps/portage-2.1.1_pre1-r1 (that applies only to that version of sys-apps/portage).
Tmpfs module
This module provides the capability to compile in ram + swap instead of the live filesystem; to activate it you need to set the compilation temporary dir's dimension:
- You can set it writing in /etc/portage/bashrc-ng/bashrc-ng.conf the line "PORTAGE_MEMSIZE=dim" where dim is one of the values accepted by mount for the size option: valid examples are 500M and 1G (500 megabytes, one gigabyte). Naturally, being a tmpfs partition, if the ram isn't enough to contain all the files, it uses the swap, and if it isn't enough too mount gives an error.
- You can modify that value on a per-package basis in /etc/portage/package.mem, specifying package and dimension: a valid example is "sys-devel/gcc 800M". The value in package.mem has priority over that in /etc/portage/bashrc-ng/bashrc-ng.conf.
- If you need to override the dimension value for a single emerge, and you don't want to modify the configuration file, you can set the OVERRIDE_MEMSIZE var: e.g. "OVERRIDE_MEMSIZE=100M emerge ...".
- If the chosen dimension is "0", it means to compile on disk.
- You can (if you really want to) run more than one emerge in ram simultaneously, if the overall dimension doesn't exceed the dimension of ram + swap.
- If when the emerge starts the $DEBUG variable is set (e.g. "DEBUG=on emerge ...") it prints more info: particularly, the command line with which mount is called, and the indication of the phase portage's in (in the form "step $EBUILD_PHASE").
- If in FEATURES there's "mantain" the tmpfs dir isn't unmounted; use with caution, or it will fill your ram.
- Problems: with very small packages it can be slower than an emerge on disk.
Per-package module
This module permits to modify $CFLAGS, $CXXFLAGS $LDFLAGS and $FEATURES on a per-package basis, first written by Ned Ludd (solar) and then modified by Ryan McIntosh (thebigslide): the original version is here, I modified it again, and then comio modified it again, so it's actually a bit different from the original now.
To activate it:
- CFLAGS: add a line with atom and CFLAGS to /etc/portage/package.cflags. You can use GLOBALCFLAGS and GLOBALCXXFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.nocflags file to filter out flags you don't want.
- CXXFLAGS: add a line with atom and CXXFLAGS to /etc/portage/package.cxxflags. You can use GLOBALCFLAGS and GLOBALCXXFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.nocxxflags file to filter out flags you don't want.
- LDFLAGS: add a line with atom and LDFLAGS to /etc/portage/package.ldflags. You can use GLOBALLDFLAGS as a flag to reintroduce the global flags. You can also use the /etc/portage/package.noldflags file to filter out flags you don't want.
- FEATURES: add a line with atom and FEATURES to /etc/portage/package.features. You can use GLOBALFEATURES as a flag to reintroduce the global features. You can also use the /etc/portage/package.nofeatures file to filter out flags you don't want.
Localepurge module
This module simply calls localepurge, if present, after each installation: original idea by Diego Pettenò (flameeyes). To activate it, then, emerge app-admin/localepurge.
Autopatch module
This module automatically applies any patch the user choses after each installation; I simply adapted the original script by solar. To activate it set the PATCH_OVERLAY variable in /etc/portage/bashrc-ng/bashrc-ng.conf (e.g. PATCH_OVERLAY=/usr/portage/local/patches/), and inside the directory to which that points recreate the overlay structure (thus a patch for unzip will be for instance /usr/portage/local/patches/app-arch/unzip/mypatch.patch).
ResumeMerge module
Attempts to resume a merge. Especially useful in combination with the tmpfs module, since if the tmpfs partition's size is underestimated it can make the emerge fail; when this module is active the next compilation will start from the last phase, without unpacking again if that was already done, or without recompiling if the ebuild failed in the install phase.
Custom modules
If you want to write any custom module for the bashrc, you need to create a bash script and name it /etc/portage/bashrc-ng/mymodule.module, respecting the following rules. Feel free to submit any module you write if you think it could be useful to others.
- Each function which has the name of a portage phase with the prefix "on_" will be called when that phase begins; for instance, you could write on_compile() and it will get called when the compile phase begins.
- The additional functions on_BEGIN() and on_END() will be called, as the name implies, respectively before and after the phase function.
- There shouldn't be any code outside of a function, or it'll be called a lot of times in each phase.
- To make your module work, you need to set to on its name in /etc/portage/bashrc-ng/bashrc-ng.conf, e.g. "mymodule=on".
- Please don't make your module print output outside of a function, or it'll get way too verbose, since the module is called several times.
- Please implement each new functionality in a different module.
- You can use the bashrc's functions
- debuginfo(), which prints its arguments when DEBUG has a non-null content
- isfeature(), which sets the variable $itsafeature to 1 if its first argument is in FEATURES
- parseconffile(), which checks if the current package matches an atom in the file which is given as the first parameter, and sets the value (if any) in $configval; it implements the syntax described in the "How to use" section.
- You shouldn't expect any variable to stay the same between calls.
- You shouldn't expect the order in which the modules are called to stay the same; however, right now it's alphabetical.
- You shouldn't write to any files, but you can remove them; the reason being that a mischievous user could create a link to an important system file, and use your code to overwrite it. Thanks to solar for pointing this out.
Final notes
The tool is still in testing, as you can see in the ChangeLog, so while I have time I'm still updating it and fixing its bugs; I hope that someone else would like to test it and maybe contribute to its development, and I'm open to any suggestion. Thanks to veonline and thewally for the precious ideas, fraido, fabius, comio and mziab for the patches and/or new modules, and Dr.Dran, !equilibrium and Philantrop for joining the project.
If you're interested, to know asap when a bug is discovered and fixed I suggest you to watch this topic by clicking on the link at the bottom of the page.
Thank you for your attention
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997
Last edited by Ferdinando on Tue Sep 05, 2006 11:19 am; edited 7 times in total |
|
Back to top |
|
|
gringo Advocate
Joined: 27 Apr 2003 Posts: 3793
|
Posted: Sun Jun 18, 2006 10:24 am Post subject: |
|
|
thanks for this, finally an all-in-one bashrc
Will try for sure asap !
cheers |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Sun Jun 18, 2006 4:01 pm Post subject: Re: [tool]portage-bashrc-ng - modular portage's bashrc |
|
|
Ferdinando wrote: | This tool was first born as a spin-off of one of Fonderiadigitale's projects, |
Check your mail (s233...). :-) |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Sun Jun 18, 2006 7:43 pm Post subject: Re: [tool]portage-bashrc-ng - modular portage's bashrc |
|
|
Philantrop wrote: | Check your mail (s233...). |
Thank you Philantrop, I've tested your ebuild and it seems perfect; tomorrow I'll add it to sourceforge, hopefully for both the 0.9 and the 0.10 beta which I'm writing right now.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Mon Jun 19, 2006 12:28 pm Post subject: beta 0.10 |
|
|
As some of you have obviously noticed, there was a misspelling in /etc/portage/bashrc-ng/bashrc-ng.conf.example. I've made some other fixes, and in particular now the custom ebuilds can use one more function, parseconffile(), to parse a configuration file with the same syntax as /etc/portage/package.*; also this way the modules can share a common syntax for managing the configuration files. This and more in portage-bashrc-ng-v0.10.
Btw, Philantrop has officially joined the project on sourceforge, so you can bother him if you think your post is important and I don't reply in a reasonable time (but give me at least a day); also, this evening he'll upload an ebuild for the tool, and we can start distributing that too.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Mon Jun 19, 2006 6:44 pm Post subject: |
|
|
The updated ebuild for version 0.10 is now in the CVS repository. You can download it directly without using CVS here. |
|
Back to top |
|
|
enderandrew l33t
Joined: 25 Oct 2005 Posts: 731
|
Posted: Tue Jun 20, 2006 10:34 am Post subject: |
|
|
What about package.ldflags filtering? _________________ Nihilism makes me smile. |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Tue Jun 20, 2006 1:13 pm Post subject: |
|
|
enderandrew wrote: | What about package.ldflags filtering? |
Yeah, that's useful; I'll add that too to the per-package module (it's just a few more lines of code).
Thanks for the suggestion.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Sat Jun 24, 2006 7:59 pm Post subject: |
|
|
I've uploaded the new version; the ebuild is now the only supported installation method, but be careful that the ebuild is only ~x86 (I'll fix that asap). Also, now you can use "eselect bashrc-ng" to activate the modules, and there's some additional documentation.
The per-package.module now can handle LDFLAGS too, but its name has now changed to perpackage.module.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Thu Jun 29, 2006 7:11 pm Post subject: |
|
|
To improve the ebuild, I'd like to get some reliable user feedback on the oldest versions of the following packages successfully used with portage-bashrc-ng:
sys-apps/portage
sys-apps/coreutils
sys-devel/patch
app-admin/localepurge
Only versions in the portage tree as of today are counted. "Reliable" refers to "I have actually used it." :-) |
|
Back to top |
|
|
gringo Advocate
Joined: 27 Apr 2003 Posts: 3793
|
Posted: Fri Jun 30, 2006 8:15 am Post subject: |
|
|
don´t know if i got you right Philantrop, if been using this since i know about it and no problems here with an up to date gentoo.
Ferdinando, would you be interested in adding this as a module https://forums.gentoo.org/viewtopic-t-472244.html ?
TIA |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Fri Jun 30, 2006 9:09 am Post subject: |
|
|
gringo wrote: | don't know if i got you right Philantrop, if been using this since i know about it and no problems here with an up to date gentoo. |
You got me right but I need versions. :-) I can currently test on ~x86 only but I need feedback for stable (x86). |
|
Back to top |
|
|
gringo Advocate
Joined: 27 Apr 2003 Posts: 3793
|
Posted: Fri Jun 30, 2006 9:13 am Post subject: |
|
|
Quote: | I can currently test on ~x86 only but I need feedback for stable (x86). |
oops, sorry then, i´m on ~amd64 here
Anyways, i don´t see why this wouldn´t run on arch ...
cheers |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Sat Jul 01, 2006 9:48 am Post subject: |
|
|
gringo wrote: | Anyways, i don´t see why this wouldn´t run on arch ... |
Me neither, but he wants to be sure (a bit paranoid maybe?).
Btw, thanks for the link, it seems an interesting functionality; as soon as I can I´ll make a module out of it and notify the author.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
Onip Advocate
Joined: 02 Sep 2004 Posts: 2912 Location: Parma (Italy)
|
Posted: Sat Jul 01, 2006 10:20 am Post subject: |
|
|
Code: | [I] sys-apps/coreutils (5.94-r1)
[I] sys-apps/portage (2.1-r1)
[I] sys-devel/patch (2.5.9):
|
I'm actually using only per-package module and it seems to work right
Code: | Hal9000 ~ # emerge yafray
Calculating dependencies... done!
>>> Emerging (1 of 1) media-gfx/yafray-0.0.8 to /
* Found CFLAGS '-O3 -march=pentium4 -mtune=pentium4 -pipe -fomit-frame-pointer -funroll-loops -msse -msse2 -mfpmath=sse' in //etc/portage/package.cflags: altering CFLAGS ... [ ok ]
...cut...
|
Byez
P.S. thanks for your work _________________ Linux Registered User n. 373835
Titus Lucretius Carus, De Rerum Natura - Tantum religio potuit suadere malorum |
|
Back to top |
|
|
fabius Guru
Joined: 29 Nov 2004 Posts: 525
|
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Sat Jul 01, 2006 12:14 pm Post subject: |
|
|
Thanks for the feedback! I've lowered the required versions in the dependencies.
Ferdinando, I wouldn't call it paranoia but professionally I'm in quality management and I've seen things break in the most obscure ways so I've become very, very cautious about such things. :)
There's no reason not to lower the requirements, though, after more testing. I'm currently in the process of setting up another testing environment which will run stable software only so that we should get over this really soon. :-) |
|
Back to top |
|
|
!equilibrium Bodhisattva
Joined: 06 Jun 2004 Posts: 2109 Location: MI/BG/LC
|
Posted: Sat Jul 01, 2006 1:24 pm Post subject: |
|
|
whatching now the CVS:
Code: | >=app-shells/bash-3.0 |
can be safely removed now, because bash-3.0+ is a dependency for sys-apps/portage and sys-apps/baselayout (stable branch), and the lastest stable release is v3.1, so it's redundand. _________________ Arch Tester for Gentoo/FreeBSD
Equilibrium's Universe
all my contents are released under the Creative Commons Licence by-nc-nd 2.5 |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Sat Jul 01, 2006 1:34 pm Post subject: |
|
|
Thanks for the information. I've reviewed the changes and I'm going to remove that dependency right now. |
|
Back to top |
|
|
Dr.Dran l33t
Joined: 08 Oct 2004 Posts: 766 Location: Imola - Italy
|
Posted: Sun Jul 09, 2006 12:40 pm Post subject: |
|
|
Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!!
Next days we release a new version of that module that clean the worst problem
Stay Tuned
Cheers
Franco _________________ :: [Dr.Dran] Details ::
- Linux User # 286282
- IT FreeLance Consultant
- President of ImoLUG [Imola & Faenza Linux User Group] |
|
Back to top |
|
|
EAD Guru
Joined: 05 Jul 2006 Posts: 352
|
Posted: Mon Jul 10, 2006 6:34 am Post subject: |
|
|
Dr.Dran wrote: | Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!!
Next days we release a new version of that module that clean the worst problem
Stay Tuned
Cheers
Franco |
When can I use this script?
Can I use if after Stage3 before I put any thing on my system?
Can I use it to make emerge -e system && emerge -e world ? |
|
Back to top |
|
|
Philantrop Retired Dev
Joined: 21 Dec 2004 Posts: 1130 Location: Germany
|
Posted: Mon Jul 10, 2006 6:52 am Post subject: |
|
|
Dr.Dran wrote: | Make ATTENTION there is a problem with the TMPFS module that cause the system to be corrupted (begins some rm -rf loops), my friend !equilibrium still work on it!!! |
Do you have some details? I've tested the tmpfs module extensively and never found a problem like that. |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Mon Jul 10, 2006 7:21 am Post subject: |
|
|
Philantrop wrote: | Do you have some details? I've tested the tmpfs module extensively and never found a problem like that. |
It can only happen with poorly written ebuilds which overwrite $PN, no ebuild in the portage tree does that but still it's legal for portage so we've got to handle it. !Equilibrium said he's going to rewrite the module, so let's wait for the final product .
@EAD: you can use it whenever you want, it should work with every portage version I know of (since a couple of years ago at least), but keep in mind that it's still an experimental version, so perhaps you'd want your system up and running first, just to make sure that any error you encounter isn't related to the bashrc.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
EAD Guru
Joined: 05 Jul 2006 Posts: 352
|
Posted: Mon Jul 10, 2006 7:33 am Post subject: |
|
|
So, is it safe to use this after I just install Gentoo ?
After stage 3?
and, if I have 768 mem (256x3) , Should I really get some speedup?
How many Swap mem should I put? twice as much as my memmory will be good enough? (1500) ? |
|
Back to top |
|
|
Ferdinando Veteran
Joined: 25 Nov 2003 Posts: 1027 Location: Gaeta (LT) - Italy
|
Posted: Mon Jul 10, 2006 8:25 am Post subject: |
|
|
EAD wrote: | So, is it safe to use this after I just install Gentoo ?
After stage 3?
and, if I have 768 mem (256x3) , Should I really get some speedup?
How many Swap mem should I put? twice as much as my memmory will be good enough? (1500) ? |
It should be safe afaik, but I still haven't tested that configuration; anyway if you're using the 2006.0 livecd you shouldn't have any problem. That said, the ebuild pulls some dependencies in, so if you've sync'd it will probably try to update system first.
The speedup depends on the package, with small packages it could be unnoticeable, and the same with packages too large as it has to use the swap so basically it relies on the disk; with 768MB you should be fine, since I can compile almost everything in 500MB.
That of the swap is an old problem, no one really knows how much you need; the recommendation to use twice the dimension of the RAM is still valid, but in some cases useless and I know a lot of people who don't use a swap partition at all. That said, there are some programs which make an extensive use of the RAM (like the tmpfs ) so it's better to have some, and if you use demanding programs like matlab the more the better.
Bye _________________ La risposta, non la devi cercare fuori, la devi cercare dentro di te: e però è SBAGLIATA!
-- Corrado Guzzanti, "Pippo Chennedy Show", 1997 |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|