An emerge wrapper for breaking emerges into chunks

[fredor]

MindEraser: Yes it was your script I used but I seem to have misunderstood. I thought the objective was to emerge the toolchain seperately from 'emerge -e system/world'
With your script as it stands it only picks up the updated ebuilds so if there are none available the build.lst is empty.
If you change the code as follows:-

Code: Select all

# Get all available Updates
emerge -e $my_world > wrld.lst

Then it appears to do what I expected.
Your objectives may well be different, of course. What I would like is a programme which which would compile seperate section:-

• 1 Mini Tollchain
  2. Toolchain
  3. emerge -e system less Toolchain
  4 emerge -e world less System less Toolchain
  5 emerge -e world less Toolchain

Heilvc has opened up this possibility with his efforts to date - Many Thanks

[hielvc]

fredor Thanks for the summury. What I actually had in mind was

1 Mini Toolchain
2. Toolchain
3. emerge -e system less Toolchain
4 emerge -e world less System less Toolchain
5 emerge -e world less Toolchain
6. emerge -uD sytem less Toolchain
7. emerge -uD world less System less Toolchain


Which would be very easy to add or compelet now that MindEraser has gotten the code so far. I not sure about MindEraser but once I started, getting all the oopies and "oh I forgot to have it do that like TCfiltering here or buid there" and getting the right working revision overrode "the lets add that feature".

The only one of the above items that migh be trickier than first appearances would be #4. The nice feature that MindEraser added and diffently has posibilities is the "-e" taking multipule agr's, particularly with what I think MindEraser wants to do with added use of emerge -v. And I'll take a look at color output MindEraser.

[hielvc]

MindEraser if the filters are run and then the "emerge for loop" youll get color output. Will this do what you need?

[fredor]

heilvc What you suggest would be great. As you say item #4 is the hardest. I'm not really a programmer and the only way I can think of is to use 'diff' but I'm sure there is a better way.
I was wondering if initially a build list could be created for each of the system and world sections. Then the rest of the programmng would be relatively easy. Anyway, I'll leave it to you experts.
Many thanks.

[hielvc]

While driveing home I was thinking about #4 "emerge -e world less system less TC" A simple " TCupdate -system && TCupdate -world." takes care of it. If we start caching files of a system update to remove from a world update its only going to be good till your next sync. I see nothing but a can of worms trying to do that and to keep track times and dates when portage already does the data base tracking for us. ARRRR taht only works for -uD and not --emtytree.

[hielvc]

MindEraser Ok I'm sorry for takeing so long to respond. But work has become quite
busy which is good I make money but getting home after 8 and 9 leaves me with a fried
brain. As to new gcc install I would want to use the newer gcc. As to haveing the user
select it, well over the last 4 months sometimes you need to it and sometimes portage does it
for you and sometimes you have to create the profile yourself. Currently libaries are
being broken, be it gcc, ncurses X but alot post are about them Which segs in to your other
question YES IF THE TC FAILS WE NEED TO STOP. I actually had that in one of my versions.
I dont think that we should let, invite someone to try and define the order of their TC
build. Those who are that advanced can hack the script easily, those who arent will have a
harder time blowing their own feet off.

[MindEraser]

Hi, guys, I've been busy lately too. If I find a couple of hours free I will try to implement some of the features.
First about the colour. What I meant is I want to grep (or otherwise pipe the output of emerge) but retain the color. Since emerge detects that and spills colorless output the only thing I can think of is putting it back in with sed (like you hielvc color your help). Then I can go on implementing -v. I would be happy to get rid of the "These are the packages that ... in order.." and "Calculating dependencies" messages by emerge. The problem is relevant mainly for TC, since for world when pretending (for speed and color reasons) I don't actually run the emerge loop as the real run does but instead do a emerge -p the_whole_build_list. Perhaps I should do the same for TC: first let portage state the exact versions (when -v the use flags too) for TC and then just print the exact order of merging.
About the 7 points.
1 done
2 done
6 done
3 & 5 I dont see the point of that if one nails down the toolchain first then no problem rebuilding it again with system.
4 & 7 Very interesting. Worth the try. I think we can figure out dynamically which packages portage considers system at the moment (perhaps with emerge -e --nodeps system or if not i will think of sth) and filter them out from emerge -e/-uD world.
If we r going to implement the 4 & 7 options then the current -e is perhaps misleading. Better change it to -x (eXclude), uh??? And then use -e for empty-tree...
hielvc, can I then assume that you tested the way my script builds the TC and you agree? Any corrections?
I'll be around when I find some time.

[dbaigrie]

The froced TC update requires the variables to be set or it won't merge anything!

[MindEraser]

The froced TC update requires the variables to be set or it won't merge anything!

True! If there are no TC updates the script should do nothing. I was thinking of adding an option to force a rebuild of the whole toolchain. It is very easy to do so you could expect it soon in my version. Meanwhile, overriding detection with lh=1 should do the trick : )))

[fredor]

MindEraser The reason for #3 and #5 is time.
For me it take 3 to 4 hours for the Toolchain (with multilib)
ca. 5 hours for 'emerge -e system
ca. 16+ hours for 'emerge -e world'
So it is an advantage if the Toolchain is removed from 'emerge -e system.
I have just updated a Gnome system to the 2005.0 profile and afterwards used your script to re-compile the Toolchain. I am now using the script with 'emerge -e system' replacing the 'emerge system -uD' to compile the system packages less Toolchain. So far it's running OK.
One thing I noticed, however, it effectively compiles one package at a time and the heading shows 'emerge (1 of 1) so unless you have a file list you don't know how many packages have been compiled at any one time. It may be difficult to alter this.

[MindEraser]

One thing I noticed, however, it effectively compiles one package at a time and the heading shows 'emerge (1 of 1) so unless you have a file list you don't know how many packages have been compiled at any one time. It may be difficult to alter this.

fredor, this is a desired behaviour. I will try to add sth so the user can see how far through the whole build process he is but there are various reasons for building the packages one by one and not all at once.

Thanks for the feedback. I hope it goes well all the way through.
You are right about #3 & #5 and again as this is easy you should expect them soon.

[fredor]

The Systems less Toolchain compilation has just finished taking just about 2 hours which is not bad since it includes 'xorg'
So I am glad you agree on #3 and #5.

[hielvc]

MindEraser I was testing your TC build. There is an extra build of gcc if you have an gcc update with either a glibc and/or linux-headers update. The rest work well.

*** Following ToolChain Updates Available:

[ebuild N ] sys-libs/glibc-2.3.4.20041102
[ebuild N ] sys-devel/gcc-3.4.3-r1

Merging following TC packages in order:

[ebuild R ] sys-libs/glibc-2.3.4.20041102
1 [ebuild R ] sys-devel/gcc-3.4.3-r1
[ebuild R ] sys-libs/glibc-2.3.4.20041102
[ebuild R ] sys-devel/binutils-2.15.92.0.2-r2
2 [ebuild R ] sys-devel/gcc-3.4.3-r1
[ebuild R ] sys-devel/binutils-2.15.92.0.2-r2
3 [ebuild R ] sys-devel/gcc-3.4.3-r1.

[MindEraser]

hielvc,
I know that. As I said before there are 15 combinations of at least one tc update. I just tried to give my best figuring out what is best in which case. I pushed one gcc more so that we start using the new compiler as soon as possible. Your help will be very appreciated here. What I know tells me this:
The priority for merging is as follows:
os-headers, glibc, gcc, binutils
(... or, perhaps better os-headers, gcc, glibc, binutils - I really don't know)
Furthermore, since os-heasers don't get compiled then just emerge them twice in the beginning.
I did a little bit of thinking and I am convinced now that I was overcomplicating things. As I see it rebuilding the TC should be in 2 steps:
STEP 1: Update all TC packages to the newest version.
- This part should vary according to available updates
STEP 2: Rebuild the TC. Fixed. I would go for glibc, bu, gcc, glibc, bu, gcc.

My question is, shouldn't we take care of portage too? I've seen it appear in the middle of the list for world updates. And although it is mainly bash and python there are a few binaries for sandboxing and stuff that get compiled.

I'm getting to work. I'll fix the toolchain and all the options I've promised. I'll post when I am ready.

[MindEraser]

Posted new version.
Addressed all requests except the thing with emerge (1 of 1).
Don't forget to read the help, options tend to change!
Use the link below to go to my post with the link.

[rhill]

wow, this thing really took off. amazing work guys.

just as a thought on the priority, i was looking into the toolchain's build order myself a little while ago and i settled on following the LFS suggestion of headers->glibc->binutils->gcc. they actually build binutils first during the bootstrap.

Binutils is installed first because the ./configure runs of both GCC and Glibc perform various feature tests on the assembler and linker to determine which software features to enable or disable. This is more important than one might first realize. An incorrectly configured GCC or Glibc can result in a subtly broken toolchain, where the impact of such breakage might not show up until near the end of the build of an entire distribution.

i don't know if it actually makes much difference once the system is built, but i figure in the end you have the choice of gcc->binutils or binutils->gcc and if there's no other reason to pick one over the other it can't hurt.

[MindEraser]

Thanks for the tip, dirtyepic
I don't know if it really matters in which order I update the packages since I am re-emerging everything twice afterwards. I was trying to push gcc in front as I don't see much sense in compiling things with the old gcc.
Anyways, if one doesn't like my order she can edit the script & redefine it

[MindEraser]

New Version again: 0.2.2.0
Cool new features and other improvements
As usual, (can't say that enough) READ HELP!
Follow, the link in my sig to go to the post with the actual link.

TODO:
(I've already laid down the bases for that so expect soon!)
- Status check. The script will write current progress info in a file and calling it from another shell with --status option will give you the info.
Can't think of any way of overriding what emerge puts on xterm titles.
- Control over which dirs are used for tmp and log

[landon]

Hmm, came across this particular thread after clicking on 2 different signatured-urls. Which script is currently being developed? Is it now a joint development?

[MindEraser]

Kinda joint development, yeah
Use mine : )) It's the latest version

[hielvc]

1st added a link above my script to MindErasers.

I did manage to look around some considering the build order. Dirtyepic is right. The question I have is should the order be " glibc binutils gcc && glibc binutils gcc " or should it be "glibc glibc && binutils gcc && binutils gcc". Ive seen both and and done both. I wonder which is more correct or if either is more correct. Any all knowing peeps out there?

Minderaser as to portage I agree I like to emerge it first then the rest of whatever.

[rhill]

The question I have is should the order be " glibc binutils gcc && glibc binutils gcc " or should it be "glibc glibc && binutils gcc && binutils gcc". Ive seen both and and done both. I wonder which is more correct or if either is more correct. Any all knowing peeps out there?

the first one definitely. you want to build the new toolchain as a whole, and then build it again with itself.

think of glibc as your knowledge base, binutils as your toolbox, and gcc as your factory. while you could go to school twice, buy yourself a toolkit, build a factory, build better tools with your new factory, and rebuild the factory with your new tools, you can alternatively go to school, get a toolkit, build a factory, learn everything about how your new factory works, take that knowledge and build a new set of tools, then rebuild the factory with complete knowledge of how it operates with tools specifically designed for it. both ways work, but the second is optimal.

[MindEraser]

Hey, I got it right then
At least the rebuilding part
What about the order of initial updating/building?
Do you agree it should be
portage - 2x os-headers - glibc - binutils-cfg - binutils - gcc-cfg - gcc

hielvc, thanks for the link

How do you guys find the new features?
Any bugs, corrections, wishes?

[fredor]

TCupdate is coming along fine now - Well Done !!!
'tcupdate.sh -tep' gives the following :-

Code: Select all

[ebuild   R   ] sys-apps/portage-2.0.51-r15
[ebuild   R   ] sys-kernel/linux26-headers-2.6.8.1-r3
[ebuild   R   ] sys-kernel/linux26-headers-2.6.8.1-r3
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-config-1.8
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-config-1.3.10-r1
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-apps/portage-2.0.51-r15

This seems a bit excessive but I am sure it's easily fixed.

[fredor]

Running tcupdate.sh with the '-e' option picks up up all the toolchain packages and adds them to 'tc_list'.
Then for the forced 'tc' update the 'tc_build' is added to this list produced the list in the previous post.
I have modified my copy adding a 'do_tc_force' flag and using this so the 'tc_list' is made to equal only 'tc_build'.
It works fine now

Code: Select all

[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-apps/portage-2.0.51-r15

Thanks for your efforts.