Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
An emerge wrapper for breaking emerges into chunks
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, ... 28, 29, 30  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Wed Jan 26, 2005 1:49 pm    Post subject: Reply with quote

MindEraser: Yes it was your script I used but I seem to have misunderstood. I thought the objective was to emerge the toolchain seperately from 'emerge -e system/world'
With your script as it stands it only picks up the updated ebuilds so if there are none available the build.lst is empty.
If you change the code as follows:-
Code:
# Get all available Updates
emerge -e $my_world > wrld.lst

Then it appears to do what I expected.
Your objectives may well be different, of course. What I would like is a programme which which would compile seperate section:-

    1 Mini Tollchain
    2. Toolchain
    3. emerge -e system less Toolchain
    4 emerge -e world less System less Toolchain
    5 emerge -e world less Toolchain


Heilvc has opened up this possibility with his efforts to date - Many Thanks
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Wed Jan 26, 2005 3:34 pm    Post subject: Reply with quote

fredor Thanks for the summury. What I actually had in mind was

1 Mini Toolchain
2. Toolchain
3. emerge -e system less Toolchain
4 emerge -e world less System less Toolchain
5 emerge -e world less Toolchain
6. emerge -uD sytem less Toolchain
7. emerge -uD world less System less Toolchain


Which would be very easy to add or compelet now that MindEraser has gotten the code so far. I not sure about MindEraser but once I started, getting all the oopies and "oh I forgot to have it do that like TCfiltering here or buid there" and getting the right working revision :oops: overrode "the lets add that feature".

The only one of the above items that migh be trickier than first appearances would be #4. The nice feature that MindEraser added and diffently has posibilities is the "-e" taking multipule agr's, particularly with what I think MindEraser wants to do with added use of emerge -v. And I'll take a look at color output MindEraser.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Wed Jan 26, 2005 3:54 pm    Post subject: Reply with quote

MindEraser if the filters are run and then the "emerge for loop" youll get color output. Will this do what you need?
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Wed Jan 26, 2005 6:16 pm    Post subject: Reply with quote

heilvc What you suggest would be great. As you say item #4 is the hardest. I'm not really a programmer and the only way I can think of is to use 'diff' but I'm sure there is a better way.
I was wondering if initially a build list could be created for each of the system and world sections. Then the rest of the programmng would be relatively easy. Anyway, I'll leave it to you experts.
Many thanks.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Jan 27, 2005 1:50 am    Post subject: Reply with quote

While driveing home I was thinking about #4 "emerge -e world less system less TC" A simple " TCupdate -system && TCupdate -world." takes care of it. If we start caching files of a system update to remove from a world update its only going to be good till your next sync. I see nothing but a can of worms trying to do that and to keep track times and dates when portage already does the data base tracking for us. ARRRR taht only works for -uD and not --emtytree.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Jan 27, 2005 2:42 am    Post subject: Reply with quote

MindEraser Ok I'm sorry for takeing so long to respond. But work has become quite
busy which is good I make money but getting home after 8 and 9 leaves me with a fried
brain. As to new gcc install I would want to use the newer gcc. As to haveing the user
select it, well over the last 4 months sometimes you need to it and sometimes portage does it
for you and sometimes you have to create the profile yourself. Currently libaries are
being broken, be it gcc, ncurses X but alot post are about them Which segs in to your other
question YES IF THE TC FAILS WE NEED TO STOP. I actually had that in one of my versions.
I dont think that we should let, invite someone to try and define the order of their TC
build. Those who are that advanced can hack the script easily, those who arent will have a
harder time blowing their own feet off. :!:
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Thu Jan 27, 2005 2:13 pm    Post subject: Reply with quote

Hi, guys, I've been busy lately too. If I find a couple of hours free I will try to implement some of the features.
First about the colour. What I meant is I want to grep (or otherwise pipe the output of emerge) but retain the color. Since emerge detects that and spills colorless output the only thing I can think of is putting it back in with sed (like you hielvc color your help). Then I can go on implementing -v. I would be happy to get rid of the "These are the packages that ... in order.." and "Calculating dependencies" messages by emerge. The problem is relevant mainly for TC, since for world when pretending (for speed and color reasons) I don't actually run the emerge loop as the real run does but instead do a emerge -p the_whole_build_list. Perhaps I should do the same for TC: first let portage state the exact versions (when -v the use flags too) for TC and then just print the exact order of merging.
About the 7 points.
1 done
2 done
6 done
3 & 5 I dont see the point of that if one nails down the toolchain first then no problem rebuilding it again with system.
4 & 7 Very interesting. Worth the try. I think we can figure out dynamically which packages portage considers system at the moment (perhaps with emerge -e --nodeps system or if not i will think of sth) and filter them out from emerge -e/-uD world.
If we r going to implement the 4 & 7 options then the current -e is perhaps misleading. Better change it to -x (eXclude), uh??? And then use -e for empty-tree...
hielvc, can I then assume that you tested the way my script builds the TC and you agree? Any corrections?
I'll be around when I find some time.
Back to top
View user's profile Send private message
dbaigrie
n00b
n00b


Joined: 21 Nov 2004
Posts: 4

PostPosted: Thu Jan 27, 2005 3:01 pm    Post subject: Forced TC update Reply with quote

The froced TC update requires the variables to be set or it won't merge anything!
_________________
--------------------------------------

The sign of a truly advanced intelligence is the ability to hold two opposing thoughts and still be able to function rationally.
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Thu Jan 27, 2005 5:36 pm    Post subject: Re: Forced TC update Reply with quote

dbaigrie wrote:
The froced TC update requires the variables to be set or it won't merge anything!

True! If there are no TC updates the script should do nothing. I was thinking of adding an option to force a rebuild of the whole toolchain. It is very easy to do so you could expect it soon in my version. Meanwhile, overriding detection with lh=1 should do the trick : )))
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Thu Jan 27, 2005 6:59 pm    Post subject: Reply with quote

MindEraser The reason for #3 and #5 is time.
For me it take 3 to 4 hours for the Toolchain (with multilib)
ca. 5 hours for 'emerge -e system
ca. 16+ hours for 'emerge -e world'
So it is an advantage if the Toolchain is removed from 'emerge -e system.
I have just updated a Gnome system to the 2005.0 profile and afterwards used your script to re-compile the Toolchain. I am now using the script with 'emerge -e system' replacing the 'emerge system -uD' to compile the system packages less Toolchain. So far it's running OK.
One thing I noticed, however, it effectively compiles one package at a time and the heading shows 'emerge (1 of 1) so unless you have a file list you don't know how many packages have been compiled at any one time. It may be difficult to alter this.
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Thu Jan 27, 2005 7:41 pm    Post subject: Reply with quote

Quote:
One thing I noticed, however, it effectively compiles one package at a time and the heading shows 'emerge (1 of 1) so unless you have a file list you don't know how many packages have been compiled at any one time. It may be difficult to alter this.

fredor, this is a desired behaviour. I will try to add sth so the user can see how far through the whole build process he is but there are various reasons for building the packages one by one and not all at once.

Thanks for the feedback. I hope it goes well all the way through.
You are right about #3 & #5 and again as this is easy you should expect them soon.
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Thu Jan 27, 2005 8:33 pm    Post subject: Reply with quote

The Systems less Toolchain compilation has just finished taking just about 2 hours which is not bad since it includes 'xorg'
So I am glad you agree on #3 and #5.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Fri Jan 28, 2005 5:32 am    Post subject: Reply with quote

MindEraser I was testing your TC build. There is an extra build of gcc if you have an gcc update with either a glibc and/or linux-headers update. The rest work well.
Quote:
*** Following ToolChain Updates Available:

[ebuild N ] sys-libs/glibc-2.3.4.20041102
[ebuild N ] sys-devel/gcc-3.4.3-r1

Merging following TC packages in order:

[ebuild R ] sys-libs/glibc-2.3.4.20041102
1 [ebuild R ] sys-devel/gcc-3.4.3-r1
[ebuild R ] sys-libs/glibc-2.3.4.20041102
[ebuild R ] sys-devel/binutils-2.15.92.0.2-r2
2 [ebuild R ] sys-devel/gcc-3.4.3-r1
[ebuild R ] sys-devel/binutils-2.15.92.0.2-r2
3 [ebuild R ] sys-devel/gcc-3.4.3-r1.

_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Fri Jan 28, 2005 3:29 pm    Post subject: Reply with quote

hielvc,
I know that. As I said before there are 15 combinations of at least one tc update. I just tried to give my best figuring out what is best in which case. I pushed one gcc more so that we start using the new compiler as soon as possible. Your help will be very appreciated here. What I know tells me this:
The priority for merging is as follows:
os-headers, glibc, gcc, binutils
(... or, perhaps better os-headers, gcc, glibc, binutils - I really don't know)
Furthermore, since os-heasers don't get compiled then just emerge them twice in the beginning.
I did a little bit of thinking and I am convinced now that I was overcomplicating things. As I see it rebuilding the TC should be in 2 steps:
STEP 1: Update all TC packages to the newest version.
- This part should vary according to available updates
STEP 2: Rebuild the TC. Fixed. I would go for glibc, bu, gcc, glibc, bu, gcc.

My question is, shouldn't we take care of portage too? I've seen it appear in the middle of the list for world updates. And although it is mainly bash and python there are a few binaries for sandboxing and stuff that get compiled.

I'm getting to work. I'll fix the toolchain and all the options I've promised. I'll post when I am ready.
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Sun Jan 30, 2005 4:32 am    Post subject: Reply with quote

Posted new version.
Addressed all requests except the thing with emerge (1 of 1).
Don't forget to read the help, options tend to change!
Use the link below to go to my post with the link.
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sun Jan 30, 2005 6:49 am    Post subject: Reply with quote

wow, this thing really took off. amazing work guys.

just as a thought on the priority, i was looking into the toolchain's build order myself a little while ago and i settled on following the LFS suggestion of headers->glibc->binutils->gcc. they actually build binutils first during the bootstrap.

Quote:
Binutils is installed first because the ./configure runs of both GCC and Glibc perform various feature tests on the assembler and linker to determine which software features to enable or disable. This is more important than one might first realize. An incorrectly configured GCC or Glibc can result in a subtly broken toolchain, where the impact of such breakage might not show up until near the end of the build of an entire distribution.


i don't know if it actually makes much difference once the system is built, but i figure in the end you have the choice of gcc->binutils or binutils->gcc and if there's no other reason to pick one over the other it can't hurt.
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Sun Jan 30, 2005 9:09 pm    Post subject: Reply with quote

Thanks for the tip, dirtyepic :lol:
I don't know if it really matters in which order I update the packages since I am re-emerging everything twice afterwards. I was trying to push gcc in front as I don't see much sense in compiling things with the old gcc.
Anyways, if one doesn't like my order she can edit the script & redefine it
:twisted:
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Mon Jan 31, 2005 12:39 am    Post subject: Reply with quote

:D
New Version again: 0.2.2.0
Cool new features and other improvements :roll:
As usual, (can't say that enough) READ HELP!
Follow, the link in my sig to go to the post with the actual link.

TODO:
(I've already laid down the bases for that so expect soon!)
- Status check. The script will write current progress info in a file and calling it from another shell with --status option will give you the info.
Can't think of any way of overriding what emerge puts on xterm titles.
- Control over which dirs are used for tmp and log
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
landon
Apprentice
Apprentice


Joined: 09 Mar 2004
Posts: 159

PostPosted: Mon Jan 31, 2005 10:24 pm    Post subject: Reply with quote

Hmm, came across this particular thread after clicking on 2 different signatured-urls. Which script is currently being developed? Is it now a joint development?
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Mon Jan 31, 2005 11:21 pm    Post subject: Reply with quote

Kinda joint development, yeah
Use mine : )) It's the latest version
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Feb 01, 2005 7:07 am    Post subject: Reply with quote

1st added a link above my script to MindErasers.

I did manage to look around some considering the build order. Dirtyepic is right. The question I have is should the order be " glibc binutils gcc && glibc binutils gcc " or should it be "glibc glibc && binutils gcc && binutils gcc". Ive seen both and and done both. I wonder which is more correct or if either is more correct. Any all knowing peeps out there?

Minderaser as to portage I agree I like to emerge it first then the rest of whatever.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Tue Feb 01, 2005 8:36 am    Post subject: Reply with quote

hielvc wrote:
The question I have is should the order be " glibc binutils gcc && glibc binutils gcc " or should it be "glibc glibc && binutils gcc && binutils gcc". Ive seen both and and done both. I wonder which is more correct or if either is more correct. Any all knowing peeps out there?


the first one definitely. you want to build the new toolchain as a whole, and then build it again with itself.

think of glibc as your knowledge base, binutils as your toolbox, and gcc as your factory. while you could go to school twice, buy yourself a toolkit, build a factory, build better tools with your new factory, and rebuild the factory with your new tools, you can alternatively go to school, get a toolkit, build a factory, learn everything about how your new factory works, take that knowledge and build a new set of tools, then rebuild the factory with complete knowledge of how it operates with tools specifically designed for it. both ways work, but the second is optimal.
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Tue Feb 01, 2005 11:09 am    Post subject: Reply with quote

Hey, I got it right then :lol:
At least the rebuilding part :!:
What about the order of initial updating/building?
Do you agree it should be
portage - 2x os-headers - glibc - binutils-cfg - binutils - gcc-cfg - gcc :?:

hielvc, thanks for the link :D

How do you guys find the new features?
Any bugs, corrections, wishes?
_________________
Portage updates your ToolChain (gcc, glibc...) in a very stupid way. Here is the solution: an emerge wrapper script
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Wed Feb 02, 2005 6:48 pm    Post subject: Reply with quote

TCupdate is coming along fine now - Well Done !!!
'tcupdate.sh -tep' gives the following :-
Code:
[ebuild   R   ] sys-apps/portage-2.0.51-r15
[ebuild   R   ] sys-kernel/linux26-headers-2.6.8.1-r3
[ebuild   R   ] sys-kernel/linux26-headers-2.6.8.1-r3
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-config-1.8
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-config-1.3.10-r1
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-apps/portage-2.0.51-r15


This seems a bit excessive but I am sure it's easily fixed.
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Wed Feb 02, 2005 8:34 pm    Post subject: Reply with quote

Running tcupdate.sh with the '-e' option picks up up all the toolchain packages and adds them to 'tc_list'.
Then for the forced 'tc' update the 'tc_build' is added to this list produced the list in the previous post.
I have modified my copy adding a 'do_tc_force' flag and using this so the 'tc_list' is made to equal only 'tc_build'.
It works fine now
Code:
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-libs/glibc-2.3.4.20041102
[ebuild   R   ] sys-devel/binutils-2.15.92.0.2-r2
[ebuild   R   ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-apps/portage-2.0.51-r15

Thanks for your efforts.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, ... 28, 29, 30  Next
Page 2 of 30

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum