An emerge wrapper for breaking emerges into chunks

[hielvc]

I just did the 2005.1 build and except for a problem with mysql I had no probs. Then again I still need to run " emwrap.sh -db && emwrap.sh -rb" . This is the first time Ive built a x86 system, not a ~x86. except for gcc-3.4.4 and binutils. The probs Ive been haveing are xorg freezing which I came came across when I was trying to figure out my mysql prob. taipan67 your plan should work fine.

[taipan67]

...I want to try two different ideas; The first is a Stage1/3 on a 2005.1-athlon-xp tarball (which i've heard might be buggy), upgrading 'gcc' to 3.4.3.20050110, which should definitely need the 'gcc-config' run. As i prefer not to modify my CFLAGS during such installs, i should be able to just use the script, without an initial manual-emerge to allow for tweaking...

The second idea i want to test is as above, but on a Jackass! 2005.0-athlon-xp tarball, to find out if updating switches it over to the 2005.1 profile. This alternative would already have the required gcc-version in situ, but the toolchain will still get rebuilt to modify glibc's locales. This will therefore test the gcc-config part of the script in a situation where no adjustment is needed.

I'll check back before i start (in the morning), in case anyone spots any howling errors in my grand-plan that i've been too blinkered to notice...

Well, my Thursday went to shit in a hurry - how was everyone else's?

Nothing wrong with the script, i just didn't get to run it's first pass until 10pm!

So far, i'm in the final stages of the first install-method quoted above, but as it's ended up incorporating all of the script-tests i envisaged from both methods, i see no point in waiting until after the second one before posting the results...

I ended up doing the equivalent of :-

Code: Select all

emwrap -ets && emwrap -ets

...though i broke it down into separate toolchain & system runs, to better monitor what was going on. On the first run of 'emwrap-et', 'gcc' was correctly re-profiled by the run of 'gcc-config', & on the second (when the rebuilt compiler was already selected), the correct profile was maintained. As i write this, the second run of 'emwrap -es' is merrily going about it's business in the background.

So! To both maguire & hielvc, my profuse thanks for all of your industry, & congratulations on successfully navigating at least this scenario.

I did encounter one tiny little bug, which i've already fixed & tested on my copy; If you run 'emwrap -etf', it errors out because 'gcc-config' tries to update to the 'newly-fetched-but-as-yet-unbuilt' version of 'gcc'. That's no real problem when it's just the toolchain, but if the options had been '-etsf' or '-etswf', it could be...

I fixed this by enclosing lines 942 thru 1010 with this conditional-check :-

Code: Select all

# Skip entire 'gcc-config' section if only fetching sources:
if [ ! -n "$fetch_flag" ] ; then
   ......
fi #<--End of the 'fetchonly' exclusion-check.

...After making this modification, the '-etf' & '-etp' options both worked as i'd hoped (i didn't try '-et' again ).

That's my first little bit of scripting, & maguire had already provided the variable & all the examples i needed, so it's not too much to beat my chest about. If any of you boffins have a better solution, i'll look forward to learning from it...

[U-stolemynick]

Ok, one thing that's always bugged me is wanting to have a cron job do an emerge -u world once a week or so (I'm currently 100% arch=x86), but being afraid to because of steady updates to linux-headers and glibc being in portage. These scripts are addressing that but are being updated/edited alot so I'm still not comfortable with using them for automatic updates yet. Hmm, I read the thread but now I'm not sure, will these scripts break when there's an update to portage? Anyway,

If I add my current versions of binutils, gcc, glibc, and linux-headers to package.mask I can write a weekly crontab that does something like emerge -u world && emerge sync && emerge -upv world and emails the output of the -upv. Then I have it in my inbox to read or ignore at my liesure a week before the next update. Also, cron would email me if emerge failed to run (i.e. an updated binutils is a dependancy of something in my world file). When that happens I can use the emwrap.sh script but otherwise use emerge or porthole. I'm thinking that I might be able to keep my systems really up to date yet go months at a time without paying attention to it. Does this make sense?

I have a few "grandmothers" that are using gentoo but cannot take care of their own systems so I'm always looking for ways to automate routine updates yet be alerted when real attention is required.

Of course I think the thing to do would be to create a toolchain meta and seperate it out from system, but I'm no dev so there's probably a good reason why that wasn't done already.

Thanks for doing this, all of you.

Rob

[sfp-a7x]

I ran into a bug in emwrap v4.2:

Code: Select all

# /usr/local/bin/emwrap -pDNus
--pretend (p)
--deep (D)
--newuse (N)
--update (u)
--system (s)

Building a list of EMERGE WORLD packages...
Building a list of EMERGE SYSTEM packages...
Building a list of (emptytree) TOOL CHAIN packages...

A Tool Chain config-file or portage update was found.
'emerge'ing these packages will only occur with a TC build:

   binutils-config-1.8-r5

No primary Tool Chain updates are available.

1 of 0  SYSTEM packages:
system_list.txt

These are the packages that I would merge, in order:

Calculating dependencies   
emerge: there are no ebuilds to satisfy "=system_list.txt".

Warning: emwrap: system_list.txt failed to build.  'emerge' exited code: 0.

NO SYSTEM or WORLD packages were 'emerge'd.


Packages that failed to 'emerge':
system_list.txt

Done!

also:

Code: Select all

# /usr/local/bin/emwrap -pDNuts
--pretend (p)
--deep (D)
--newuse (N)
--update (u)
--toolchain (t)
--system (s)

Building a list of EMERGE WORLD packages...
Building a list of EMERGE SYSTEM packages...
Building a list of (emptytree) TOOL CHAIN packages...

A Tool Chain config-file or portage update was found.
'emerge'ing these packages will only occur with a TC build:

   binutils-config-1.8-r5

No primary Tool Chain updates are available.

NO TOOL CHAIN packages were 'emerge'd.

1 of 0  SYSTEM packages:
system_list.txt

These are the packages that I would merge, in order:

Calculating dependencies   
emerge: there are no ebuilds to satisfy "=system_list.txt".

Warning: emwrap: system_list.txt failed to build.  'emerge' exited code: 0.

NO SYSTEM or WORLD packages were 'emerge'd.


Packages that failed to 'emerge':
system_list.txt

Done!

What other information should I include?

[taipan67]

I ran into a bug in emwrap v4.2...

This one cropped up before, on this post.

The solution was to manually emerge the update for 'portage' first, by the look of things...

[hielvc]

U-stolemynick emwrap.sh will not update the TC unless the " t or b " switch is used . If you use cron then I would sugest

Code: Select all

emerge sync 2>&1 && emwrap.sh -wuD 2>&1

The -wud >>> world -uD.
The only problem still left would be "etc-update"

[sfp-a7x]

I ran into a bug in emwrap v4.2...

This one cropped up before, on this post.

The solution was to manually emerge the update for 'portage' first, by the look of things...

Ah, I glossed over that previous posting a long time ago so I didn't realize it was the same bug. Sorry for the duplicate. However, it's not an upgrade to portage -- it's an upgrade to binutils-config (1.8-r4 to 1.8-r5). I imagine updating binutils-config by itself first would make the problem go away. Perhaps this bug only shows up when there are updates to the toolchain packages that don't require a rebuild of the whole toolchain?

[sam_i_am]

Also, please let me know if the gcc re-build (where gcc-config is automatically run) works. As far as I know, that aspect remains untested.

I do see it when I do the --pretend. Also I remember seeing it flash by during the actual emerge. I can confirm in a few more hours as I'm updating two more machines using emwrap and I'm capturing the emerge output. Will let you know when that is done.

On another note, at work, I have these two machines (my desktop and a server) which use distcc to help each other on emerges. After reading all these stuff about TC updates, I have a feeling that distcc should be disabled for TC updates.

For this update (which includes gcc-3.4.4-r1, and linux-headers) I've disabled distcc on both machines to be on the safe side. What do you'll think?

Sam

[Pseud]

On another note, at work, I have these two machines (my desktop and a server) which use distcc to help each other on emerges. After reading all these stuff about TC updates, I have a feeling that distcc should be disabled for TC updates.

For this update (which includes gcc-3.4.4-r1, and linux-headers) I've disabled distcc on both machines to be on the safe side. What do you'll think?
Sam

Yes, distcc should be disabled during TC updates as it can create weird problems if the machines have different versions of gcc (the Gentoo distcc doc talks about this, too).
So, what I would suggest is this (the gurus here, please correct me if I'm goofing up) :
1. Do all the steps which involve rebuilding the TC on one of the two machines, with distcc disabled, having turned on the "buildpkg" feature of emerge.
2. Copy these well-baked binaries (the TC plus probably some other system packages, depending on the details of the steps in 1) over to the second machine and emerge them with the -k (usepkg) option.
3. Turn on distcc on both machines and build the rest of the packages on the first machine, repeating builds as many times as your present level of paranoia dictates
4. Again, copy over the binaries, and use the emerge -k stuff
5. Build the remaining world packages on the second machine (if any), again repeating builds as you want.
6. Feel damn kicked about the gazillion complies your machines did over the past few days/weeks

[sam_i_am]

6. Feel damn kicked about the gazillion complies your machines did over the past few days/weeks

Heh heh. I can certainly relate to that (and damn proud too )

As for testing gcc update in emwrap script, it looks like everything went as planned. Heres the relevant emerge output:

Code: Select all

... end of gcc merge ...
>>> Auto-cleaning packages ...

>>> No outdated packages were found on your system.


 * Regenerating GNU info directory index...
 * Processed 6 info files.



BEFORE selecting new GCC:
[1] i686-pc-linux-gnu-3.4.4 *
[2] i686-pc-linux-gnu-3.4.4-hardened
[3] i686-pc-linux-gnu-3.4.4-hardenednopie
[4] i686-pc-linux-gnu-3.4.4-hardenednopiessp
[5] i686-pc-linux-gnu-3.4.4-hardenednossp

**********************************************************************
I'm going to execute:  # gcc-config i686-pc-linux-gnu-3.4.4
**********************************************************************
Kill me NOW, or forever hold your peace!
(sleeping for 15 seconds...)
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
(continuing...)
 * Switching to i686-pc-linux-gnu-3.4.4 compiler ...                      [ ok ]


AFTER selecting new GCC:
[1] i686-pc-linux-gnu-3.4.4 *
[2] i686-pc-linux-gnu-3.4.4-hardened
[3] i686-pc-linux-gnu-3.4.4-hardenednopie
[4] i686-pc-linux-gnu-3.4.4-hardenednopiessp
[5] i686-pc-linux-gnu-3.4.4-hardenednossp

5 of 7  TOOL CHAIN packages:
...[snipped rest of emerge output]

[todw1fd]

Does Emwrap play nicely with http-replicator? I'm just about finished running updating my whole server with emwrap and would like to do the same on two client boxes using bin packages built on the server. Does emwrap accept the -g option to use packages from the binhost? Would it be possible to add this functionality if it doesn't currently?

[hielvc]

Well emwrap.sh does and emwrap should also. If your make.conf is settup to use http-replicator and you have no problms with a noraml plain jane emerge then it will work fine, I use it to with no probllems.

[todw1fd]

emwrap doesn't take the -g option. Is there another way? Is there something I could put in the FEATURES variable to get it to always use bin packages from another server if possible?

EDIT : Ok, read first ask second. Looks like you can add "getbinpkg" to FEATURES...should take care of this.

[kopte]

Hi,

i do appreciate your work it is a really good script. I had always masked my actual toolchain, to not mess it up. This is no longer needed.

Greetings
kopte

[SlackwareInAZ]

maguire,

Just thought you would like to know that I recently installed Jackass-2005.1 and then used your emwrap with -etsw to completely rebuild and update the system. Took about 20 hrs on a 2.4 P4 with 768mb ram. No errors. Ran dispatch-conf when I was done to safely view and update about 10 conf files.

One question. Being familiar with Slackware, followed by using Arch to become proficient before I tried Gentoo, what would be the emwrap equivalent to pacman -Syu in Arch?

Thanks for any assistance,
Gary

[davidblewett]

When I ran emwrap -ut, everything ran fine except the switch to the new gcc. I had been using 3.3.6, and wanted it to upgrade to 3.4.4. When it got to that point it actually ran

Code: Select all

#gcc-config i686-pc-linux-gnu-3.4.4
i686-pc-linux-gnu-3.4.4-nopie
i686-pc-linux-gnu-3.4.4-nopiessp
i686-pc-linux-gnu-3.4.4-nossp 

Including the line breaks, as one command. This obviously failed, and stopped the process.

[piavlo]

While trying to update the TC because of binutils upgrade on amd64,
but since i also have cross i686 TC installed by crossdev, i get the following error.

Code: Select all

#emwrap.sh -twp

...
Calculating dependencies   

!!! The short ebuild name "=binutils" is ambiguous.  Please specify
!!! one of the following fully-qualified ebuild names instead:

    sys-devel/binutils
    cross-i686-pc-linux-gnu/binutils

binutils failed to build. Stoping script.

[hielvc]

Sorry Im lat ein responding but I just got home. Youve got an interesting problem. Have you run " emerge system -etp ". There are several progs in the portage tree that have the same name but different category. Yours though isnt in my portage tree "cross-i686-pc-linux-gnu " you would noramlly emerge these by " emerge =cross-i686-pc-linux-gnu/binutils " or "emerge =sys-devel/binutils "

[piavlo]

There are several progs in the portage tree that have the same name but different category.

Then how does emwrap.sh behaves if i have two such progs installed and listed in the world file (supposing these
progs are NOT part of TC as in my case)?

Yours though isnt in my portage tree "cross-i686-pc-linux-gnu "

As i mentioned cross-i686-pc-linux-gnu was installed by crossdev as a cross compiler
and it's under my overlay.

you would noramlly emerge these by " emerge =cross-i686-pc-linux-gnu/binutils " or "emerge =sys-devel/binutils "

Of course i could merge it that way, but it would be nice if emwrap.sh could take care of this case.
I maybe it should just always use the sys-devel/binutils sys-devel/gcc sys-libs/glibc and sys-kernel/linux-headers
ebuilds to aviod the ambigiosity problems with TC ebuild.

[hielvc]

Well Im looking at what I can find about crossdev and how portage works with it. If i have time I might install it on my sempron to help out my via c2 800 , a fast pentium.

[Kurse]

Im getting the same error as sfp-a7x. Not sure why. There are no updates my system is missin atm.

Code: Select all

# emwrap -pvuD system tool
--pretend (p)
--verbose (v)
--update (u)
--deep (D)
--system
--toolchain

Building a list of EMERGE WORLD packages...
Building a list of EMERGE SYSTEM packages...
Building a list of (emptytree) TOOL CHAIN packages...

No primary Tool Chain updates are available.

NO TOOL CHAIN packages were 'emerge'd.

1 of 0  SYSTEM packages:
system_list.txt

These are the packages that I would merge, in order:

Calculating dependencies
emerge: there are no ebuilds to satisfy "=system_list.txt".

Warning: emwrap: system_list.txt failed to build.  'emerge' exited code: 0.

NO SYSTEM or WORLD packages were 'emerge'd.


Packages that failed to 'emerge':
system_list.txt

[entity]

When emerge -e world Is Not Enough:
Unlikely, yet possible scenario when updating toolchain, system and world packages.


So, the story begins...

31337 h4x0r installs package A with emerge --oneshot package A
The same culprit installs package B normally, which depends on package A
Without remorse, he continues the same behavior by installing of package C, which depends on A and B.

Time passes and critical updates to toolchain appear: our happy hacker calls emwrap to the rescue.
Now, package A is not in the world list. Packages C and B are.

After emwrap -etsw packages C and B are compiled with new toolchain, but package A isn't. If the package A happened to be very picky about glibc version, packages C and B might be in trouble, since they still use A's libraries. In some lucky cases package A might get updated due to the version dependency in C and B.

Therefore, instead of emerge -e world, one should get a list of ALL INSTALLED PACKAGES and pass them to emerge -e, instead of world.

At least in my box emerge -ep world and emerge -e `fancy-script-to-list-all-installed-packages` differ, where fancy-script-to-list-all-installed-packages would be something like

Code: Select all

emerge -ep $(eix -I -c -n|awk '{print $2;}'|grep -v ^[0-9])

Of course, TC etc. should be removed from the output if this is to be used in emwrap'ish manner.

Just my 2¢.

Then again, I could be wrong, though.

[hielvc]

Edit Well you showed me something that I didnt realize You can run --swe Thers nothing wrong with it but I need to make sure running -wse builds in the right order, it wouldnt do if world emerged before system would it.

emwrap.sh generates its list from either the system output or the world output of emerge. It also filters out the TC componets of both. You also cant combine -s and -w. If you want to build your system and TC its emwrap.sh -esb. To bulild just the world files you use -w , but that is world minus system. If your system files number 110 and world 400 then when you use emwrap.sh your only building the world - system= 290 files. So to build everything like an "emerge world -e" = emwrap.sh -seb && emwrap.sh -we " or "emrap.sh -set && emwrap.sh -se && emwrpa.sh -we "

[psyqil]

To get the current version of my script emwrap.sh

This doesn't exactly work atm...

[hielvc]

psyqil
SIgh its on my sons sever. It will be this evening probably before he can fix it. If someone really wants it then the best cut and paste method is to use nano -wm. Its doesnt add spaces and such. To paste into nano use SHIFT+center button.

entity when this was origanlly written " emerge package -e " didnt work or just gave you the world output. Be that as it may emwrap.sh takes the output of the equivalent emerge command and puts it in a file. It then filters out the TC components leaving the order the same as was. Hopefully the output of emerge syspkg -e and the output of emerge system -e would be the same. If its not then portage has some big problems and its time to SourceMage and start casting spells