An emerge wrapper for breaking emerges into chunks

Message

hielvc · Post by **hielvc** » Sun Apr 10, 2005 2:46 pm

2 weeks ago I saw post in Portage & Programing about some weird problem and nixnut I think suggested that the person reinstall linux-headers. I was thinking what da fsck. Well the person did and it solved his probs.

Personnelly if all it took was to reinstall linux-headers 4 or 5 times Id pick that over reinstalling anything else

drphibes · Post by **drphibes** » Tue Apr 12, 2005 12:14 am

powerman wrote:Information in this thread help me a lot, thanks!!

You try to solve two different problems: compile toolchain in right order and minimize compilation time. I just don't understand why not use internal portage feature (buildpkg) to speedup compilation without complex scripts? Please explain why this example is wrong or not-so-fast as your scripts:
Code: Select all
rm -r $(portageq pkgdir)/All/    # just a dirty example of cleanup
emerge      linux-headers glibc binutils gcc-config gcc
gcc-config YOUR_SELECTION && source /etc/profile    # ONLY if major gcc upgrade
emerge -b   glibc binutils gcc portage
emerge -bke system
emerge -bke world
For me this is sort of resume for all 4 pages in this thread (excluding some crazy ideas like emerging linux-headers twice).

I learned more from this single post about portage than I have in while. Thank you. Two questions: 1) should binutils-config be considered part of the toolchain? and 2) why emerge -bke world and not just emerge -ke world given that the objective is to avoid unnecessary recompilation of the toolchain and system. Why build binaries for the world packages at all unless you need them for some other purpose?

Such a sweet post to simplify the process!

hielvc · Post by **hielvc** » Tue Apr 12, 2005 3:05 am

No gcc-config and binutils-config arnt part of the toolchain, but they're used by gentoo to configure parts of the TC, so both scripts include them. As to question 2, we;re not doing this to keep our computers busy.

You have to do recompiles to build things cleanly.. Building it once doesnt guarentee that it built cleanly. Thats what disscounts the use of packages The problem is that programs are built against libaries, the most important being glibc. glibc is built aginst linux-headers, so when either of these is updated it affects the whole system. So dont causaly update them. That si probalbly what these to scripts do best for you. filter out these updates until you want to do them. Then it would wise to use one of the scripts provided at the minuim.

drphibes · Post by **drphibes** » Tue Apr 12, 2005 3:32 am

are there non-toolchain libraries in the system list that are compiled against by other system-list packages? also, and similarly, are there libraries in the world that are compiled against by other packages in the world? if yes to first, doesn't that demand that you do emerge -e system twice? and if the second is true, emerge -e world twice?

hielvc · Post by **hielvc** » Tue Apr 12, 2005 2:56 pm

Thats the safest way to go.

drphibes · Post by **drphibes** » Tue Apr 12, 2005 6:11 pm

ok, so it seems to me that

Code: Select all

emerge linux-headers glibc binutils-config binutils gcc-config gcc
emerge -b glibc binutils gcc portage
emerge -ke system
emerge -bke system
emerge -ke world
emerge -ke world

would be a slightly more efficient way to accomplish the conventional, two-pass recompilation of the system and world. First recompile the toolchain in two passes, building binaries on the second pass. Next recompile the system in two passes, excluding the toolchain, building binaries again on the second pass. Finally recompile the world excluding all system/toolchain packages (use the binaries just built) and then again to nail it down.

lmcogs · Post by **lmcogs** » Tue Apr 12, 2005 8:36 pm

Hi

Approx how long does either program take to finish job, I know it depends on system, say standalone. I was having terrible bother with gcc, libstdc++ etc and reinstall gcc, binutils,bash, glibc. Ran emerge -e systme twice. That was ok then I did emerge -e world and it failed on 180 out of 460 with the following

make[4]: Nothing to be done for `all-am'.
make[4]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdetoys'
make[3]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdetoys'
Making all in kdelibs
make[3]: Entering directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdelibs'
Makefile:226: *** commands commence before first target. Stop.
make[3]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdelibs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2'
make: *** [all] Error 2

!!! ERROR: kde-base/kde-i18n-3.3.2 failed.
!!! Function kde_src_compile, Line 166, Exitcode 2
!!! died running emake, kde_src_compile:make

I emerged kde-i18 on its own and that installed ok but I am wary of running emerge -e world again because it has taken almost 24 hours to reach that failure. What do you think. Would you advise using either tcupdate or emwrap.sh. I am a novice.

lmcogs

andrewd18 · Post by **andrewd18** » Tue Apr 12, 2005 11:09 pm

MindEraser,

After using tcupdate.sh (0.2.2.0) with the -et options, I noticed that I was having issues whenever compiling against libbonoboui or QT. Recompiling both of those would fix any issues I had, but it took a while to figure out exactly what the issues were.

My buddy Xian had the same problem. He thinks its the recompiling of glibc that's the issue.

Is this a known issue, and would it be at all possible to modify the script to account for this? Have you seen any evidence of this?

~~ Andrew D.

*edited to further clarify the exact issue*

rhill · Post by **rhill** » Wed Apr 13, 2005 8:18 am

drphibes wrote:are there non-toolchain libraries in the system list that are compiled against by other system-list packages? also, and similarly, are there libraries in the world that are compiled against by other packages in the world? if yes to first, doesn't that demand that you do emerge -e system twice? and if the second is true, emerge -e world twice?

yes, but that would take me, for example, about a week to do. i use this script to make sure my foundation is solid. a broken package can only affect so much. a broken toolchain affects everything.

hielvc · Post by **hielvc** » Wed Apr 13, 2005 8:22 pm

andrewd18 Its not a prob with tcupdate or emwrap. What your seeeing is why its recommended that after updating glibc that the whole shebang be rebuilt. With a new glibc when qt asks for "a" which used to return b,c but now returns c,b and now your broke

Thats why the safest policy is if you're going to update glibc or linux-headers your should do the minium equivlent of
emerge system -e && emerge world -e for emwrap it would be emwrap.sh -bd && emwrap.sh -r but it would be -130 files faster in world rebuild

and better would be
emerge system -e && emerge system -e && emerge world -e && emerge world -e
emwrap.sh -seb && emwrap.sh -d && emwrap.sh -r && emwrap.sh -r

If you see that theres an update to sometthing do an emerge <pkg> -lp , little "L". From man emerge

--changelog (-l)
Use this in conjunction with the --pretend action. This will show the
ChangeLog entries for all the packages that will be upgraded.

Theres a bunch of new options

hielvc · Post by **hielvc** » Wed Apr 13, 2005 9:23 pm

drphibes I just use FEATURES="sandbox buildpkg userpriv "

lmcogs on my Via Samuel 2 899Mhz, basically a fast pentium, the complete TC took 8 hours. On my athlon 2000XP its 2.5 to 3 hours I think. Useing the scripts adds about a 1 minute to make the package list. but it can save boo coo time in not rebuilding the TC items. In the case of a world --emptytree update, alot of time, but it was still 58 hours plus the 8hrs for the TC.
And as to

That was ok then I did emerge -e world and it failed on 180 out of 460 with the following

you should have done " emerge world --resume " That picks up where it stoped. It would have started with " kde-i18 " an then continued with the REST OF " emerge world -e "

lmcogs · Post by **lmcogs** » Thu Apr 14, 2005 1:09 pm

Hi hielvc

Thanks for reply. I will just get the following off my chest and then let you get on with developing what I see is a vital piece of software or area especially after my ordeal.

I have an athlon 1800xp and after over 50 hrs the emerge -e system, emerge -e system, emerge -e world finally succeeded. Hope this solves my woes! This invovled having restarting emerge -e world again from scratch and again I emerge -e --resume world after some kind of system upset on 409 of 460 files ( at mozilla-client).

I can not bear to start the final emerge -e world again if it has to go through the whole 460 files again, what another 48 hrs without a glitch, e.g electric cut.

This problem has taken over a week so far. I have to say I did not fiddle with CHOST but I did and update world after installation from disks (I believe this is where the problems began, when gcc etc were upgraded). Then I started out getting messages re libtool and libstc++ errors and couldn't install kde. With the help of 'mens' another thread I did

/sbin/fix_libtool_files.sh 3.3.5 --oldarch i386-pc-linux-gnu and this helped to install kde. However I started getting other problems with emerge --update --deep world when the command would stall at different packages. I then did an reemerge of gcc,binutils, glibc and finally I had re emerge bash. This was a little better but still I had problems with emerge world. I then did the emerge -e system and emerge -e world and this is where I am now.

As a newbie this experience is extremely difficult and time consuming. I can only follow what I read here in the forums and docs but I am still confuse as to what I should do really to keep the system in good order. Do I do an emerge sync, emerge --update world, emerge -e system, world or perhaps your code which if it takes that amount of time would be much better? And finally is this problem with gcc going to continue if I upgrade to latest gcc and other system related software?

lmcogs

hielvc · Post by **hielvc** » Thu Apr 14, 2005 11:04 pm

This is where I differ with a lot of smart peoples on this board. If your system is working now and you see no probs with your appications that you use, then I wouldnt. You know what todo if you have gremblins. One of the probs with gentoo is that most of the time " emerge world -uD " is fine even if you upgrade linux-headers or glibc etc. but that 10th time your screwed. I use to use the old MARK-1 eye ball filter method even after I wrote the script and after it screwed up twice

, I now use the script PERIOD. There is no need to upgrade your TC in most cases so dont. Use the script to do updates and prevent MARK-1 eyball filtering failures. When you get bored and have the time go ahead and update the TC

Re-reading yor post, for normal useage just use -uD in your emerges, there is usually no need to use emptytree unless you.ve upgraded the TC. I build packages of all my emerges so I will usually upgrade binutils and gcc because I can recover easily, usually. But its what I like todo, figureing out what got fouled up. I also have a seperate home parittion which, if my system really borks, so far has saved my bacon. I store my packgages on it

Gcc has been haveing probs for the last 4 months. Its usually not gcc itself as it is selecting it or haveing it install to a nonstandard location. If not into testing then dont upgrade if it works.

Satori80 · Post by **Satori80** » Sat Apr 16, 2005 10:15 pm

Okay, I'm not sure I understand all of this. When this script is invoked, does it compile the new TC ebuild(s) before the toolchain "thang" is run?

Because if it doesn't, I have some real concern about the order of the building process.

Bob P · Post by **Bob P** » Sat Apr 16, 2005 11:27 pm

lmcogs wrote:As a newbie this experience is extremely difficult and time consuming. I can only follow what I read here in the forums and docs but I am still confuse as to what I should do really to keep the system in good order. Do I do an emerge sync, emerge --update world, emerge -e system, world or perhaps your code which if it takes that amount of time would be much better? And finally is this problem with gcc going to continue if I upgrade to latest gcc and other system related software?

you've found out first-hand how difficult and time consuming it is to maintain a properly constructed toolkit after changing one of its individual components. the answer to your question is really very simple: if it ain't broke don't fix it.

for example, i'm using GCC 3.4.3, which is a testing branch toolkit component. it takes a HUGE amount of time to rebuild a system after changing GCC, so my approach is to not implement changes to the toolkit just because an upgrade comes along -- especially for software in the testing branch. if i haven't been hit by the bug that the upgrade is designed to fix, i don't bother with the upgrade. in general, you'll find that your life is greatly simplified by building a gentoo system properly, and then LOCKING DOWN the toolkit once it is stable. this is EXACTLY what these emerge wrappers were designed to do -- rebuild your world files without tampering with your system files.

granted, this seems to be the opposite of what all of the Gentoo Ricers would have you do. most n00bs feel that its important to constantly have the most up-to-date system on their block. most Gentoo veterans don't look at it that way. instead, they focus on reliability. ymmv.

hielvc · Post by **hielvc** » Sun Apr 17, 2005 12:22 am

Both scripts require a switch tobe used to initiat TC building. WHen run in pretend mode they show what is avalible for upgrade including the TC items. If you dont use, in emwrap case the -t of -b option it doesnt build any TC packages. If either script is told to build the TC then the TC packages are built first or only, depending on the flag. Then you or the script builds the rest.

Satori80 · Post by **Satori80** » Sun Apr 17, 2005 6:19 am

Thanks for your reply, hielvc, though I'm not sure it answered my question. My concern is... say I pull in a new gcc version with 'emerge -uDva world. If the script does the rebuild toolchain without building the new gcc first it seems the toolchain never gets rebuilt properly.

I've been thinking about the order here on and off for some time now. emerge linux-headers glibc && emerge glibc binutils gcc && emerge binutils gcc.

The compiler never gets re-built against itself until the end, so binutils and glibc never get compiled against the clean compiler?

Okay, from the order above; headers are put in order, then glibc gets any new functionality from the new headers, then it is rebuilt again (against an old toolchain, for some reason), then the utils get built against the new libs, then the new compiler gets compiled from the old version using the new libs and utils. Next the utils are compiled against the semi-clean compiler and then the compiler gets rebuilt against itself. Now gcc is in a clean state finally. What about the libs and utils?

Somehow that order seems out of place. Shouldn't the libs and utils be rebuilt against the compiler after it has been built against itself in its current version? I thought gcc could be flaky if this is not done. Build the new version from the old version. Then re-build the new version (which was built from the old version) against itself. Then it should be built against itself again, so that the new version is compiled from the new version. Wouldn't this be the best way to make sure you have a clean compiler? Once that is done shouldn't the libs and utils be rebuilt from the clean compiler?

Perhaps I misunderstood, but that was the impression I was under. If I'm mistaken I'd like to know. And I'm sorry if I don't understand the workings of the scripts properly, I intend no offense.

hielvc · Post by **hielvc** » Tue Apr 19, 2005 6:00 am

When ever gcc is buitl it bootstraps itself which means that it builds itself 3 times. 1st time old gcc makes new gcc, then old/new makes new and that new makes new and there checked that they are the same.

Current build order of TC is linux-headers glibc gcc-config bin-config binutils gcc glibc binutils gcc

This thought will change as Im working on a system rebuild useing LFS, rubyx/heretix build order and Rob Moss's runtime dependency checker to determine the order of about 50 to 80 packages. This would help out with libs problem.

rhill · Post by **rhill** » Wed Apr 20, 2005 3:35 am

yeah, the best thing to do would be a rolling dependency order. like if gcc were to be upgraded it would go

gcc, glibc, binutils, gcc, glibc, binutils

if it were glibc it would be

glibc, binutils, gcc, glibc, binutils, gcc

and throw linux headers in there only if it's being upgraded (and then use the glibc example). if more than one element had an upgrade available then priority is from left to right - headers > glibc > binutils > gcc.

hielvc · Post by **hielvc** » Fri Apr 22, 2005 4:42 am

Hum didnt think of that. On first thump it seems to make sense.

dol-sen · Post by **dol-sen** » Fri Apr 22, 2005 6:00 am

I just finished using emwrap.sh to rebuild the system and world. I have a feature request. It is not something that may happen often or at all for most people, but I have an intermittent problem with gnome that freezes up my entire sytem forcing me to hard reset my system. It happened to me several times while running the emwrap script. It seems that the emwrap uses the wrld.tst file for its -r list. Since I saw what was last emerged just before the crash I discovered that if I deleted the preceding files, that it would start the rebuild with the new list I created.

Would it be possible to modify the code to keep a running list of completed emerges, and then on a restart, continue where it left off?

hielvc · Post by **hielvc** » Sat Apr 23, 2005 2:19 pm

should have your rrequest done today or tomorrow., dol-sen

hielvc · Post by **hielvc** » Sat Apr 23, 2005 8:27 pm

Ok updated emwrap.sh with -c, continue command. Basically it reinvents the "--resume" command for emerge. If your emwrap dies as dol-sen;s did, then the re-run the command, hit the up arrow on the arrow pad, type "c" then RETURN and emwrap will start from where it stoped. The way it works is after a package is built it is removed from your wrld.lst or build.lst. which when everything is built means that all the files are removed from those files. This also changes the d and r builds somewhat. emwrap use to keep the build list around. Now it doesnt as things are built they are removed. What was nice about that was you could do a sys -e one weekend and then world the next useing the original list. You can do the same thing with this newer version just dont sync. Dont like that then run emwrapsh -suD to pick up anything new and then emwrap.sh -r to rebuild the world half minus the sys files.

Fixed the broken link to emwrap.sh, new cable modem.

Moriah · Post by **Moriah** » Wed Apr 27, 2005 2:12 am

The emwrap.sh script looks interesting, and seems to be getting rave reviews, but maybe I am old fashioned; maybe my old eyes are going bad, but all those colors make it very hard to read! Perhaps I should preface that remark with the fact that I am using xterm with a white background and black text as the default color (I know, how boring, but its works for pages in a book, so it works on my screen).

It would be nice if you could use more subdued and darker colors. The yellow is especially difficult to read against the white background. I have to paste it with the mouse into a window with a different color (like black) background in order to make it out.

Also, the "clear" is a real pain too; it means I cannot scroll back and look for the command I invoked the script with, because it gets cleared.

This script is a system maintenence tool, not a video game. Could you please either tone down the colors, or provide a switch to disable them. I prefer toned down, as the colors can be an aid to readability if they are not too psychadelic.

gerard27 · Post by **gerard27** » Thu Apr 28, 2005 12:30 pm

I installed Gentoo 2005.0 a while ago.
I kept adding programs and after a while I got all kinds of problems.
Like Xine and XMMS freezing.
Browsing the forum it appeared that alsa was the culprit.
I downgraded to 1.0.7 and things got better.
Then I went into KDE Control Center Sound & Multimedia>Sound System and clicked on
Full Duplex and everything froze,I mean the whole computer!
Only a hard reset made it restart.
I then found this thread about rebuilding the toolchain and all.
I did not use hielvc script but started with emerge linux-headers etc. on the first page of this thread.
When I do emerge -e system twice in succession it will error out the second time.
I made a little script to rm -r /var/tmp/portage followed by mkdir /var/tmp/portage before doing
the second emerge -e.
That kept the system from erroring out on me.
I did memtest,zero errors on my 512MB DDR

My system:
Asus A7V8X-X mobo
Athlon-xp 1700+ processor
/dev/hda 40 GB IBM deskstar with Win98
/dev/hdb 20 GB Seagate for backups and swap
Samsung 80 GB HD
/boot is on /dev/hde1
/ is on /dev/hde2
swap is on /dev/hdb3 (1 GB)

What can be the cause of this problem?

An emerge wrapper for breaking emerges into chunks

Timothy Leary visual effects