An emerge wrapper for breaking emerges into chunks

hielvc · Posted: Sun Apr 10, 2005 2:46 pm Post subject:

2 weeks ago I saw post in Portage & Programing about some weird problem and nixnut I think suggested that the person reinstall linux-headers. I was thinking what da fsck. Well the person did and it solved his probs. :!:

Personnelly if all it took was to reinstall linux-headers 4 or 5 times Id pick that over reinstalling anything else :wink:

_________________
An A-Z Index of the Linux BASH command line

drphibes · Guru Joined: 30 Nov 2004 Posts: 432

hielvc · Posted: Tue Apr 12, 2005 3:05 am Post subject:

No gcc-config and binutils-config arnt part of the toolchain, but they're used by gentoo to configure parts of the TC, so both scripts include them. As to question 2, we;re not doing this to keep our computers busy. :wink:

You have to do recompiles to build things cleanly.. Building it once doesnt guarentee that it built cleanly. Thats what disscounts the use of packages The problem is that programs are built against libaries, the most important being glibc. glibc is built aginst linux-headers, so when either of these is updated it affects the whole system. So dont causaly update them. That si probalbly what these to scripts do best for you. filter out these updates until you want to do them. Then it would wise to use one of the scripts provided at the minuim.
_________________
An A-Z Index of the Linux BASH command line

drphibes · Guru Joined: 30 Nov 2004 Posts: 432

are there non-toolchain libraries in the system list that are compiled against by other system-list packages? also, and similarly, are there libraries in the world that are compiled against by other packages in the world? if yes to first, doesn't that demand that you do emerge -e system twice? and if the second is true, emerge -e world twice?

hielvc · Posted: Tue Apr 12, 2005 2:56 pm Post subject:

Thats the safest way to go.
_________________
An A-Z Index of the Linux BASH command line

drphibes · Guru Joined: 30 Nov 2004 Posts: 432

ok, so it seems to me that

lmcogs · Guru Joined: 03 Apr 2005 Posts: 340

Hi

Approx how long does either program take to finish job, I know it depends on system, say standalone. I was having terrible bother with gcc, libstdc++ etc and reinstall gcc, binutils,bash, glibc. Ran emerge -e systme twice. That was ok then I did emerge -e world and it failed on 180 out of 460 with the following

make[4]: Nothing to be done for `all-am'.
make[4]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdetoys'
make[3]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdetoys'
Making all in kdelibs
make[3]: Entering directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdelibs'
Makefile:226: *** commands commence before first target. Stop.
make[3]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages/kdelibs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2/messages'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/var/tmp/portage/kde-i18n-3.3.2/work/kde-i18n-uk-3.3.2'
make: *** [all] Error 2

!!! ERROR: kde-base/kde-i18n-3.3.2 failed.
!!! Function kde_src_compile, Line 166, Exitcode 2
!!! died running emake, kde_src_compile:make

I emerged kde-i18 on its own and that installed ok but I am wary of running emerge -e world again because it has taken almost 24 hours to reach that failure. What do you think. Would you advise using either tcupdate or emwrap.sh. I am a novice.

lmcogs

andrewd18 · Posted: Tue Apr 12, 2005 11:09 pm Post subject:

MindEraser,

After using tcupdate.sh (0.2.2.0) with the -et options, I noticed that I was having issues whenever compiling against libbonoboui or QT. Recompiling both of those would fix any issues I had, but it took a while to figure out exactly what the issues were.

My buddy Xian had the same problem. He thinks its the recompiling of glibc that's the issue.

Is this a known issue, and would it be at all possible to modify the script to account for this? Have you seen any evidence of this?

~~ Andrew D.

*edited to further clarify the exact issue*
_________________
Keep Your Toolchain Stable! - emwrap.sh

There's no place like ::1

rhill · Posted: Wed Apr 13, 2005 8:18 am Post subject:

hielvc · Posted: Wed Apr 13, 2005 8:22 pm Post subject:

andrewd18 Its not a prob with tcupdate or emwrap. What your seeeing is why its recommended that after updating glibc that the whole shebang be rebuilt. With a new glibc when qt asks for "a" which used to return b,c but now returns c,b and now your broke :cry:

Thats why the safest policy is if you're going to update glibc or linux-headers your should do the minium equivlent of
emerge system -e && emerge world -e for emwrap it would be emwrap.sh -bd && emwrap.sh -r but it would be -130 files faster in world rebuild

and better would be
emerge system -e && emerge system -e && emerge world -e && emerge world -e
emwrap.sh -seb && emwrap.sh -d && emwrap.sh -r && emwrap.sh -r

If you see that theres an update to sometthing do an emerge <pkg> -lp , little "L". From man emerge

hielvc · Posted: Wed Apr 13, 2005 9:23 pm Post subject:

drphibes I just use FEATURES="sandbox buildpkg userpriv "

lmcogs on my Via Samuel 2 899Mhz, basically a fast pentium, the complete TC took 8 hours. On my athlon 2000XP its 2.5 to 3 hours I think. Useing the scripts adds about a 1 minute to make the package list. but it can save boo coo time in not rebuilding the TC items. In the case of a world --emptytree update, alot of time, but it was still 58 hours plus the 8hrs for the TC.
And as to

lmcogs · Guru Joined: 03 Apr 2005 Posts: 340

Hi hielvc

Thanks for reply. I will just get the following off my chest and then let you get on with developing what I see is a vital piece of software or area especially after my ordeal.

I have an athlon 1800xp and after over 50 hrs the emerge -e system, emerge -e system, emerge -e world finally succeeded. Hope this solves my woes! This invovled having restarting emerge -e world again from scratch and again I emerge -e --resume world after some kind of system upset on 409 of 460 files ( at mozilla-client).

I can not bear to start the final emerge -e world again if it has to go through the whole 460 files again, what another 48 hrs without a glitch, e.g electric cut.

This problem has taken over a week so far. I have to say I did not fiddle with CHOST but I did and update world after installation from disks (I believe this is where the problems began, when gcc etc were upgraded). Then I started out getting messages re libtool and libstc++ errors and couldn't install kde. With the help of 'mens' another thread I did

/sbin/fix_libtool_files.sh 3.3.5 --oldarch i386-pc-linux-gnu and this helped to install kde. However I started getting other problems with emerge --update --deep world when the command would stall at different packages. I then did an reemerge of gcc,binutils, glibc and finally I had re emerge bash. This was a little better but still I had problems with emerge world. I then did the emerge -e system and emerge -e world and this is where I am now.

As a newbie this experience is extremely difficult and time consuming. I can only follow what I read here in the forums and docs but I am still confuse as to what I should do really to keep the system in good order. Do I do an emerge sync, emerge --update world, emerge -e system, world or perhaps your code which if it takes that amount of time would be much better? And finally is this problem with gcc going to continue if I upgrade to latest gcc and other system related software?

lmcogs

hielvc · Posted: Thu Apr 14, 2005 11:04 pm Post subject:

This is where I differ with a lot of smart peoples on this board. If your system is working now and you see no probs with your appications that you use, then I wouldnt. You know what todo if you have gremblins. One of the probs with gentoo is that most of the time " emerge world -uD " is fine even if you upgrade linux-headers or glibc etc. but that 10th time your screwed. I use to use the old MARK-1 eye ball filter method even after I wrote the script and after it screwed up twice :oops:

, I now use the script PERIOD. There is no need to upgrade your TC in most cases so dont. Use the script to do updates and prevent MARK-1 eyball filtering failures. When you get bored and have the time go ahead and update the TC :lol:

Re-reading yor post, for normal useage just use -uD in your emerges, there is usually no need to use emptytree unless you.ve upgraded the TC. I build packages of all my emerges so I will usually upgrade binutils and gcc because I can recover easily, usually. But its what I like todo, figureing out what got fouled up. I also have a seperate home parittion which, if my system really borks, so far has saved my bacon. I store my packgages on it :wink:

Gcc has been haveing probs for the last 4 months. Its usually not gcc itself as it is selecting it or haveing it install to a nonstandard location. If not into testing then dont upgrade if it works.
_________________
An A-Z Index of the Linux BASH command line

Satori80 · Tux's lil' helper Joined: 24 Feb 2004 Posts: 137

Okay, I'm not sure I understand all of this. When this script is invoked, does it compile the new TC ebuild(s) before the toolchain "thang" is run?

Because if it doesn't, I have some real concern about the order of the building process.

Bob P · Posted: Sat Apr 16, 2005 11:27 pm Post subject:

hielvc · Posted: Sun Apr 17, 2005 12:22 am Post subject:

Both scripts require a switch tobe used to initiat TC building. WHen run in pretend mode they show what is avalible for upgrade including the TC items. If you dont use, in emwrap case the -t of -b option it doesnt build any TC packages. If either script is told to build the TC then the TC packages are built first or only, depending on the flag. Then you or the script builds the rest.
_________________
An A-Z Index of the Linux BASH command line

Satori80 · Tux's lil' helper Joined: 24 Feb 2004 Posts: 137

Thanks for your reply, hielvc, though I'm not sure it answered my question. My concern is... say I pull in a new gcc version with 'emerge -uDva world. If the script does the rebuild toolchain without building the new gcc first it seems the toolchain never gets rebuilt properly.

I've been thinking about the order here on and off for some time now. emerge linux-headers glibc && emerge glibc binutils gcc && emerge binutils gcc.

The compiler never gets re-built against itself until the end, so binutils and glibc never get compiled against the clean compiler?

Okay, from the order above; headers are put in order, then glibc gets any new functionality from the new headers, then it is rebuilt again (against an old toolchain, for some reason), then the utils get built against the new libs, then the new compiler gets compiled from the old version using the new libs and utils. Next the utils are compiled against the semi-clean compiler and then the compiler gets rebuilt against itself. Now gcc is in a clean state finally. What about the libs and utils?

Somehow that order seems out of place. Shouldn't the libs and utils be rebuilt against the compiler after it has been built against itself in its current version? I thought gcc could be flaky if this is not done. Build the new version from the old version. Then re-build the new version (which was built from the old version) against itself. Then it should be built against itself again, so that the new version is compiled from the new version. Wouldn't this be the best way to make sure you have a clean compiler? Once that is done shouldn't the libs and utils be rebuilt from the clean compiler?

Perhaps I misunderstood, but that was the impression I was under. If I'm mistaken I'd like to know. And I'm sorry if I don't understand the workings of the scripts properly, I intend no offense.

hielvc · Posted: Tue Apr 19, 2005 6:00 am Post subject:

When ever gcc is buitl it bootstraps itself which means that it builds itself 3 times. 1st time old gcc makes new gcc, then old/new makes new and that new makes new and there checked that they are the same.

Current build order of TC is linux-headers glibc gcc-config bin-config binutils gcc glibc binutils gcc

This thought will change as Im working on a system rebuild useing LFS, rubyx/heretix build order and Rob Moss's runtime dependency checker to determine the order of about 50 to 80 packages. This would help out with libs problem.
_________________
An A-Z Index of the Linux BASH command line

rhill · Posted: Wed Apr 20, 2005 3:35 am Post subject:

yeah, the best thing to do would be a rolling dependency order. like if gcc were to be upgraded it would go

gcc, glibc, binutils, gcc, glibc, binutils

if it were glibc it would be

glibc, binutils, gcc, glibc, binutils, gcc

and throw linux headers in there only if it's being upgraded (and then use the glibc example). if more than one element had an upgrade available then priority is from left to right - headers > glibc > binutils > gcc.
_________________
by design, by neglect
for a fact or just for effect

hielvc · Posted: Fri Apr 22, 2005 4:42 am Post subject:

Hum didnt think of that. On first thump it seems to make sense.
_________________
An A-Z Index of the Linux BASH command line

dol-sen · Posted: Fri Apr 22, 2005 6:00 am Post subject:

I just finished using emwrap.sh to rebuild the system and world. I have a feature request. It is not something that may happen often or at all for most people, but I have an intermittent problem with gnome that freezes up my entire sytem forcing me to hard reset my system. It happened to me several times while running the emwrap script. It seems that the emwrap uses the wrld.tst file for its -r list. Since I saw what was last emerged just before the crash I discovered that if I deleted the preceding files, that it would start the rebuild with the new list I created.

Would it be possible to modify the code to keep a running list of completed emerges, and then on a restart, continue where it left off?
_________________
Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

hielvc · Posted: Sat Apr 23, 2005 2:19 pm Post subject:

should have your rrequest done today or tomorrow., dol-sen
_________________
An A-Z Index of the Linux BASH command line

hielvc · Posted: Sat Apr 23, 2005 8:27 pm Post subject:

Ok updated emwrap.sh with -c, continue command. Basically it reinvents the "--resume" command for emerge. If your emwrap dies as dol-sen;s did, then the re-run the command, hit the up arrow on the arrow pad, type "c" then RETURN and emwrap will start from where it stoped. The way it works is after a package is built it is removed from your wrld.lst or build.lst. which when everything is built means that all the files are removed from those files. This also changes the d and r builds somewhat. emwrap use to keep the build list around. Now it doesnt as things are built they are removed. What was nice about that was you could do a sys -e one weekend and then world the next useing the original list. You can do the same thing with this newer version just dont sync. Dont like that then run emwrapsh -suD to pick up anything new and then emwrap.sh -r to rebuild the world half minus the sys files.

Fixed the broken link to emwrap.sh, new cable modem.
_________________
An A-Z Index of the Linux BASH command line

Moriah · Posted: Wed Apr 27, 2005 2:12 am Post subject: Timothy Leary visual effects

The emwrap.sh script looks interesting, and seems to be getting rave reviews, but maybe I am old fashioned; maybe my old eyes are going bad, but all those colors make it very hard to read! Perhaps I should preface that remark with the fact that I am using xterm with a white background and black text as the default color (I know, how boring, but its works for pages in a book, so it works on my screen).

It would be nice if you could use more subdued and darker colors. The yellow is especially difficult to read against the white background. I have to paste it with the mouse into a window with a different color (like black) background in order to make it out.

Also, the "clear" is a real pain too; it means I cannot scroll back and look for the command I invoked the script with, because it gets cleared.

This script is a system maintenence tool, not a video game. Could you please either tone down the colors, or provide a switch to disable them. I prefer toned down, as the colors can be an aid to readability if they are not too psychadelic.

gerard27 · Posted: Thu Apr 28, 2005 12:30 pm Post subject:

I installed Gentoo 2005.0 a while ago.
I kept adding programs and after a while I got all kinds of problems.
Like Xine and XMMS freezing.
Browsing the forum it appeared that alsa was the culprit.
I downgraded to 1.0.7 and things got better.
Then I went into KDE Control Center Sound & Multimedia>Sound System and clicked on
Full Duplex and everything froze,I mean the whole computer!
Only a hard reset made it restart.
I then found this thread about rebuilding the toolchain and all.
I did not use hielvc script but started with emerge linux-headers etc. on the first page of this thread.
When I do emerge -e system twice in succession it will error out the second time.
I made a little script to rm -r /var/tmp/portage followed by mkdir /var/tmp/portage before doing
the second emerge -e.
That kept the system from erroring out on me.
I did memtest,zero errors on my 512MB DDR

My system:
Asus A7V8X-X mobo
Athlon-xp 1700+ processor
/dev/hda 40 GB IBM deskstar with Win98
/dev/hdb 20 GB Seagate for backups and swap
Samsung 80 GB HD
/boot is on /dev/hde1
/ is on /dev/hde2
swap is on /dev/hdb3 (1 GB)

What can be the cause of this problem?