An emerge wrapper for breaking emerges into chunks

[hielvc]

One of the nice things about MindEraser's script is that the "-p or --pretend" works with the other switchs. This allows you to see whats going to be done. To understand this script better, play with "emerge world -ep|wc " and then emerge system -ep|wc" the same with -uD. Redirect them into files and look at them.

evoweiss technically portage and gcc-config arent part of the basic toolchain but they are part of gentoos toolchain and toolchain config. I started this with the idea that most people who would be interested are staying on top keeping their system pretty well up to date already. and so including portage and gcc-config werent that important. When I check out what emerge wants do for me I usually will emerge portage by its self to get it out of the list and that is why MindEraser included it and it is part a gentoo system toolchain . I guess you can put it this way " gentoo without portage would be LFS, LInux From Scratch." and thats not a put down of LFS. At the time when Minderaser added gcc-config to tcupdate.sh it seemed to depend on what phase the moon was in as whehter gcc-config would change your profile for you or not. At least once I was half way through a long emerge before I thought to check and it hadnt switched of course.

[reteo]

TODO:
...
Can't think of any way of overriding what emerge puts on xterm titles.

Doesn't "FEATURES=notitles" work?

[Bob P]

Moved from Portage & Programming.

Unsupported Software!?!

Congrats, Gents. you are now Officially members of the Lunatic Fringe.

[kimchi_sg]

Congrats, Gents. you are now Officially members of the Lunatic Fringe.

Come on... which sane user would want to touch (no, remerge) his toolchain with a 5-foot long wooden pole (this script) ?

[mlybarger]

could someone please give a 1000ft. overview of what these scripts are intended to do? it's a tad challenging to pick it out of the 4 pages of posts here and other related threads.

my impression from reading these threads is that emerge has some major issues with upgrading "system" type stuff, and sometimes it needs to emerge them twice?

basically my problem is that i'm getting tons of unresolved symbols when trying to start xorg. i've tried various versions of xorg in portage, each build and install ok, but none will start. this just started recently, perhaps after doing an "emerge -uD world". i've been considering just backing up /etc and /home and other areas, then re-installing, but would like to avoid that if possible just to keep as much stuff in tact. this seems like perhaps one solution to rebuild/clean the system w/o impacting other stuff.

any comments would be most appreciated.

Thanks~
~mark

[reteo]

It would work something like this:

Using tools: old-gcc, old-glibc, old-binutils, old-linux26-headers:

build new-linux26-heders
build new-gcc
build new-glibc
build new-binutils

New tools are now made, but they've been built using the logic of the old tools; this could pose problems if enough changes were introduced to confuse the optimizations, or especially if there is a bug in the old version of one of those tools. So we repeat the process so the new tools are remade, but using themselves rather than their previous versions.

Using tools: new-gcc, new-glibc, new-binutils, new-linux26-headers:

build new-linux26-heders
build new-gcc
build new-glibc
build new-binutils

They are now rebuilt, but using themselves as a base, thus we have a system toolchain in parity with itself. Since they all depend on each other to work properly, it's important all are remade when one of them is updated... and before they are used to build anything else.

I hope I've gotten all the necessary cogent points elaborated.

[hielvc]

the basic progs needed to build a TC and the order built : linux-headers glibc binutils gcc. For an explanation of these particularly linux-headers and glibc and why they should be installed first "emerge app-doc/linuxfromscratch-html " and read chapters 5 and 6.

Currently when portage builds the TC its done willy nilly. I just did an "emerge system -ep>>sys.lst and here is the order listed.

10 [ebuild N ] sys-devel/binutils-config-1.8-r1
11 [ebuild N ] sys-devel/binutils-2.15.92.0.2-r7
......
45 [ebuild N ] sys-devel/gcc-config-1.3.10-r1
46[ebuild N ] sys-devel/gcc-3.4.3.20050110-r1
47[ebuild N ] sys-libs/libstdc++-v3-3.3.4
48[ebuild N ] sys-kernel/linux26-headers-2.6.8.1-r2
49[ebuild N ] sys-apps/sysvinit-2.86
50[ebuild N ] sys-apps/baselayout-1.11.10-r4
51[ebuild N ] sys-libs/glibc-2.3.4.20050125-r1

If you look in your /var/log/portage/linux-headers you'll see that at the end it even says that its recomended that you re-emerge glibc. Now is this build order super critical no its not. As I said in Bob_Ps Stage 1/3 Installation with GCC 343 on 2005.0-rc5 = Problems

Bob tcupdate.sh uses the order " glibc binutils gcc && glibc binutils gcc". Both work, hell going by the forums 98 to 99% of the time "emerge -uD/or -e " work well other wise there would be tons of post complaining about " me broke after emerge -uD .. " and traceing it back to a broken TC. It happens, but a hugh majority of post are about newbei errors, this package is blocked or broken, I optimized the shit out my CFLAGS so that when I jump out the window with it we go faster but then it breaks , and portage circular dependenceies that crop up and go away, and how to use emerge. If you go by the number of registered users 80747 and figure just a quater of them use gentoo full time and update once or twice a year, then a very small percentage of people have problems. Its mainly us nutters jumping out the window

The advantage of useing tcupdate.sh is that you have control of when and if you want to update your TC, preventing one set of pesky probs.

[eandry]

Excellent work on this script. I use it everytime there's a change in toolchain components. And for the most part it works great!

After rebuilding a new TC, I rebuild 'system' and then 'world' packages to take advantage of any new TC features.

However when I build 'world', packages in 'system' are also rebuilt. Is there a way to keep this from happening?

Here's the tcupdate.sh commands I used in the order described above:

Code: Select all

tcupdate.sh -et

Code: Select all

tcupdate.sh -ew system

Code: Select all

tcupdate.sh -ew world --exclude='system'

On the third use of tcupdate.sh, it just rebuilds the entire system including system and world packages. Unless I specify '-e' isn't there a chance that I won't get a clean rebuild of the entire system? From the help documentation looks like exclude is experimental. Is this still the case? I'm using version 2.2.0.

[mope]

When I do this:

tcupdate.sh -ew --exclude='system' world

it looks like it excludes the system files.

Am I looking at it correctly?


Also, there doesn't appear to be linux26-headers anymore.
Is it time to update the scripts again?

(and which one do we use anyway? are they both essentially the same?)

[agent_jdh]

When I do this:

tcupdate.sh -ew --exclude='system' world

it looks like it excludes the system files.

Am I looking at it correctly?


Also, there doesn't appear to be linux26-headers anymore.
Is it time to update the scripts again?

(and which one do we use anyway? are they both essentially the same?)

w.r.t. the 2nd point, it looks like with the 2005.0 release, linux26-headers has been consumed by linux-headers, which now uses 2.6 kernel headers by default; also development-sources is no more, vanilla-sources is now a 2.6 kernel. This is on x86 at least.

[mope]

also development-sources is no more, vanilla-sources is now a 2.6 kernel. This is on x86 at least.

sweet!

only problem is that the latest gentoo-sources hangs my computer when I reboot.
don't know what that's about, but I was waiting for dev-sources to figure it out. bummer, now it looks like I just have to refrain from rebooting!

[wiktorw]

Hello,

Just wanted to say that I used the "tcupdate.sh" script with great success.
My intention was to use it as the preliminary step for a new Gentoo installation.
The method I used:

Code: Select all

# tcupdate.sh -e -t -s

Of special interest may be the length of this process.
<irony mode="sweet"> Whew, it took only short 10 hours... </irony>
But at least I'm confident that the whole toolchain is correct.
Which bothered me sometime ago, actually.

Now it's time to follow the Stage 1/3 Installation - Gentoo 2005.0 & GCC 3.4.3 by Bob P
Even though I've done most of the steps myself already
(like: partitioning, untarring stage3 & portage snapshot, setting USE flags, fstab, etc.)
And a new 2.6.11-nitro2 awaits too (/me also wonders when 2.6.12 nitro comes out...)

A very fine piece of work! Thanks!

Machine:
939-pin Athlon64 3000+ on ASUS A8V Deluxe with 1GB dual-channel RAM and 120 GB SATA HDD.

Code: Select all

# emerge info
Portage 2.0.51.19 (default-linux/amd64/2005.0, gcc-3.4.3, glibc-2.3.4.20050125-r1, 2.6.11-nitro1 x86_64)
=================================================================
System uname: 2.6.11-nitro1 x86_64 AMD Athlon(tm) 64 Processor 3000+
Gentoo Base System version 1.4.16
Python:              dev-lang/python-2.3.4-r1 [2.3.4 (#1, Mar  4 2005, 20:56:19)]
dev-lang/python:     2.3.4-r1
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.7.9-r1, 1.8.5-r3, 1.6.3, 1.9.4, 1.5
sys-devel/binutils:  2.15.92.0.2-r1
sys-devel/libtool:   1.5.10-r4
virtual/os-headers:  2.6.8.1-r4
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CFLAGS="-O3 -march=k8 -ffast-math -fomit-frame-pointer -mfpmath=sse -funit-at-a-time -fpeel-loops -ftracer -funswitch-loops"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown/usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O3 -march=k8 -ffast-math -fomit-frame-pointer -mfpmath=sse -funit-at-a-time -fpeel-loops -ftracer -funswitch-loops"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox"
GENTOO_MIRRORS="http://gentoo.prz.rzeszow.pl http://src.gentoo.pl http://gentoo.netnitco.net http://gentoo.inode.at"
LANG="pl_PL"
LC_ALL="pl_PL"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="amd64 acl acpi alsa bash-completion berkdb bitmap-fonts cdr crypt dio doc dvd dvdr font-server fortran gdbm gif gpm hal inifile ipv6 java jp2 jpeg lzw lzw-tiff mime mozilla mp3 ncurses nls nptl nptlonly opengl oss pam pcre perl pic png python readline ssl tcpd tiff truetype truetype-fonts type1-fonts unicode usb userlocales xml2 xpm xrandr xv zlib linguas_pl"
Unset:  ASFLAGS, CBUILD, CTARGET, LDFLAGS

[Bob P]

you know, when i was writing the Stage 1/3 Guide for 2005.0, i thought very seriously about just turning it into a Stage 3 install, then after rebooting, having the user do something like this:

Code: Select all

# tcupdate.sh -et --keeplog
# cp make.conf.343 make.conf
# tcupdate.sh -etw --keeplog

the only downside to doing it this way is that you trade-off by adding some compile time for the sake of command line convenience. you build some world packages twice this way, and you don't do that in the Stage 1/3 guide.

hiel, is this similar to what you did on your new install?

[hielvc]

Yes But I built the TC with a oneliner.

I just updated my script and you could use the oneliner or 2 liner for changing the CFLAGS. WIth mine you would then "emwrap.sh -se" to do a "emerge system -e" minus the TC. In other words the TC wouldnt get rebuilt but everything else would be updated saveing time. tcupdate.sh Im sure does the same but Id have to look at it again.

[eandry]

When I do this:

tcupdate.sh -ew --exclude='system' world

it looks like it excludes the system files.

Am I looking at it correctly?

No you're not.

For example xorg-x11.

Code: Select all

# tcupdate.sh -p -ew --exclude='system' world | grep xorg
[ebuild   R   ] x11-base/xorg-x11-6.8.2-r1

Code: Select all

# tcupdate.sh -p -ew system | grep xorg
[ebuild   R   ] x11-base/xorg-x11-6.8.2-r1

It wants to be re-emerged if doing world or just system.

But I think though this is considered "normal" because when you use '-e' the package is removed from the repository. So of course it has to replace the package again because it's no longer there.

Am I right about this...script authors?

[hielvc]

Updated my script "emwrap.sh" changed command line args to more closely follow portage short args. the big difference sicne last bight is added "-d and -r" options. "emwrap.sh -d" does a "emerge system -e" and no TC. After running it if you run "emwrap.sh -r" it builds the wrld list minus the system files from the "emwrap.sh -d" command. That saves you from re-emergeing 100+ files

[hielvc]

If you were really quick and got the new emwrap.sh before the time of this post then the -r wont work unless you add "r " to this line, 210, in the script

while getopts ":bdehoprstz" OPT; do

[eandry]

If you were really quick and got the new emwrap.sh before the time of this...

Would have had I known you posted a link to the script somewhere!

Did I miss a new thread for "emwrap.sh"? I didn't see an update at the beginning of the thread for this new script.

Is it used in conjunction or replace tcupdate.sh?

[hielvc]

MIndEraser wrote tcupdate.sh and thats the link. I finally updated my script which Im calling emwrap.sh now, and it is the one that you have to cut and paste from the listing.
Ive got to get on the road but just though of more error. in lines 155 and 156 change the

155 tc_filter>>wrld.tst from to tc_filter>wrld.tst
156 tc_filter>>sys.tst from to tc_filter>sys.tst

Change " >> " to ">" other wise the files grow and grow or just copy it again.

[Bob P]

have either of you guys considered adding a --fetchonly option to the script?

personally, i'd like to cut down on the total rebuild time by fetching concurrently with rebuilding (running tasks in separate consoles), instead of fetching every ebuild, pausing during the download, and sequentially building it. concurrent fetching instead of sequential fetching could dramatically cut down on the total build time by essentially removing the sum of the download times from the total build time.

even though this would help peeps like me with broadband connections, the fetching concept would really help dial-up users more than anyone else.

just a thought.

[zendal]

What is to stop them from just running

Code: Select all

emerge -f system 
emerge -f world

to get the packages then to run the script.

It would download all the files and put them in /usr/portage/distfiles

[powerman]

Information in this thread help me a lot, thanks!!

You try to solve two different problems: compile toolchain in right order and minimize compilation time. I just don't understand why not use internal portage feature (buildpkg) to speedup compilation without complex scripts? Please explain why this example is wrong or not-so-fast as your scripts:

Code: Select all

rm -r $(portageq pkgdir)/All/    # just a dirty example of cleanup
emerge      linux-headers glibc binutils gcc-config gcc
gcc-config YOUR_SELECTION && source /etc/profile    # ONLY if major gcc upgrade
emerge -b   glibc binutils gcc portage
emerge -bke system
emerge -bke world

For me this is sort of resume for all 4 pages in this thread (excluding some crazy ideas like emerging linux-headers twice).

[rhill]

that's exactly what the script does, except you're building packages on the second pass. throw "buildpkg" in to FEATURES in make.conf and it does the same thing.

but how does that speed up compile times?

edit: this would give the same result as yours:

Code: Select all

# tcupdate --nodeps -etw system && tcupdate -ew --exclude=system world 

and in my .bashrc i alias that to "overhaul".

[powerman]

but how does that speed up compile times?

Speedup mean don't recompile packages when this isn't needed. My `emerge -bke system` will not recompile glibc, binutils and gcc - it will just unpack them (unpacking will take some additional time, but I think we don't notice it because compilation of other packages take much more time).

[rhill]

oh. no, the script actually takes that into account. if you use the -t flag it builds only the toolchain, if you use -w system, it builds the system packages MINUS the toolchain.