Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
An emerge wrapper for breaking emerges into chunks
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3 ... 28, 29, 30  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Jan 20, 2005 4:23 am    Post subject: An emerge wrapper for breaking emerges into chunks Reply with quote

Filed a bug#97087 about emerge casual handling of TC updates. Bug closed, its not consider a problem at this time.

Update:This script does not do the entire job yet.,at least in one step, It helps with the TC but it does not address the problem yet of the libraries.
If you have a program, prog3, that builds against 2 libraries lib1 and lib2 from other programs, then if they havent been updated
and prog3 is,then you might have problems. If lib1 and/or lib2 get updated by the normal "emerge world -uD" or emwrap.sh prog3
could break.

tcupdate.sh and emwrap.sh prevent unplanned for TC updates. That is probably their best feature. They rebuild the TC in the right
order and only when you want to. This still leaves you open for the libaries problem. Thats why you shouldnt lightly up grade your
TC unless your prepared to do some major emergeing if something hickups. Its an attempt to
build the TC and all the system files in the order such that the libs are built cleanly in one go through as when the system is being built from scratch, after the bootstrap.

This is a compilation of form several threads. The main 2 are GCC 3.4.3 is upon us! pg3
Quote:
robmoss
Recompilation of glibc is not necessarily required, but it is recommended. It was compiled using a more buggy compiler than the one you have installed; as such, it is an absolute false economy not to recompile it. Personally, after every update of a particularly important package (kernel headers, glibc, gcc, binutils, kernel etc. etc.) I do the following:

Code:

emerge -e system
emerge -e system
emerge -e world
emerge -e world


The first command ensures my toolchain is in order; the second ensures it's been compiled with a clean and current toolchain; the third re compiles all my packages with a clean and current toolchain; the fourth ensures that all my packages are compiled against packages compiled with a clean and current toolchain.
and as
Quote:
dirtyepic
according to genlop -p, i'll have that completed just around the time gcc 4.0 goes stable. :(


The other thread was rac's A more robust installation method

This works usually,but only if you do all 4 emerges. It waste time and depending on how long its been since you upgraded your linux-headers and glibc it might break your gentoo. Now to some very smart people, who definitlly know more about linux than I, these are fightin words. Its time you rethink about whats happening in the " emerge system ....".
Quote:
emerge system -ep
[ebuild N ] sys-devel/patch-2.5.9-r1
[ebuild N ] sys-libs/zlib-1.2.2
[ebuild N ] dev-python/python-fchksum-1.7.1
[ebuild N ] sys-apps/sandbox-1.2.9
[ebuild N ] sys-libs/gpm-1.20.1-r4
[ebuild N ] sys-libs/ncurses-5.4.20050319
[ebuild N ] app-shells/bash-3.0-r11
[ebuild N ] sys-devel/gnuconfig-20050602
[ebuild N ] sys-libs/readline-5.0-r2
[ebuild N ] sys-apps/diffutils-2.8.7-r1
[ebuild N ] sys-devel/binutils-config-1.8-r3
[ebuild N ] sys-devel/binutils-2.16.1
[ebuild N ] sys-devel/m4-1.4.3
[ebuild N ] sys-devel/bison-2.0
[ebuild N ] sys-apps/sed-4.1.4
[ebuild N ] sys-devel/gcc-config-1.3.11-r3
[ebuild N ] sys-apps/texinfo-4.8
[ebuild N ] sys-devel/gcc-3.4.4
[ebuild N ] sys-libs/libstdc++-v3-3.3.4
[ebuild N ] sys-kernel/linux-headers-2.6.11-r2
[ebuild N ] sys-libs/glibc-2.3.5
[ebuild N ] dev-java/java-config-1.2.11-r1
[ebuild N ] dev-java/blackdown-jre-1.4.2.02
[ebuild N ] app-arch/unzip-5.52

the first 24 lines of emerge system -ep and what do we have zlib, ncurses, bash, readline, binutils, gcc, linux-headers and then glibc. Quit a fouled up build order.
Code:
emerge binutils gcc linux-headers glibc
It could definitlly break some peoples systems and for those whos systems arent broken its just a waste of time except for building linux-headers and glibc. All the programs and their libararies prior to those two are built against the old TC and everything after is built by a fouled up half assed new TC. The second emerge system finally builds the entire TC once, even if it is built backwards and the last 2/3 to 3/4 of the files are finally rebuilt once with a clean TC. At this point we have built 200+ packages to finally rebuild the complete TC and only a 2/3s of the other important progs and their libs with a once built TC. With the first world build the 9 files that are run time deps for things like glibc ,zlib, ncurses, bash, readline, diffutils. m4, bison, sed and texinfo are finally built against the new TC. Now we are in the first "world" and the TC has started its second build, backwards of course, so it wont be completely built and the rest of system programs are built twice now. It has also started the first go through of world files. Well the last emerge finally rebuilds the TC twice and most of the system files built with it. The world files have been built once with the once built TC and once with the twice built TC. So the TC gets built twice most of the system files get built once with the twice built TC and all the world files are built twice the last time with twice built TC. Not quite what was advertized.

For those who are new to this the 4 basic programs in the ToolChain are linux-headers, glibc, binutils and gcc. There are other programs that should be included because the TC progs use them and their libararies, but these 4 are what a system is built against, linux-headers and glibc and by, gcc and binutils. So a safer less wastefull way to do this would be
Code:
]gcc-config -l     # Important!! to make sure of which gcc your useing. Switch if necessary
emerge linux-headers glibc binutils-config binutils gcc-config gcc && emerge glibc binutils gcc && \
emerge system -e && emerge world -e && emerge world -e

This is safer and saves time.

Heres a PPE ( PissPoorExample/Explination ) Linux-headers is like road map to a hugh factory or a small city. It has the locations of where the Hard Drives, keyboard, mouse, memory, NIC, video and all the other stuff your computer has and how to talk to them. Now progarms can be written useing this info directly, but its tedious and makes for bigger programs AND they will only run on a computer that has the same linux-headers or something very very close ( nvida and ATI bin drivers). So once upon a time all the programmers got togather ( Look up Rick Cook for more info ) and had big party meeting. They all agreed that having to write programs directly against linux-headers sucked big time. So they came up with " glibc ". Now glibc is like a detailed phone book that gives you a phone number, Extentsion and exactly what info in what order and type the kernel will take and what info it will return and in what order and type to send something lets say to the disk drives. The application programmers were happy they no longer had to do the work of writing glibc code and apps code and the users liked it because they didnt have to by exact same computer that their favourite programs were written on. Now the dwarfs and drones who maintain linux-headers and glibc occassionally change things, and no, there is no truth to the rumour that they get bored or like to see apps developers jumping up and down, will change something and things that worked dont any longer. They always have a good reason for changing the phone number or putting Miss losseGosse in the video department or having Msssss TightAss answer the phone. So when the kernel developers change things, fix things or add things the map changes. To take advantage of these new toys or fixs and such you have to redo phone book. SO it behoves yourebuild glibc when linux-headers are up graded. Gcc and binutils are the building crews. They take the fancifull flights of imagination that the apps developers come up with and try to make them. Gcc knows that when deweeb mentions Miss losseGosse to put the info in for her. When gcc finishes binutils takes the parts and puts them togather. Now gcc is built against glibc so when then phone book gets upgraded so should gcc. Now gcc makes blocks out the code feed to it like lego blocks. binutils takes these blocks and yup puts togater to the prog thta apps developer dreamed up. Since works so closely with gcc you need to rebuild it also to make sure its the same page as its three buddies.

Now you have seen mentioned here and probably else where the term "libaries". If I didnt mention it above, glibc is the biggest badass library on your system. Now it is not the only lib on your system. Alot of programs make baby phone books so that programs that tie into them, multimedia, sound, video know how to communicate with them and provide facilities for doing so. How many libs are there? Well cut and paste this into a xterm " for i in /lib /usr/lib /usr/X11R6/lib ;do find $i -iname lib* ;done|wc " => 6660 6660 245874 { wc - print the number of newlines, words, and bytes in files } and thats not all of them !

Now back to "emerge system ....". You might be wondering if the TC is nailed down why go through the bother of rebuilding everything a couple times? Well because all the programs and their libraries are built against glibc in one way or the other and it gets worse. Many progarms are built against 3, 4, or more existing programs. If you were to do a single "emerge world -e" to rebuild everything, thats means everything got updated right? Wrong, that would be ideal but think about if progA is built against glibc, progB is built against progA and progC against progB glibc and progA. You emerge world and it updates progC first then progA then glibc and finally progB. Theres a damn good chance that progC will be buggy particularly if there were any sort of major changes in linux-headers and/or glibc. So to cleanly rebuild progC glibc has to come first then progA then progB then progC. But if progB was rebuilt 1st then it isnt cleanly built. ProgB is still useing progA's old libs. But wait I mentioned, I think, that glibc gcc and binutils use other programs and the same appllies to them. The problem has been around ever since you started runing gentoo or any other Linux. Have you ever heard of RPM hell? Or getting packages for Slack from LinuxMafia, they would list what glibc gcc linux-header and other pertinent info that was required for a package to run as advertised. And yes Virginia even everybodies favourite, Windows has the same dang problem. Try applying service and security updates out of order. Now before you decide to switch back to Windows how many "emerge system/world -uD" have you done without noticeing anything being broken?

Ok now we know that there are some problems but hey I havent been bit yet, then again it might be why your here reading my ravings. The one thing that everybody agrees on is if your TC aint broke dont fix it. The problem now is that people are haveing to because of the gcc-3.4.4 switch.
If you look back on my proposed modiified "emerge ... emrge ..." We have "built a full TC, built a partial TC, and built a sys which has another TC build,ass backwards, embeded in it. Now we need that sys rebuild to start building the libs and progs cleanly but we also know that several of the more important progs, because they are run time or compile time dependencies of the TC need rebuilding to really nail down the TC. So rebuilding the TC now is a waste. Here is the deep dark secret of emwrap.sh. (hint you can copy and paste this stuff to play with it) We "emerge system -ep>>sysfile". That takes the output of emerge and puts it in to a file. Now with an editor we can remove the TC commpents and clean the "[ebuild U ]" x11-libs/gtksourceview-1.2.0 "[1.1.1]" out . Or useing some of unix's tools we "cut -f2 -d "] -s" to get rid "[ebui...] and cut -f1 -d "[" to remove the version info. This leaves a list of "category/fullpackageName", x11-libs/gtksourceview-1.2.0. The scripts use sed, awk and grep to find and filter out the TC items. Any way we now have a list that we can feed to emerge with a "for" loop.
Code:
for i in $(cat sysfile); do
   emerge =$i
done

This simple loop reads one line at a time out of "sysfile" and into the varible "i" and then feeds it to emerge. It has a couple of probs though. 1 it doesnt care if the emerge is successfull or not, it just grabs the next item in the list and builds it. 2 Every program it does emerge successfully, which should be all of them, gets added to your world file. This is not good. So we add --oneshot to keep from adding the files to the world file and --nodeps which makes emerge do one file at a time. But since I have emwrap.sh all I have to do, to "emerge system -e " minus the TC is " emwrap.sh system -e " or since Im real lazy "emwrap.sh -se". If you still think Im selling snake oil add a "p" to -ep or -sep , surprise "p" is --pretend. Ok, we now can change the modified "emerge .. emerge.." but, besides lazy Im a rotten typer and I can misspell anything. So in emwrap.sh if you put a -t in -se => "-set" that means build the entire TC only, when used with --emptytree. To build the TC and the rest of the files use "b". In this case Id use "t" because after building the TC I want to check gcc-config to make sure Im useing gcc-3.4.4
Code:

gcc-config -l   # make sure of gcc before beginning
emwrap.sh -set or emwrap.sh -St
gcc-config -l    #to select new gcc
emwrap.sh -seb or emwrap.sh -Sb  # We rebuild the TC and then the rest of the system
emwrap.sh -se   or emwrap.sh -S  # doing the 2nd system emerge minus the TC


Now all thats left are the 2 "emerge world -e" which are the eqivilent of "emwrap.sh world -eb or emrwap.sh -web ". It builds the entire TC first and then everything in a regular "emerge world -e" minus the full TC. Now we are at the point where most everything in the system has been rebuild correctly including the programs that are runtime dependencies for glibc, gcc and binutils. So its time for our last TC rebuild plus the first go through of the world files. But before we do this, if you look at the Robs original post we've already built the clean TC and all the system packages against it. Now wouldnt it be nice if we could build just the world files with out the system files. Well " emwrap.sh -W " does just that. It gets the system list and world list and as it did with the TC elements it removes the system files from the world list.
Code:
emwrap.sh -wet    # rebuilds TC against clean system
emwrap.sh -W           # first build of the world files against clean TC system
emwrap.sh -W           # second rebuild and done
On my system that 256 system packages and one full TC build shorter and its at least as clean as the original method.
To tie it all to gather
Code:

gcc-config -l          # make sure of gcc before beginning
emwrap.sh -set or -St
gcc-config -l           #select the new gcc
emwrap.sh -seb or -Sb       # We rebuild the TC and then the rest of the system
emwrap.sh -se  or -S         # doing the 2nd system emerge minus the TC
emwrap.sh -wet or -St        # rebuilds TC against clean system
emwrap.sh -W             # first build of the world files against clean TC system
emwrap.sh -W             # second world rebuild and done


Not maintained and broken. A rewrite of emwrap by maguire on page 9

Not maintained, MindEraser's script tcupdate

To get the current version of my script emwrap.sh Points to Seren's emwrap.sh-r4 link.

Quote:

seren wrote:
I have mirrored both emwrap.sh (old) and the emwrap.sh-r4

http://glibcsnap.googlecode.com/files/emwrap.sh
http://glibcsnap.googlecode.com/files/emwrap.sh-r4

google code should remain up for a while



Version 4.0.8-r4. NOTE gcc_conf will now use either "config" or config-${CHOST} to set gcc-config.
Added a choice for those of you who want a different work directory, see page 24 of this thread.
Code:

#!/bin/bash
# Copyright (c) 2004-2007 Hiel Van Campen <hielvc>@{behind the at}<cox.net>
# This program is licensed under the terms of the GPL version 2.
# No blame for this prog should be assinged to Gentoo. Ell they went so far as to stick me in the
# cellar of unsuported software and all I wanted was to be is a dev like them. To drink Mai-tai's,
# chasem wild women's and drivem sporty cars. Well at least
# the mUderators promoted me to a OSER so eat your heart out
#
# Use at yee own risk. It works for me, but then I wrote it.
# Thanks to ecatmur's dep script, gentoo forums, for help_fmt and print_help and clever bashing.
#  "   MindEraser for her, I think, ideas and script, tcupdate when we first started this beast.
#  "   maguire's "emwrap" script, I borrowed his gcc-config. This will hopefully stop
#     your perfered gcc from being switched, as alot of us have had happen at one time or the
#     other. Oh maquire what a dumb script name. ;^)
# Version 3.0.6 , 12-15-05 Added saving of the failed to roots dir.
# Cleaned up and fixed formatting.
# Version 3.0.7 Added build TC once flag such that TC is built only once instead  of twice.
# Added build file and failed file detection. If found emwrap.sh will offer to build them
# instead of what you were going to do. Ergo failed is no longer moved to /root
# Removed Portage gcc-config and binutils-config from TC filtering and from TC update
# notification. When they're found they get built
# 1-20-06 Minor updates to help showing -S and -1 usage. Re-added resume filter to diff_emerge
# 2-1-06 The end of a major rewrite and the addition of several new features including adding
# libstdc to the TC  Some minor things to do or not. Better input control? Do I want to trust
# a script to switch gcc?
# 2-13-06 removed tc_filter from sys.lst and wrld.lst in to fix filtering error and save an
# unnecessary step. Also fixed TC from running -se in a -sept case. Added dagurasu suggestion for
# a CTRL-C trap.I will no longer be able to build a failed list for when I test :(
# Bumped up Rev to 4.0.1
# Fixed help. Added " exit " to print_help. 2-18-06
# Added version info to "function info" 2-17-06
# Added version to TC list in tc_emerge 3-27-06. Found a package longer than
# "-c17-60".
# Added binutils-config and gcc-config filter to remove them from wrld.lst if TC is being
# built.4-1-06
# 5-5-06 Went back to using just "cut" for file clean up. Its more general purpose than sed. Also
# added "exclude" filtering. Note "exclude or -X" have to be the last option because everything
# after them is put into the exclude_filter. Thanks to dol-sen and Tron_5
# 5-7-06 Added "-q" to flags for more concise pretend output. Also hardwired into all emerges
# for list generation. 5-8-06 moved exclude filter to wrld_lst that way it works for everything
# except the TC. 5-18-06 Changed libstdc to a variable such that if you don't have an stdc I wont
# give you one ! 5-22-06 added clean-up code to clean_up to remove the files created during sys
# file removal from wrld list in World_minus_system_emerge function.
# 5-28-06 put maguire's gcc-config in. bumped version to 4.0.5
# July and early August added removed added gcc-config and eselect joy POCKIN joy
# 8-14-06 Realized it would be nice to jsut get the version info so added version_info and
# added parameter checking to optargs.
# 8-18-06 added the exclude    url=http://www.fi3nd.net/linux/emwrap.sh
# 9-18-07 fixed changed of gcc config from /etc/env.d/gcc/config to gcc/config-i686-pc-linux-gnu
# 10-14-07 Makeing use of CHOST for gcc_config -> gcc/config-${CHOST}
# 10-15-07 Added "config" search and use for gcc-config.

PROG="emwrap.sh"
VERSION="4.0.8 with gcc-config-r4"
DATE="Mon Oct 15 21:41:56 PDT 2007"
TAG="The fuzzy tailed rats are winning, Elp"
DESC="A wrapper for emerge so you can break emerges into chunks, TC, system, world
 or world minus system"

print_help(){
      cat <<-END #|help_fmt
   
      ${GR}${PROG}  v. ${VERSION} ${Cy}"${TAG}"${NO}
      ${GR}${DESC} ${DATE}${NO}

Usage: ${PROG} [OPTION]

Date 4-23-05
Use at yee own risk. It works for me, but then I wrote it.
The main idea behind emwrap.sh is if the TC is going to be rebuilt or updated do
it first, then do the rest. The script takes the output of emerge and puts
it into a list, which we can break into chunks, the TC, system, and world.

1. Test if there is a toolchain, TC, update. emwrap.sh will not do anything to your TC
    unless you tell it to with the "t" or "b" flags. Unles they're passed, whatever emerge
    is going to do will have the TC files removed. So a "emerge system -e" will really be
    system - TC, unless you pass the "t" or "b" flag.
2. If there is and you want to, then you can rebuild all or part of your TC and then
      using an edited list to build the other items scheduled to be updated.
3. Do the update without updating the TC, toolchain.
Options:
    These first flags are the same as what you use with emerge [options] See "man emerge"
        -h or ? or  help   Display this help (default)
        -f      Fetches files only
        -u      update
        -D      deep   
        -e      Does an  emerge "--emptytree" world/system
        -p      Just like pretend in emerge. Works with all other flags.
        -N      Tells emerge to include installed packages that are affected by the new USE flags
                that have changed since the program was built.
        -B/b    build packages only or build them as you emerge packages
        resume, -r    Continues emwrap from where it stopped. To use rerun the same command
                and add "r" ${Rd}DONT${Yl} USE EMWRAP.SH FOR ANYTHING ELSE UNTIL YOU RUN THE -r${NO}
                and emwrap will start from where it stopped without having to rebuild everything
                already done.
        -K/k   use packages. For use if you have --buildpkg in your /etc/make.conf  or use -B/b
        -G/g    use packages from a  PORTAGE_BINHOST server.
        system, -s   "-s" is short for system.The same as "emerge system"
        world, -w    "-w" is short for world. The same as "emerge world"
       
For use by the script
        -q      Quite, it changes the output format in when you run pretend. Try it to see
                which output you like.
        -t      Rebuilds the toolchain components only
        -b      Rebuilds the toolchain and resumes the emerge.
        -1      For use with "t" or "b" only builds the TC once.
        -S      This is short for a "emerge system --emptytree" build.
        -W      builds the world files, "emerge world -e", BUT minus the system files.
                failed  builds the files in the failed list.
        nc      Turns off color in the script. You still get color in emerge out put.
        exclude or -X   After either must follow a space separated list of files to be excluded
                from being built. EX: emwrap.sh world -uDX gentoo-source openoffice KDE my_program
                or emwrap.sh -uDN system exclude (or -X) openoffice KDE my_program
        -V, or version    Gives short version info.
 
The flags are chainable and change whether your doing a system or world emerge and
from -uD, -N, to -e. There is also a pretend mode, -p. Like emerge -p it
shows you what will be emerged. I recommend that instead of boldly going forth, add
the -p to the end of the switch's first, to see whats going to be emerged. Then you
can remove it.

Example emwrap.sh -sep ==> emerge system --emptytree -pretend. If you remove the
      -p it will do a build of all packages in system except for the TC.
      ${Yl}NOTE -se or -S do the same thing${NO}
emwrap.sh -set or -wet ==> updates the entire TC.  If the -b switch is used instead
      of -t then the entire TC is built and then all system/world packages minus
      toolchain files.

Examples
      ${CY}emwrap.sh${NO}  prints help.
      ${CY}emwrap.sh -wuDbp${NO} Checks for updates to the TC and in the world.
      ${CY}emwrap.sh -wuDt${NO}==>  emerge world -uD and only does TC update if any.
      ${CY}emwrap.sh -wuDb${NO}==> updates, if any, the TC and then the world files.
      ${CY}emwrap.sh -f{other flags}${NO} will fetch the files for you.
      ${CY}emwrap.sh system -uDN${NO}==> emerge system -uDN, update deep newuse
      ${CY}emwrap.sh -Sb1${NO}==> emerge system -e build TC once and then system files

 emwrap.sh can "emerge system -e" with or without the TC and remove all packages
      built during the system emerge for a following "emerge world -e". This is a major
      time savings as 130+ packages wont be rebuilt during the world half.
      ${RD}NOTE -d -c -r have been changed${Rd}. -d is now -S, -W takes on the old
      role of -r.   -r is now resume and -c has been removed.${NO}
      ${CY}emwrap.sh -Sb${NO} ,builds TC and system -e. When it completes run
      ${CY}emwrap.sh -W${NO} ,this picks where -Sb stopped and builds the rest of
      the files in the " world -e ". Why use this, well if you want to break an
      " emerge world -e " up into 2 chunks for two different nights this will do it.

Here are the TC build list used in this wrapper when doing updates. The first 3 are
      double builds.
      If linux-headers   TC="linux-headers glibc \$tc_conf binutils gcc glibc binutils gcc"
      If glibc      TC_glb="glibc \$tc_conf binutils gcc glibc binutils gcc"
      If binutils or gcc   TCmini="\$tc_conf binutils gcc binutils gcc"
These are the single build TC list
      If linux-headers   TC1="linux-headers glibc \$tc_conf binutils gcc "
      If glibc      TC_glb1="glibc \$tc_conf binutils gcc "
      If binutils or gcc   TCmini1="\$tc_conf binutils gcc"

The TC build scripts are basically fall through. If you have a linux-headers update
      then the script will use TC, if its a glibc update then TC_glib and
      if binutils and/or gcc then TCmini. To force a full TC build use -set, -St or
      -wet or -seb, -Sb or -web.

If you used "exclude" when you last ran emwrap.sh and you dont use "exclude" again
   all the packages thet were execulded will be listed and an offer to now build
   them is made. if you enter any key + ENTER it will build useing the exclude
   list from your last run. If you used exclude during an -uD, update deep, then
   you can just re-run useing -uD and anything still haveing an update will be updated.
   Where this comes in handy is if you have kde and or gnome. To split out kde and
   gnome to build later nights you could do something like 1st night emwrap -wuDX kde
   gnome. 2nd night emwrap.sh -wuDX kde, this builds gnome but not kde. 3rd night
   night emwrap.sh -wuD. What happens is that exclude.lst is detected
 
END
exit
}


function color(){
   if [[ $nc != "yes" ]]; then
      #Set up colors
      NO=$'\x1b[0;0m'
      BR=$'\x1b[0;01m'
      RD=$'\x1b[31;01m' Rd=$'\x1b[00;31m'
      GR=$'\x1b[32;01m' Gr=$'\x1b[00;32m'
      YL=$'\x1b[33;01m' Yl=$'\x1b[00;33m'
      BL=$'\x1b[34;01m' Bl=$'\x1b[00;34m'
      FC=$'\x1b[35;01m' Fc=$'\x1b[00;35m'
      CY=$'\x1b[36;01m' Cy=$'\x1b[00;36m'
   fi
}

source /etc/make.conf

########  Variables ############################
bclass="none";eargs=""; count="0"
nc="" ;do_tc="";tc_once="";do_sys="";do_wrld="";both="";resume="";ftch=""
tc="no"
# Had to give "exclud" a dummy value. If its set to "" everything is filtered out 8^(
exclud="catsisrats"; exclude_list="catsisrats" ; quite="" ; tc_stdc="" ; cat_stdc=""
WORK_DIR=""
TIME=5  # sets the time the script displays failed_build info

function version_info(){
   echo "${GR}        ${PROG}  Version-${RD}${VERSION} "${NO}
   echo "${Gr} ${DESC}${NO}"
   echo "${Rd} ${DATE}${NO}"
   exit 1
}

function info(){
   clear
   cat   <<-END
   ${GR} Running ${RD}$PROG $VERSION $DATE
   ${GR} If emwrap.sh is run and it detects a ${CY}"failed' or "build"${GR} file
   ${GR} then it will offer to build them. You can ignor this by not doing 
   ${GR} anything and what you wanted todo will start after automaticlly.
   END
}

function build_info(){
   echo
   cat <<-END
   ${Yl}You can now enter ${RD}" r "${Yl} to resume building. If you dont enter anything
   then the old build file will be deleted.and ${Rd}emwrap.sh $bclass -$eargs${Yl} will
   continue. If you want to stop and scratch your head do a CRTL-C${NO}
   END
}

function faild_info(){
   echo
   cat <<-END
   ${Yl}You can now enter ${RD}" f "${Yl} to build the faild file list.If you dont enter
   anything then the faild file will be deleted and ${Rd}emwrap.sh $bclass -$eargs${Yl}
   will continue. You only need to run ${Rd}"emwrap.sh faild"${Yl} after doing a
   emptytree system or world. If you want to stop and scratch your head do a CRTL-C${NO}
   END
}

function exclude_info(){
   echo
   cat <<-END
   ${Yl}You can now enter ${RD}" x "${Yl} to build the exculded file list.If you dont enter
   anything then the excluded file will be deleted and ${Rd}emwrap.sh $bclass -$eargs${Yl}
   will be run. If you want to stop and scratch your head do a CRTL-C${NO}
   END
}

# changing work dir to $WORK_DIR/emwrap
function wrk_dir(){
   # change file locations to $WORK_DIR/emwrap/{FILES}
   # Changed to explicit directory invoke, because if used to build a system from scratch,
   # ergo a new build, $HOME isn't set and every thing for root is written to " / ".
   if [ -e ~/.emwrap ]; then
      source ~/.emwrap
   fi
   echo;echo "work_dir is $WORK_DIR"
   # If ".emwrap" exist then WORK_DIR will be what you set up, else it defaults to
   # "/tmp/emwrap"
   WORK_DIR=${WORK_DIR:-/tmp}
   
   if [ ! -d  $WORK_DIR/emwrap ];then
      mkdir $WORK_DIR/emwrap;chmod 777 $WORK_DIR/emwrap; cd $WORK_DIR/emwrap
   else
      cd $WORK_DIR/emwrap
   fi

   if [ "$(pwd)" != "$WORK_DIR/emwrap" ] ;then
      echo;echo $RD"Did not change to /tmp/emwrap. Bailing out!"$NO;echo
      exit 1
   fi
}

# gets "read input" for "chk_failed_build" function following. It un-cluters
# "chk_failed_build".Also used for eselect.
function read_input(){
#   if [ "${resume}" != "yes" ] && [ "$failed" != "yes" ] ; then
      OUTPUT="no"
      for loop_counter in 14 12 10 8 6 4 2 0 ; do
      echo -n ${RD} "$loop_counter "${NO}
      read -t 2 INPUT
         if [ "$?" == "0" ] ; then
            OUTPUT="yes"
            export OUTPUT
            # As soon as that DUMB ASS in front of the screen gets off thier ass
            # and does an "f" or an "r" ENTER, I can quit counting backwards !!!
            break 1
         fi
      done
#   fi   
}

function chk_failed_build(){
    if [ "$resume" != "yes" ]; then
      if [ "$failed" == "yes" ]; then
         mv failed build
         resume="yes"
      elif [[ -e failed || -e build || -e exclude.lst ]]; then
         if [ -s build ]; then
            echo $RD "these remain to build"$NO
            cat build
            build_info
            read_input
            resume="$OUTPUT"
            if [ "${resume}" != "yes" ]; then
               echo;echo "   $Yl Input was empty I will continue with emwrap.sh $bclass -$eargs"
               rm $WORK_DIR/emwrap/build
            fi
         elif [[ -s failed && ! -e build ]]; then
            echo $RD "these failed to build"$NO
            cat  failed
            faild_info
            read_input
            failed="$OUTPUT"
            if [ "${failed}" == "yes" ]; then
               mv failed build
               resume="yes"
            elif [ ${failed} == "no" ]; then
               echo;echo "   $Yl Input was empty I will continue with emwrap.sh $bclass -$eargs"
               rm $WORK_DIR/emwrap/failed
            fi
         elif [ -s exclude.lst ] ; then
            echo ${RD}"These were excluded can now be built"${NO}
            cat exclude.lst
            exclude_info
            read_input
            exclude="$OUTPUT"
            if [ "${exclude}" == "yes" ]; then
               mv exclude.lst build
               resume="yes"
            elif [ "${exclude}" != "yes" ]; then
               echo;echo " $Yl Input was empty I will continue with emwrap.sh $bclass -$eargs"
               rm $WORK_DIR/emwrap/exclude.lst
            fi
         fi
      fi
      # keeps exclude.lst from becomeing hugh if emwrap.sh -X is run multiple times.
      if [ -e excluds.lst ]; then
         rm $WORK_DIR/emwrap/eexclude.lst
      fi
    fi
}

# For cleaning files. if's to keep rm from complaining
function clean_up(){
    if [ -s failed ] ; then
      echo;echo ${RD} "these failed to build"${NO}
      cat  failed
    elif [[ -e failed && ! -s failed ]]; then
      rm failed
    fi
   
    if [ -s build ] ; then
      echo;echo ${RD} "these remain to build" ${NO}
      cat build
    elif [[ -e build && ! -s build ]] ; then
      rm build
    fi
   
    if [ -s exclude.lst ] ; then
        echo;echo ${RD} "these were excluded and can be built"${NO}
        cat  exclude.lst
    elif [[ -e exclude && ! -s exclude ]]; then
        rm exclude.lst
    fi
   
    # clean up
   if [ -e sys.lst ] ; then rm sys.lst ; fi
    if [ -e wrld.lst ] ; then rm wrld.lst ; fi
    if [ -e wrld.tmp ] ; then rm wrld.tmp ; fi
    if [ -e sys.lst ] ; then rm ssys.lst ; fi
    if [ -e wrld.lst ] ; then rm swrld.lst ; fi
    if [ -e sorted.lst ] ; then rm sorted.lst ; fi
}
 
function getargs(){
   if [ $# -eq 0 ]; then
      print_help
   fi
   # LONG OPTIONS
   while [[ $1 != "" ]]; do
      if echo $1|grep -v - ; then
         case $1 in
            system   )   bclass="system" ;;
            world   )   bclass="world" ;;
            help   )   print_help ;;
            resume   )   resume="yes" ;;
            nc       )   nc="yes" ;;
            failed   )   failed="yes";;
            ignore-default-opts) largs="ignore-default-opts" ;;
            exclude )   shift
                     while [[ $1 != "" ]]; do
                        if [ "$exclud" == "catsisrats" ]; then
                           exclud="$1"
                        else
                           exclud="$1|$exclud"
                        fi
                        # adding exclude_list for greping of pkg's
                        exclude_list="$exclude_list $1"
                        shift
                     done;;
            version )   version_info ;;
            * )   echo ${RD}"$OPT is not a valid long flag" ; exit 1 ;;
         esac
      fi

       while getopts "1bDefgGhKkNpqrSWstuwVX*" OPT; do
         case $OPT in
            s ) bclass="system" ;;
            w ) bclass="world" ;;
            D ) eargs="${eargs}${OPT}" ;;
            e ) eargs="${eargs}${OPT}" ;;
            N ) eargs="${eargs}${OPT}" ;;
            u ) eargs="${eargs}${OPT}" ;;
            g ) eargs="${eargs}${OPT}" ;;
            G ) eargs="${eargs}${OPT}" ;;
            k ) eargs="${eargs}${OPT}" ;;
            K ) eargs="${eargs}${OPT}" ;;
            q ) eargs="${eargs}${OPT}" ; quite="q";;
            1 ) tc_once="yes" ;;
            h ) print_help ;;
            p ) eargs="${eargs}${OPT}" ; prtnd="p" ;;
            f ) ftch="f" ;;
            S ) do_sys="yes" ;bclass="system" ;eargs="e" ;;
            W ) do_wrld="yes" ;bclass="system" ;eargs="e";;
            b ) both="yes";;
            r ) resume="yes" ;;
            t ) do_tc="yes" ; Tc="yes";;
            V ) version_info ;;
            X ) shift
               while [[ $1 != "" ]]; do
                  if [ "$exclud" == "catsisrats" ]; then
                     exclud="$1"
                  else
                     exclud="$1|$exclud"
                  fi
                  # adding exclude_list for greping of pkg's
                  exclude_list="$exclude_list $1"
               shift
               done;;
            ? )  echo ${Rd}"$1 is not a valid flag string"; exit 1 ;;
         esac
      done
      shift
   done
   if [ "$bclass" == "none" ] ; then
      echo ${RD}"You didnt select world or system so no, emwrap.sh"${NO}
      echo ${RD}"isnt broken,but you are just as dumb as me!"${NO}
      exit 1
   fi
}

# emerges from build list and keeps count.
function build_count(){
   count="0" ; s=$(wc -l build)
   for z in $(< build) ;do
      count=$(( count + 1 ))
      echo -n "${Yl}${count} of ${s} "${NO}
      # Added gcc_config which will run only when the TC is being updated.
      {
         emerge -O1${prtnd}${ftch}${quite} =${z} \
          && gcc_config ${z}
        # The $1 is for which failed function, emerge_faild or tc_failed, is to be used.
      } || $1
      grep -v "${z}" build>tmp;mv tmp build
    done
}

# merg_faild records which package fails to emerge buid_count
function emerge_faild(){
    echo "${z}">> failed
}

# The next 4 functions are TC functions
# tc_faild bails out of the build_count if emerge error
function tc_faild(){
    echo ${RD}"${z} failed to build. Stopping script."${NO}
    exit 65
}

function tc_filter(){
    awk "!/linux-head|glibc|gcc-[0-9]+|binutils-[0-9]+|libstdc+/"
}

function tc_check(){
    # testing for TC components
    # 3-27-06 removed awk when I put emerge tc in tc_emerge function to getting
    # specific version info also changed gcc and binutils regex. 5-18-06 Changed
    # libstdc testing, if you have build it, if not, don't.
    if grep -Eq linux-h wrld.lst
      then tc="linux-headers"
    elif grep -Eq glibc wrld.lst
      then tc="glibc"
    elif grep -Eq binutils-[0-9].* wrld.lst
      then tc="binutils"
    elif grep -Eq gcc-[0-9].* wrld.lst
      then tc="gcc"
    elif grep -Eq libstdc+  wrld.lst
      then tc="libstdc" ; tc_stdc="libstdc++-v3" ; cat_stdc="sys-libs/"
    fi
   #shows what TC items have updates
   if [[ $do_tc == "yes" || $both == "yes" ]] ; then
      if [ -n "$tc" ] ;then
         echo;echo $RD"      ToolChain updates found"$NO;echo
         echo ${Rd}"      $tc"${NO}
         echo;echo $BL"======================================================"$NO
         echo
      else
         echo;echo ${RD}"   No toolchain update "${NO};echo
      fi
      # for getting and building config tools in an TC build. Added libstdc so that
      # variable tc_stdc could be set, its wasnt being reached if anything above it passed.
      tc_conf="" ; tc_conf_regx=""
      if grep -Eq binutils-config wrld.lst ;  then
         tc_conf="sys-devel/binutils-config $tc_conf"
         tc_conf_regx="binutils-config $tc_conf_regx" ; fi
      if grep -Eq gcc-config wrld.lst ; then
         tc_conf="sys-devel/gcc-config $tc_conf"
         tc_conf_regx="gcc-config $tc_conf_regx" ; fi
      if grep -Eq eselect-compiler wrld.lst ; then
         tc_conf="app-admin/eselect-compiler $tc_conf"
         tc_conf_regx="eselect-compiler $tc_conf_regx" ; fi
      if grep -Eq eselect-[0-9].+ wrld.lst ; then
         tc_conf="app-admin/eselect $tc_conf"
         tc_conf_regx="eselect-[0-9].* $tc_conf_regx"; fi
      if grep -Eq libstdc+  wrld.lst ; then
         tc_stdc="sys-libs/libstdc++-v3" ; fi
      # added awk filter so not to build these two twice 4-1-06
      awk '!/binutils-config|gcc-config|eselect-[0-9].+|eselect-comp/' wrld.lst >tmp
      mv tmp wrld.lst
   fi
}

function tc_emerge(){
    # added tc-once to build TC once only which is what I do. 3-27-06 changed list
    # for useing tighter regex in grabing version explict TC items to feed tc_emerge.
    # CHanged libstdc to a variable so that if you dont have libstdc I wont give you a stdc ;^P
    TC="linux-headers glibc $tc_conf_regx binutils-[0-9].* gcc-[0-9].* glibc binutils-[0-9].* gcc-[0-9].* $tc_stdc"
    TC_glb="glibc $tc_conf_regx binutils-[0-9].* gcc-[0-9].* glibc binutils-[0-9].* gcc-[0-9].* $tc_stdc"
    TCmini="$tc_conf_regx binutils-[0-9].* gcc-[0-9].* binutils-[0-9].* gcc-[0-9].* $tc_stdc"
    TC1="linux-headers glibc $tc_conf_regx binutils-[0-9].* gcc-[0-9].* $tc_stdc"
    TC_glb1="glibc $tc_conf_regx binutils-[0-9].* gcc-[0-9].* $tc_stdc"
    TCmini1="$tc_conf_regx gcc-[0-9].* binutils-[0-9].* $tc_stdc"
    # Added tc build list for getting version info
    TC_build="sys-kernel/linux-headers sys-libs/glibc $tc_conf sys-devel/binutils sys-devel/gcc $tc_stdc"
   if [[ "${resume}" != "yes" ]];then  #prevents filling the build list after a failed TC resume
      if [[ $tc_once != "yes" ]];then
         case ${tc} in
            linux-headers   )  tc_regx=$TC;;
            glibc       )  tc_regx=$TC_glb;;
            gcc         )  tc_regx=$TCmini;;
            binutils    )  tc_regx=$TCmini;;
            libstdc     )  tc_regx=$TCmini;;
         esac
      else
         case ${tc} in
            linux-headers   )  tc_regx=$TC1;;
            glibc       )  tc_regx=$TC_glb1;;
            gcc         )  tc_regx=$TCmini1;;
            binutils    )  tc_regx=$TCmini1;;
            libstdc       )  tc_regx=$TCmini1;;
         esac
      fi
   fi
    # if for determining if build is empty before refilling it.
    if [ ! -e build  ] || [[ -e build && ! -s build ]];then
      # added emerge to get the version specific info. It also sorts the tc packages.
      :>tc.lst
      emerge -qp $(echo $TC_build) |grep "/"|cut -c17-65|cut -f1 -d " " >tc.lst
         for t in $(echo $tc_regx); do
            grep $t tc.lst>>build
         done
   fi
   echo $RD"$(< build)"$NO;echo
   build_count tc_faild
   echo;echo ${Rd}"End of ToolChain update"${NO};echo
   tc="no"
   rm tc.lst
}


function gcc_config(){
   # The eselect added 7-7-06 A varition of mcquires gcc-config
   if [ "$tc" != "no" ] ; then
      if echo "$1" | grep "gcc-[0-9].*" ; then
      # Adding maguire's gcc_config  setup here
         gcc_current=""
         gcc_new_version_number=""
         gcc_for_gcc_config=""
         gcc_current_no_dot=""
         gcc_version_number_no_dot=""
      ###############################################################
      # Let games begin
      ###############################################################
         #10-14-07 switched to CHOST usage so that i686, x86_64 and whaterver else are picked up.
         #10-15-07 added test for "config" or "config-"${CHOST}"
         if [[ -e "/etc/env.d/gcc/config" ]] ;then
            gcc_current=$(sed 's/^.*\([0-9]\.[0-9]\.[0-9]\)/\1/' /etc/env.d/gcc/config)
         elif [ -e "/etc/env.d/gcc/config-"${CHOST}"" ] ; then
            gcc_current=$(sed 's/^.*\([0-9]\.[0-9]\.[0-9]\)/\1/' /etc/env.d/gcc/config-"${CHOST}")
         fi
         # $1 is the "gcc-version" that the function build-count, passes to function gcc_config
         gcc_new_version_number=$( echo "$1" |sed 's/^.*\([0-9]\.[0-9]\.[0-9]\).*/\1/')
         ## echo "${gcc_new_version_number} TEST TEST TesT CHOST"
         # For selecting the highest version gcc
         if echo "$1" | grep -Eq "gcc-[0-9].*" ; then
            # stripping out the dots 4.1.1 ==> 411
            gcc_current_no_dot=$(echo ${gcc_current} | sed 's/\.//g')
            gcc_version_number_no_dot=$(echo ${gcc_new_version_number} | sed 's/\.//g')
            # Now finding which no_dot is larger
            if [ "${gcc_current_no_dot}" -eq "${gcc_version_number_no_dot}" ] ; then
               high_gcc="${gcc_new_version_number}"
            elif [ "${gcc_current_no_dot}" -lt "${gcc_version_number_no_dot}" ] ; then
               high_gcc="${gcc_new_version_number}"
            else
               high_gcc="${gcc_current}"
            fi
            
            gcc_for_gcc_config="${CHOST}-${high_gcc}"
            # Select the new GCC version for future compilations:
            echo
            echo "${Rd}BEFORE selecting new GCC:${CY}"
            gcc-config -l
            echo
            # Changed output of pretend to be more readable.Sun May 28 23:20:39 new time stamp.
            # For testing. use the first "if"
            #if [ "$prtnd" != "p" ] ; then
            if [ "$prtnd" == "p" ] ; then
               echo "${BL}**********************************************************************${nc}"
               echo
               echo "${Yl} I would do:${CY} # gcc-config -f ${gcc_for_gcc_config}${nc}"
               echo
               echo "${BL}**********************************************************************${nc}"
               echo
            else
               echo "${BL}**********************************************************************${nc}"
               echo "${Yl} I'm going to execute:  # gcc-config ${gcc_for_gcc_config}${nc}"
               echo "${BL}**********************************************************************${nc}"
               echo "${Yl} Kill me NOW, or forever hold your peace!${nc}"
               echo "${Yl} (sleeping for 10 seconds...)${nc}"
               for loop_counter in 1 2 3 4 5 6 7 8 9 10;do
                  echo -n "$loop_counter "
                  sleep 1
               done
               echo
               echo "${Rd}(continuing...)${nc}"
               gcc-config -f "$gcc_for_gcc_config" \
                  || {
                     return_code=$?
                     echo "${RD}ERROR:${nc} ${Rd}gcc-config $gcc_for_gcc_config failed!${nc}" >&2
                     exit $return_code
                     }
               echo
               echo "${RD}AFTER selecting new GCC:${nc}"
               gcc-config -l
               echo
               echo "${Yl}Now sourcing /etc/profile${nc}"
               source /etc/profile
               echo
            fi
         fi
      fi
   fi
}

function exclude_filter(){
   #First run grep and put the excluded files into exclude.lst
   for line in "$(echo $exclude_list)"; do
      grep "$line" wrld.tmp >>exclude.lst
   done
   awk "!/$exclud/" wrld.tmp > wrld.lst
}

# generates the the build list, checks for blocked & masked pkgs
function wrld_lst(){
   # If for blocking regeneration of list if resume or do_tc is used
   # Added exclude_filter to filters here. That way it works "W" 5-8-06
   #if [ "$resume" != "yes"  ];then
   if [[ "$resume" != "yes" && "$failed" != "yes" ]];then   
      emerge $bclass -${eargs}pq >wrld.lst
      # checking for blocked or masked packages
         if grep "blocks" wrld.lst ; then
            echo
            cat <<-END
            ${Yl}You have packages ${RD}blocking${Yl} each other. To fix run emerge
            and take care of the blockage${NO}
         END
            echo
            exit
         elif grep "masked" wrld.lst ; then
            echo
            cat <<-END
            ${Yl}You have ${RD}masked${Yl} ebuilds. If its keyword masked add it
            to /etc/portage/package.keywords. For more info see man emerge.${NO}
         END
            echo
            exit
         fi
      # Changed slice dice 5-5-06
      #  wrld.lst includes TC files if any. Which if your doing -W, a world-system emerge means they
      # get filtered out of the world list along with the system files
      # removed sed -e 's/USE=.*//' -e 's/\[[0-9].*//' for cut. Sed is to specific re modular X's VIDEO_CARDS
      #grep "/" wrld.lst | cut -c17-65 | cut -f1 -d " " | exclude_filter >>wrld.tmp
      grep "/" wrld.lst | cut -c17-65 | cut -f1 -d " " >>wrld.tmp
      if [ "$exclude.lst" != "catsisrats" ]; then
         for line in "$(echo $exclude_list)"; do
            grep "$line" wrld.tmp >>exclude.lst
         done
      fi
      awk "!/$exclud/" wrld.tmp > wrld.lst
      rm wrld.tmp
      #mv wrld.tmp wrld.lst
   fi
}

# The general purpose emerge function
function emerge_from_list(){
   # blocks do_tc from running emerge_from_list with prntd
   if [[ "$resume" != "yes" && "$failed" != "yes" ]] ;then
      #if [[ "$resume" != "yes" ]] ;then
      cat wrld.lst | tc_filter >> build
   fi
   # Now emerge   
   echo;echo -n ${YL}"$(wc -l build) to emerge "${NO}
   build_count emerge_faild
}

# This is for the generation of the world files minus the system files and building them
function World_minus_system_emerge(){
   # stops regenerating the build file if your resuming build
   if [ "$resume" != "yes" ]; then
   # wrld_list already run for system files.
   cat wrld.lst >sys.lst
   # filters out sys and TC files and puts the results into into the build list.
      if  [ "${do_wrld}" ==  "yes" ] ; then
         bclass="world";eargs="eq"
         # Calling wrld_lst function. It generates all the list
         wrld_lst >wrld.lst
         # It seems crazy but using sort comm and then re-ordering the resulting list is
         # faster than a for loop and grep -v  8}
         sort sys.lst>ssys.lst ; sort wrld.lst>swrld.lst
         comm -1 -3 ssys.lst swrld.lst > sorted.lst
         for i in $(< wrld.lst);do
            grep $i sorted.lst>>build
         done
      fi
   fi
    # Now to build what was not removed by removing the system files
    sleep 1
    echo;echo -n ${YL}"$(wc -l build) to emerge "${NO}
    build_count emerge_faild
}

# Main where the functions are called and things get done.
trap "exit 1" 1 2 3 15
color
getargs $@
info
wrk_dir
chk_failed_build
wrld_lst

#for doing resume
if [ "$resume" == "yes" ]; then
    emerge_from_list
fi

# for doing the excluded list
if [ "$exclude" == "yes" ]; then
    mv exclude build
    emerge_from_list
fi

# For building the TC
if [[ "$do_tc" == "yes" || "$both" == "yes" ]];then
    tc_check
    if [ "$tc" != "no" ];then
        tc_emerge
    fi   
fi

# For most of the emerges inculding -S
if [[ "$do_tc" != "yes" && "$do_wrld" != "yes" ]] && [[  "$both" == "yes" || -n "$bclass" ]]; then
    emerge_from_list   
fi


# does world --emptytree MINUS the system files
if [ "$do_wrld" == "yes" ]; then
    # system files already generated now removal
    World_minus_system_emerge
fi


clean_up
exit

# vim:set ts=4 sw=4:



# changing work dir to /tmp/emwrap
function wrk_dir(){
# change file locations to /tmp/emwrap/{FILES}
# Changed to explicit directory i
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Fri Jul 06, 2012 5:53 pm; edited 98 times in total
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Thu Jan 20, 2005 4:32 am    Post subject: Reply with quote

OH YES!!!
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Jan 20, 2005 5:40 am    Post subject: Reply with quote

Three things if you got Version 1.8 theres no update with out TC upgrade option :oops: Thats the -N option.

If you want to see the actual ebuild update of the toolchain item remove the " -q " form
Code:

emerge world -uDp>>wrld.lst
   if echo "linux26" | grep -q linux26 wrld.lst
      then a=1 lh2=1
   fi
   if echo "linux-h" | grep -q linux-h wrld.lst
      then a=1 lh=1
   fi
   if echo "glibc" | grep -q glibc wrld.lst
      then a=1 glb=1
   fi
   if echo "binutils" | grep -q binutils wrld.lst
      then a=1 bu=1
   fi
   if echo "gcc" | grep gcc -q wrld.lst
      then a=1 gc=1
   fi
from the grep lines.

If you want to remove or modify the list blockers you edit this
Code:
list_filter(){
   awk !/DirectFB|source/ {print $1}'}

in particular "!/DIrectFB|source/ " to block all kernel updates only change to " !/source/ "
Its easier and BETTER to use /etc/portage/package.mask.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
drphibes
Guru
Guru


Joined: 30 Nov 2004
Posts: 432

PostPosted: Thu Jan 20, 2005 6:06 pm    Post subject: Reply with quote

why emerge the linux26-headers twice? isn't it just a sea of .h files with nothing really to build?
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Thu Jan 20, 2005 8:06 pm    Post subject: Reply with quote

Hielvc, sweet idea 'n all : ))
I have some critique about the implementation though :twisted:
Ok, I'll start with the minor things:

line 202: typeo: e[m]erge
- around 160: what is this supposed to mean?
Doesn't sth like
Code:
if grep -q our.tc.keyword wrld.lst ...
do the same?
- obviously you have given up function toolchain: can be deleted!

Now, the not so trivial stuff:
-If we want to --pretend, isn't it good to exit instead of continuing?
-So we have 16 possible combinations of TC updates availability and u only use 2 possible scenarios for the 2 versions of linux??-headers (which u dont need coz u can just store the exact version in a var and use it later). So if say we just have a gcc update:
Code:
emerge linux-headers && emerge linux-headers glibc && emerge glibc binutils gcc && \
      emerge binutils gcc
We will totally UNNECESSARY and with almost NO effect emerge linux-headers, then again, then glibc too, then again glibc and binutils and finally come to the point where we upgrade gcc with the very same system we started with. Nice! Sorry, If this is not how your script works and I turn out to be a bad script reader!
The solution I would propose is: Figure out what makes most sence in each of the 15 cases ( :P This is your job, you re-re-compile the TC maniacs :P ) and assign the TC components to vars (Par1, Par2...) according to the order we want them re-compiled and pass these vars to emerge. Still we might need several scenarios with different number of emerge calls. Even if we don't act differently to each of the 15 cases a little bit more logic has to be applied than now exists. The least we could do here is start re-emerging the ebuild(s) that did change before the others and that with a predefined precedence if >1 changed.

So, what do u think of my suggestions, I hope they were useful. I will be happy to help with coding if u help me figure out what makes sence in what case.
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Thu Jan 20, 2005 10:34 pm    Post subject: Reply with quote

For all of the new users who aren't familiar with the perils of rebuilding your toolkit, it is absolutely essential to perform redundant builds of toolkit after changing a major toolkit element, such as linux26-headers, glibc, or gcc! heil's design to perform these functions redundantly was not an accident -- it was a design objective. this is what you have to do to ensure a stable, reliable, and robust toolkit.

if these steps seem unnecessary to you, then you should read this:
https://forums.gentoo.org/viewtopic.php?t=248245&start=53

it turns out that heil's script is a more efficient method of performing the same redundant operations. but instead of taking a brute force approach, heil's script hits the exact components that need to be hit, in proper order, to perform all of the necessary steps while reducing your compile time.
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Fri Jan 21, 2005 4:45 am    Post subject: Reply with quote

MInderser
Quote:

Code:
if grep -q our.tc.keyword wrld.lst ...
I'll try it and thanks the typo check.
When you run the -p, pretend, option thats all it does is checks and stops.

As to the TC rebuild, is a double emerge of linux-headers over kill, was sort of what I was leaning towards until last week when I came across a thread about "tk or tcl" and the fix from the bug report was to re-emerge linux-headers and glibc and that fixed it for 4 or 5 people. If you look at the header youll see the original TC thang only did a single linux-headers emerge . For most users of gentoo that is probably adequate. For 2 years I didnt care about the toolchain. Then I went up to linux26-headers and then glibc ~ARCH and in the threads people were haveing weird probs. They were solved by emergeing glibc twice and in some cases linux-headers twice. If your going to do this why not make damn sure and thats the point of this, its LH twice.

As to a gcc or binutils up date no I wouldnt rush out and rebuild my TC. For my use and my thoughts if LH and glibc are nailed down then just rebuilding gcc and binutils should be fine. For Robmoss and someone who is testing everything that is built then this should save them time and they will use it every time they have a TC update. But I am getting to the point where monthly or bi-monthy I am doing either "emerge system or world -e ". After really looking at how portage handles the building the TC I'll be using my wrapper. As a matter of fact the last two options I'll add will be TC with system -e and TC with world -e. The other reason why this is important to Gentoo is that its source based and everything depends on the TC. With fedora. mandrake and such you dont even need gcc.installed. Try that with Gentoo. :lol:
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
piwacet
Guru
Guru


Joined: 30 Dec 2004
Posts: 486

PostPosted: Fri Jan 21, 2005 8:24 am    Post subject: Reply with quote

Hi. Thanks for the script. Quick question. My system was build with Bob P's stage 1 NPTL on Stage 3 tarball using gcc3.4.3. I needed to umerge linux-headers and emerge linux26-headers when creating my system. I assume I should edit your script to use the same linux26-headers?

Thanks!
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Fri Jan 21, 2005 3:34 pm    Post subject: Reply with quote

No need it already checks for which version of linux-headers that your running. and then uses the tc2_filter or tc_filter.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Fri Jan 21, 2005 3:58 pm    Post subject: Reply with quote

Hielvc, Bob P,
Now don't get me wrong guys. I'm a noob only in the forums (havn't posted much) but definitely not to gentoo. I do consider emerging and re-emerging linux??-headers a good thing IF there is an update to it (especially that it is done in minutes). My point was to do all the re-emerges in the right order according to what TC updates are available. Now you know what you are doing but suppose a noob uses the script, sees an update of gcc and decides to go for the TC option in the script. This is the scenario I tried to explain in my previous post. If there is an update only to gcc and the latest linux-headers and glibc have already been re-re-emerged, wouldn't it be a redundancy to do it again? So what you are suggesting is basically go for the TC only for linux-headers and glibc updates but don't rush for gcc and binutils. Ok, that's fine and still the script would better do the right thing when someone decides to go for the TC, or?
One more thing: Again if there is no update to linux-headers but to sth else and someone goes for a TC. The script will then do the linux26-headers TC even if the user has linux-headers installed as $lh will be 0!
And a TYPEO at line (about) 220: }} brackets where ]] are needed.

I hope I have better expressed my point now and am waiting for comments. Thank you!
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Fri Jan 21, 2005 10:11 pm    Post subject: Reply with quote

No the current listing should be correct. Man version control is driving me nuts. After I tested the "if grep -q tc.word wrld.lst" I edited it posted and started try some other things and script was broken. As you pointed out my " ]]", end brackerts had transformed to "}}", end braces and either at the same time or a little latter hafl of " fi "s were gone. I dont know exactly how I managed that but clever me did it somehow.

As to " to TC or not to TC that is the question" I agree that for most people if linux-heaaders or glibc arnt updated then doing the total TC thang isnt needed. Oh well maybe things will slow down and I can work on it more on Sunday. Lets see instead of 2 more options I'll need at least 3. 1 for binutils and gcc mini-thang and sys/world -e. I thing the same guy who said "to TC or not to TC" also said "permutation, fermentation which do you perfer?"
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
marder7
n00b
n00b


Joined: 15 Nov 2002
Posts: 56

PostPosted: Sun Jan 23, 2005 12:55 pm    Post subject: Reply with quote

I just have tried to run the script, but it says:
Code:

emrg-wrapper.sh: line 252: syntax error: unexpected end of file

I double checked if anything went wrong copy&pasting it, but everything seems to be ok. My editor isn't complaining neither. I followed your installation instructions exactly.

Any idea what's wrong in the script?
_________________
This ist just another signature.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Sun Jan 23, 2005 3:33 pm    Post subject: Reply with quote

You were lucky enough to get the original first draft that was accidently posted.
I just put up the lastest revision. It changes structure clarifies help some and should run.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Sun Jan 23, 2005 5:59 pm    Post subject: Reply with quote

Found a missing brace " } " onthe toolchain function.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
marder7
n00b
n00b


Joined: 15 Nov 2002
Posts: 56

PostPosted: Sun Jan 23, 2005 8:03 pm    Post subject: Reply with quote

the same error still occurs. hm.
I have taken a close look at the code again, but since I am no shell script wizard (far from it), no wonder that I couldn't find a mistake...
_________________
This ist just another signature.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Mon Jan 24, 2005 1:36 am    Post subject: Reply with quote

marder7 Just posted version 1.8.8 dated 1.23.05 I swear its working I just used it :!: It offers correct filtering for binutils and gcc. If you do a "emerge gcc -s or the same for binutils theres about 4 packages for binutils and 8 for gcc. They are now handled correctly.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Mon Jan 24, 2005 5:18 am    Post subject: Reply with quote

Ok, Hiel! Here is my version! I hope you all will appreciate that:

Download my version of tcupdate.sh

!!! Attention. Since this version 0.2.1.0 -e option has a NEW meaning. It's been replaced by -x. Please read help!
!!! Please note - I have totally changed the way the options work. Each of -w and -t specifies if the world (rest) and the TC should be build accordingly. They can be supplied together or only one of them. -p with any combination of the other options will tell you the exact consequences of the supplied options.

Enjoy!

EDIT: Ha, and of course now one can specify the target: world or system
EDIT2: Posted again very revised and improved version
EDIT3: New options, some changes and improvements. Uploaded the script in the web
EDIT4: 0.2.2.0


Last edited by MindEraser on Mon Jan 31, 2005 12:52 am; edited 4 times in total
Back to top
View user's profile Send private message
Genfoo'
Tux's lil' helper
Tux's lil' helper


Joined: 24 Feb 2004
Posts: 112

PostPosted: Mon Jan 24, 2005 5:43 am    Post subject: Reply with quote

MindEraser, how did you do that? I mean, how did you update it before it was even written.

:D
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Mon Jan 24, 2005 6:26 pm    Post subject: Reply with quote

MindEraser I think you forgot to include the " exclude="-sources-2." list. When I run " tcupdate -pw|grep gcc " I get
Code:
 ./tcupdate.sh -pwe|grep gcc
[ebuild  N    ] sys-devel/gcc-3.4.3-r1 
[ebuild   R   ] sys-devel/gcc-config-1.3.9


Re-read the help listing and removed "e", but it still throws out the above and I cant find the "exclude" listing besides -sources-2. and its not difined :wink: Haveing become an expert in these sorts oopsise :P it was the first thing I spoted.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Tue Jan 25, 2005 12:37 am    Post subject: Reply with quote

Hiel, honestly I did not get at all what you are trying to tell me and what the problem is. The exclude list is by default "-sources-2." It is appleid to the world list when the -e option is used. If a regular expression is supplied after -e it overrides the default. A little bit more about it at the end...
If your consern is about:
Code:

[ebuild  N    ] sys-devel/gcc-3.4.3-r1
[ebuild   R   ] sys-devel/gcc-config-1.3.9

Then for gcc: it doenst mean that it will be build. Look at my output. It lists it as an available tc update but as you see it has no intention of mergeing it. For gcc-config: we havn't yet treated it specially. Should we?
Code:

MindEraser ~ # tcemerge -pw -e xorg system
 *** Following ToolChain Updates Available:
[ebuild     U ] sys-devel/gcc-3.3.5-r1 [3.3.4-r1]
Merging following world/system packages in order:
These are the packages that I would merge, in order:
Calculating dependencies ...done!
[ebuild     U ] sys-apps/man-1.5p [1.5o_p2]
[ebuild     U ] sys-devel/m4-1.4.2 [1.4.1]
[ebuild     U ] sys-apps/help2man-1.35.1 [1.33.1]
............. And so on...
MindEraser ~ #

As expected an update to xorg-x11 which would otherwise be included gets filtered out. The TC exclude list if that is what you meant is defined in the awk phrase in the tc_filter() function as it has awlays been.
And now back to the -e option. It is a little bit tricky. This comes from how getopts parses options. As I have declared that -e can accept an argument, if one does not supply one to it and puts it on the wrong place it will brake option parsing:
Good uses of -e:
Code:

#tcupdate -wte
#tcupdate -wt -e
#tcupdate -e -wt system
#tcupdate -e xorg -tw system

Bad uses of -e:
Code:

#tcupdate -epw
#tcupdate -wt -e system

In the first case it will parse pw as an argument to -e and in the second - sytem, while the user would expect that system is what is to be updated.
Maybe I should undeclare it as accepting an argument and include parsing code for --exclude=regexpr.
Besides that, do you like what I have done? Can I assume that you accept my changes? I will try to merge the TCmini functionality ASAP.
A beautiful day/night to all!!!
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Jan 25, 2005 1:57 am    Post subject: Reply with quote

No I like what youve done. I'm sort of flat tonight so Im enjoying myself by playing with your exclude. It has possibilblities.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Tue Jan 25, 2005 2:23 pm    Post subject: Reply with quote

I have run the tcupdate program with the '-t' argument and it works fine. If, however, I run 'tcupdate -e -pw system' there is no output.
It's very useful to ba able to update TC without doing the full system.
Many Thanks
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Jan 25, 2005 3:14 pm    Post subject: Reply with quote

The "print \$1 should be "print $1 in
Code:

exclude_filter(){
   awk "!/$exclude/ {print \$1}
########should be#########
exclude_filter(){
   awk "!/$exclude/ {print $1}

_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
fredor
Tux's lil' helper
Tux's lil' helper


Joined: 13 Oct 2003
Posts: 84
Location: UK

PostPosted: Tue Jan 25, 2005 3:40 pm    Post subject: Reply with quote

Adjusted 'print \$1' in exclude_filter but there is still no output with 'tcupdate -e -pw system'
If I run 'tcupdate -pw system' this seems to work OK telling me there is an update for gcc but I thought using '-e' would have listed the system files less TC.
Thanks for your quick response.
Back to top
View user's profile Send private message
MindEraser
n00b
n00b


Joined: 06 Jul 2004
Posts: 26
Location: Germany

PostPosted: Wed Jan 26, 2005 1:37 am    Post subject: Reply with quote

fredor:
You are obviously using my version. Copy the new one, the problem should be fixed. Read the help. Thanks for the bug tip : )))

hielvc:
No you were wrong about the "print \$1". I need to escape it because of the double quotes.
Posted new version.
Fixed the problem with -e (now the short option does not accept an argument). Added --exclude= for that. Read the help!
Put some logic in the tc_emrg(). Needs further development but for now should be ok. If sth in tc fails though the script exits with a warning. There is a serious problem if sth in tc fails and continuing is not a good idea according to me. If you want to test it just override the values of lh, bu, glbc, gcc & gcccfg after the detection and see what happens.

My thoughts about the future:

Could you please write a sed function to put back the original color in piped emerge output. Especially for the use flags when invoked with -v as I am thinking of adding -v option with the same effect.
We could easily extend the script so that not world and system but any package(s) can be updated.

I was thinking also about emerge -e world/system after tc option.

And last but not least these 2 that would require user input (perhaps this behaviour could be everriden with an option):
- if the new gcc installed fills a new slot ask the user to choose a profile
- give the user the option to interactively define the contents and order of the tc build : ))) (just say what you think about that, I will do it).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2, 3 ... 28, 29, 30  Next
Page 1 of 30

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum