An emerge wrapper for breaking emerges into chunks

Message

hielvc · Post by **hielvc** » Wed Apr 05, 2006 1:13 am

I had a nice long answer and then instead of SHIFT W I did CNTRL W, sigh. So heres a short answer.

gcc-config -l
gcc-config #
emwrap.sh -set  or -St
gcc-config -l 
gcc-config #
emwrap.sh -se  or  -S
emwrap.sh -set or  -St
emwrap.sh -W
emwrap.sh -W

The above is the safest way. I do this (if im not doing major upgrade of gcc)

Code: Select all

gcc-config -l
gcc-config #
emwrap.sh -set1   builds the TC once
gcc-config -l  and if needed gcc-config #
emwrap.sh -S
emwrap.sh -St1 
emwrap.sh -W
emwrap.sh -W

If Im bootstraping in particularly if your changing the CHOST

Code: Select all

emerge gcc gcc-config && gcc-config -l && gcc-config #
emerge linux-headers glibc binutils-config binutils && binutils-config  <never had to change binutils-config was always right>
emwrap.sh -S  < stop here if youre useing a stage3 >
emwrap.sh -W && emwrap.sh -W

I havent had any probs with doing a single go through on rebuilding the system only once. Rebuilding the world twice is usely nesisary because of the multipule interconnections of the libraries.

S = se and W = we - se. " -r " now stands for resuming a stoped emwrap.sh and starts, where you stoped.

Dol-san Ive been thinking of what you said but besides OOo and kde progs

For OOo I could add a " if grep openoffice sed 's/openoffice//' build ; echo "openoffice" >> build"

EDIT added some clarification

dol-sen · Post by **dol-sen** » Wed Apr 05, 2006 2:23 pm

You could probably set it up to exlcude any generic list provided on the commandline

Code: Select all

emwrap.sh -w --exclude openoffice KDE my-liscened-program

I don't think that should be too difficult.

Anyone else like the idea before hielvc goes thru the trouble?

Tron_5 · Post by **Tron_5** » Mon Apr 17, 2006 1:45 pm

hielvc:
Thanks for the great tool.

dol-sen:
sounds good.
I second that.

carpman · Post by **carpman** » Sun Apr 30, 2006 10:12 pm

Hello, i know the answer is in this thread but there seems to be a few changes i as this an important task i want to be sure.

I have just a done a new install of jackass on my new system, but as jackass is built for i686 only and i have an amd64 (run as x86) i need to change cflags and change use= and rebuild base system before installing rest of system.

I will not be updating any TC only changing that mentioned above.

what is the best way using emwrap?

cheers

hielvc · Post by **hielvc** » Sat May 06, 2006 4:45 am

Ok added " exclude " or " -X " to [url=http://68.111.181.40::9002/] emtst.sh[/url] which back up for those who want to test it. When you run it it will say emwrap.sh 4.0.3
USAGE: emtst.sh world -uDNX gentoo-source openoffice KDE my-liscened-program
OR emtst.sh -e system exclude gentoo-source openoffice KDE my-liscened-program
OR emtst.sh -SX gentoo-sources <== prevents building any gentoo-sources kernel, filters out on the catergory or atom match.

If want to understand its behavior

Code: Select all

emerge world -ep>>build.lst
awk '!/perl/' build.lst>>edit.lst{/code] This prevents building any perl programs [color=red]WHICH IS BAD[/color]
[quote][ebuild  N    ] sys-devel/libperl-5.8.7  
[ebuild  N    ] dev-lang/perl-5.8.7-r3  
[ebuild  N    ] app-admin/perl-cleaner-1.01  
[ebuild  N    ] dev-perl/SGMLSpm-1.03-r5  
[ebuild  N    ] dev-perl/XML-Parser-2.34  USE="-minimal%" 
[ebuild  N    ] dev-perl/Net-Daemon-0.38  USE="-minimal%" 
[ebuild  N    ] perl-core/Storable-2.15  USE="-minimal%" 
[ebuild  N    ] virtual/perl-Storable-2.15  
[ebuild  N    ] dev-perl/PlRPC-0.2018  USE="-minimal%" 
[ebuild  N    ] dev-perl/DBI-1.50  
[ebuild  N    ] dev-perl/DBD-mysql-2.9007  USE="-minimal%" 
[ebuild  N    ] dev-perl/XML-NamespaceSupport-1.09  USE="-minimal%" 
[ebuild  N    ] dev-perl/XML-SAX-0.13  USE="-minimal%" 
[ebuild  N    ] dev-perl/XML-LibXML-Common-0.13  USE="-minimal%" 
[ebuild  N    ] dev-perl/XML-LibXML-1.58-r1  USE="-minimal%" 
[ebuild  N    ] dev-perl/Net-SSLeay-1.25  USE="-minimal%" 
[ebuild  N    ] dev-perl/IO-Socket-SSL-0.97  USE="-minimal%" 
[ebuild  N    ] dev-perl/Compress-Zlib-1.41  USE="-minimal%" 
[ebuild  N    ] dev-perl/IO-Zlib-1.04  USE="-minimal%" 
[ebuild  N    ] virtual/perl-PodParser-1.30  
[ebuild  N    ] dev-perl/Net-IP-1.24  USE="-minimal%" 
[ebuild  N    ] perl-core/Test-Simple-0.62  
[ebuild  N    ] virtual/perl-Test-Simple-0.62  
[ebuild  N    ] dev-perl/Socket6-0.17  USE="-minimal%" 
[ebuild  N    ] dev-perl/IO-Socket-INET6-2.51  USE="-minimal%" 
[ebuild  N    ] virtual/perl-MIME-Base64-3.05  
[ebuild  N    ] virtual/perl-Digest-MD5-2.33  
[ebuild  N    ] perl-core/digest-base-1.13  USE="-minimal%" 
[ebuild  N    ] dev-perl/Digest-SHA1-2.11  
[ebuild  N    ] virtual/perl-digest-base-1.13  
[ebuild  N    ] dev-perl/Digest-HMAC-1.01-r1  USE="-minimal%" 
[ebuild  N    ] dev-perl/Net-DNS-0.53-r1  USE="-minimal%" 
[ebuild  N    ] virtual/perl-Test-Harness-2.48  
[ebuild  N    ] dev-perl/IO-String-1.08  
[ebuild  N    ] dev-perl/Archive-Tar-1.28  
[ebuild  N    ] virtual/perl-libnet-1.19  
[ebuild  N    ] dev-perl/Crypt-SSLeay-0.51  USE="-minimal%" 
[ebuild  N    ] dev-perl/HTML-Tagset-3.10  USE="-minimal%" 
[ebuild  N    ] dev-perl/HTML-Parser-3.48  
[ebuild  N    ] dev-perl/URI-1.35  USE="-minimal%" 
[ebuild  N    ] dev-perl/HTML-Tree-3.19.01  USE="-minimal%" 
[ebuild  N    ] dev-perl/libwww-perl-5.803-r1  USE="-minimal%" 
[ebuild  N    ] perl-core/DB_File-1.814  USE="-minimal%" 
[ebuild  N    ] virtual/perl-DB_File-1.814[/quote]
So be cautious. Things that would be safe to block kde, gnome, xfce and X11 or specific packages like openoffice.gentoo-sources vanilla-sources ....

hielvc · Post by **hielvc** » Sat May 06, 2006 4:54 am

carpman To just build your system files use -se or -S . If you dont give the "-t or -b" flag your TC is not updated or built. When in douth use "-p" and see what happens.

carpman · Post by **carpman** » Sat May 06, 2006 8:35 am

hielvc wrote:carpman To just build your system files use -se or -S . If you dont give the "-t or -b" flag your TC is not updated or built. When in douth use "-p" and see what happens.

thanks for reply, i used "set" then "we" but will make note of your suggestions.

hielvc · Post by **hielvc** » Sat May 06, 2006 3:04 pm

Ok its up.

TIME Sat May 6 09:02 PDT 2006

slycordinator · Post by **slycordinator** » Sat May 06, 2006 9:34 pm

I just downloaded the newest version of emwrap.sh (4.0.2) from your site and found an error in package version parsing.

Code: Select all

./emwrap.sh -sept


                                 Runnig emwrap.sh 4.0.2
         if emwrap.sh is run and it detects a "failed' or "build"
         file then it will offer to build them. You can ignor this by not doing 
         anything and what you wanted todo will start after 10 seconds.
 these remain to build
sys-libs/glibc-2.4-r1
sys-devel/gcc-config-1.3.13-r2  0 kB
sys-devel/binutils-config-1.8-r7  0 kB
sys-devel/binutils-2.16.1-r2
sys-devel/gcc-3.4.6-r1
sys-libs/glibc-2.4-r1
sys-devel/binutils-2.16.1-r2
sys-devel/gcc-3.4.6-r1
sys-libs/libstdc++-v3-3.3.6
        You can now enter " r " to resume building. If you dont enter anything
        then the old build file will be deleted.and emwrap.sh system -ep will continue.
        If you want to stop and scratch your head do a CRTL-C
         Input was empty I will resumeinue with emwrap.sh system -ep

           ToolChain updates found

                linux-headers
======================================================

sys-kernel/linux-headers-2.6.11-r5  0 kB
sys-libs/glibc-2.4-r1
sys-devel/gcc-config-1.3.13-r2  0 kB
sys-devel/binutils-config-1.8-r7  0 kB
sys-devel/binutils-2.16.1-r2
sys-devel/gcc-3.4.6-r1
sys-libs/glibc-2.4-r1
sys-devel/binutils-2.16.1-r2
sys-devel/gcc-3.4.6-r1
sys-libs/libstdc++-v3-3.3.6

1 of 10 build
These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] sys-kernel/linux-headers-2.6.11-r5  0 kB

Total size of downloads: 0 kB

2 of 10 build
These are the packages that would be merged, in order:

Calculating dependencies
emerge: there are no ebuilds to satisfy "=0".

0 failed to build. Stoping script.

Seems to be parsing the "$size kb" as being part of the package. And then when it strips out actual packages it sees the "$size" as a package.

edit:
And when I did "emwrap.sh -sep" peridically it would say

Code: Select all

emerge: there are no ebuilds to satisfy "=0".

followed by

Code: Select all

emerge: there are no ebuilds to satisfy "=kb".

Also happens with emtsh.sh (version 4.0.3)

hielvc · Post by **hielvc** » Sat May 06, 2006 10:13 pm

slycordinator try emtst.sh. It has 2 changes. One is the "exclud" option but the other is a change in filtering. I was useing a"cut" with "sed". but it was to specific. It couldnt handel VIEDO_CARDS if you were useing modular X. So I went back to " cut -f1 -d " " " . Could you try it ? In side a week it will be posted as emwrap.sh 4.0.3

Kasumi_Ninja · Post by **Kasumi_Ninja** » Sat May 06, 2006 10:28 pm

hielvc wrote:No use my emwrap.sh or macquires emwrap. Any follow up post should be on my thread so that this thread isnt hyjacked.

Ok thanks! Will try out your script

Edit: If I read the above post correctly I should also use emtst.sh?

hielvc · Post by **hielvc** » Sat May 06, 2006 10:59 pm

HXC Yeah emtst.sh works for me, so now its time to see sly can break it

Unless something happens I'll move it to emwrap.sh next weekend, as it is when you run it it says its emwrap.sh 4.0.3

slycordinator · Post by **slycordinator** » Sun May 07, 2006 3:09 am

hielvc wrote:HXC Yeah emtst.sh works for me, so now its time to see sly can break it Unless something happens I'll move it to emwrap.sh next weekend, as it is when you run it it says its emwrap.sh 4.0.3

slycordinator wrote:Also happens with emtsh.sh (version 4.0.3)

I'm guessing it's something with portage.

I'm running portage-2.1_pre10-r3 if that helps.

mamoo · Post by **mamoo** » Sun May 07, 2006 3:04 pm

hielvc wrote:I had a nice long answer and then instead of SHIFT W I did CNTRL W, sigh. So heres a short answer.
Code: Select all
gcc-config -l
gcc-config #
emwrap.sh -set  or -St
gcc-config -l 
gcc-config #
emwrap.sh -se  or  -S
emwrap.sh -set or  -St
emwrap.sh -W
emwrap.sh -W 
The above is the safest way. I do this (if im not doing major upgrade of gcc)
Code: Select all
gcc-config -l
gcc-config #
emwrap.sh -set1   builds the TC once
gcc-config -l  and if needed gcc-config #
emwrap.sh -S
emwrap.sh -St1 
emwrap.sh -W
emwrap.sh -W 
If Im bootstraping in particularly if your changing the CHOST
Code: Select all
emerge gcc gcc-config && gcc-config -l && gcc-config #
emerge linux-headers glibc binutils-config binutils && binutils-config  <never had to change binutils-config was always right>
emwrap.sh -S  < stop here if youre useing a stage3 >
emwrap.sh -W && emwrap.sh -W 
I havent had any probs with doing a single go through on rebuilding the system only once. Rebuilding the world twice is usely nesisary because of the multipule interconnections of the libraries.

S = se and W = we - se. " -r " now stands for resuming a stoped emwrap.sh and starts, where you stoped.

Dol-san Ive been thinking of what you said but besides OOo and kde progs For OOo I could add a " if grep openoffice sed 's/openoffice//' build ; echo "openoffice" >> build"

EDIT added some clarification

First of all I'd like to express my thanks for this wonderfull script. It has been a real time-saver for me.

One questions is still bugging me however and it is related to your post quoted above.

Currently I'm doing a emtst.sh (yes the new one) enabled build of the toolchain, which is a first step in rebuilding my system from a clean stage3. I've been looking into the steps as suggested by you, based on the safest approach. This would be:

Code: Select all

gcc-config -l
gcc-config #
emwrap.sh -set  or -St
gcc-config -l 
gcc-config #
emwrap.sh -se  or  -S
emwrap.sh -set or  -St
emwrap.sh -W
emwrap.sh -W

However, after giving this some considerable thought, I wonder if the following approach isn't more clean.

Code: Select all

gcc-config -l
gcc-config #
emwrap.sh -set  or -St
gcc-config -l 
gcc-config #
emwrap.sh -se  or  -S
emwrap.sh -se  or  -S
emwrap.sh -set or  -St
emwrap.sh -W
emwrap.sh -W

My line of thinking is that in your suggested approach the system files are only build ones. However on the first page of this thread it is mentioned that some system packages rely on other system packages, hence building the system wice might be necessary for some of them.

I might be missing something here but I'm interested in your comments.

Also, I would appreciate if you could elaborate on the other two approaches aswell. Especially why you can get away with building the toolchain only once (which is a real time-save!)

Thanks

hielvc · Post by **hielvc** » Sun May 07, 2006 4:23 pm

sly Ive seen simular probs but have never really solved them. Ive 2 gentoo boxs right now and neither show the prob joy. I to am running

Code: Select all

emerge portage coreutils bash -p|grep "/"|cut -c17-65
sys-apps/portage-2.1_pre10-r3  
sys-apps/coreutils-5.94-r1  
app-shells/bash-3.1_p17

AND FOR BOX 2 SAME COMMAND
sys-apps/portage-2.1_pre10-r2  
sys-apps/coreutils-5.94-r1  
app-shells/bash-3.1_p16

If you could edit your emtst.sh and at line 439 in function wrld_lst()

emerge $bclass -${eargs}pq >wrld.lst

add a "q" to the end of -${eargs}p .
This will hopefully fix yoru prob.

Also you want to add a "q" to line 328 in function build_count()

emerge -${ftch} -${prtnd} --oneshot --nodeps -q =${z}

.This shouldnt fix the probem, but it changes summury output. to

Runnig emwrap.sh 4.0.3
if emwrap.sh is run and it detects a "failed' or "build"
file then it will offer to build them. You can ignor this by not doing
anything and what you wanted todo will start after 10 seconds.

13 builds to emerge
1 of 13 build[ebuild U ] sys-apps/man-pages-2.30 [2.28]

2 of 13 build[ebuild U ] sys-apps/baselayout-1.12.0_pre19-r2 [1.12.0_pre18-r1]

3 of 13 build[ebuild U ] x11-base/xorg-x11-6.8.2-r7 [6.8.2-r6]

4 of 13 build[ebuild U ] media-libs/tiff-3.7.4 [3.7.3]

5 of 13 build[ebuild U ] dev-db/mysql-4.1.19 [4.1.14-r1]

6 of 13 build[ebuild U ] dev-util/strace-4.5.12 [4.5.11]

7 of 13 build[ebuild U ] dev-lang/nasm-0.98.39-r3 [0.98.39-r1]

8 of 13 build[ebuild U ] app-misc/mc-4.6.1-r1 [4.6.1]

9 of 13 build[ebuild U ] dev-lang/php-5.1.4 [5.1.2-r1]

10 of 13 build[ebuild U ] kde-base/kdepim-3.5.2-r1 [3.5.2]

11 of 13 build[ebuild U ] media-sound/cdparanoia-3.9.8-r3 [3.9.8-r2]

12 of 13 build[ebuild U ] kde-base/kdemultimedia-3.5.2-r2 [3.5.2-r1]

13 of 13 build[ebuild U ] app-antivirus/clamav-0.88.2 [0.88.1

hielvc · Post by **hielvc** » Sun May 07, 2006 5:25 pm

mamoo 1. Your thinking I'm like Bob_P but I kan't type nor spill nor am I on speed to type near as fast as speedy Gonzales. So no dissertations

There are basiclly 3 schools of thought on updateing the TC :

A. emerge whatever -uDN
B. emerge emerge emerge emerge (keeps the electrons in your box from getting fat)
C. and emwrap.sh

Ok A, regular portage, works fine 98-99% of the time. Its gets dangerous though when updates are few and far between. gcc and glibc upgrades that are several rev levels different or major version bumps. gcc-3.3.4 to gcc-3.4.6 can break systems.

Then there is B, robmoss electron dieting plan. If you read the first page it takes 3 emerge systems, the 3rd being an emerge world, to build the TC and the system progs with that TC.

Bob_P swears by emerge emerge emerge emerge. It works for him and Rockhopper & Jackass show his thorughness. Then again part of it might be to cut down the support post. To update your Rockhopper TC emerge emerge ... if you dont dont post. I didnt talk about useing emwrap to bootstrap till some one posted that they had done so with success. Why? I cant and am not interested in running a bootstraping support thread

though it would probably improve my typing.

The same with building the TC once. I did it on the command line and got tired of it. So I put the "1" option into emwrap.sh. My current system started as a stage3-i386 => to a 586 doing the TC twice, emwrap.sh -se once and then emergeing world to my taste. I then, a year later, changed CHOST to i686 and did emwrap.sh -seb1 (TC once + system) and then rebuild the world progs twice. Its what I'm running now. Ive delibertly bootstraped stage3-386 to 686 to see if it works and havent had any probs. I spent a lot of time once trying to come up with the correct build order for the system files so that it would need to be built once like a stage1. Dang if my order didnt end up matching "emerge system -e" minus the TC. So I gave up trying to re-invent wheel.Back to the point building the TC and system once works for me and if it doesnt Ill just build it a second time. Thats my experince, and I still relate back to my P166 days thinking that there has to be a faster way. Hey all this compliling cuts down my seti results.

But even if you build the TC and system files twice and then built the world files, emwrap.sh -W, twice thats just 2 "emerge worlds -e". So its still a lot quicker.

Hope this helps

EDIT I was in a hurry this morning and when I got home I reread the mess I posted this morning. Hopefully this is more readable.

hielvc · Post by **hielvc** » Mon May 08, 2006 2:43 am

sly I think I found the problem. Ive added "q" to all the list generation emerge commands. I also added "-q" to flags so that you can pass it to the scriipt. It changes output format to a more concise form Changed the version to 4.0.4 , Im gettin bump happy.

Randy R · Post by **Randy R** » Mon May 08, 2006 10:00 pm

I'm having a couple of problems with the script.

1. When i do an emwrap.sh -set, or any other tool chain update, i recieve an error indicating that the script can't find gcc-3.3.6. The current gcc is 3.4.5.

2. When I try to do emwrap.sh -W I get a screen indicating that -W is not one of the options.

This is anexcerpt of my curent emwrap.sh.

Code: Select all

#!/bin/bash
#
# EMWRAP
#
# emwrap - A wrapper for 'emerge'.  Type:  $ emwrap --help
#
# 23-Aug-2005: * Extracted the 'emerge' from the SYSTEM+WORLD and the TC
# Version 4.2    updates---to eliminate a lot of common code.  Also, fixed a
# WBMII          bug in the selection of the GCC version with GCC-CONFIG.
#                Thanks "sfp-a7x" (from the Gentoo Forums)!
#                (There were a *lot* of modifications made for this version!)

Do I not have the curent script?

hielvc · Post by **hielvc** » Mon May 08, 2006 11:31 pm

Randy R your useing maguire's emwrap script. Its probably an unfortunate choice of names

as it does cause confussion. There are people though who use it and still cruise by who might be able help. The quickest thing to do is to read the help section for his commands. The commands and recommendations you see me posting are for "emwrap.sh" the bottom link on the front page.

slycordinator · Post by **slycordinator** » Mon May 08, 2006 11:59 pm

hielvc wrote:sly I think I found the problem. Ive added "q" to all the list generation emerge commands. I also added "-q" to flags so that you can pass it to the scriipt. It changes output format to a more concise form Changed the version to 4.0.4 , Im gettin bump happy.

Fixed. Though it didn't quite work the first time I ran it since it was confused by the failed files. Upon subsequent runs it went fine.

Tested it w/ options of "-sept", "-sep", and "-Wp" and saw no problems.

hielvc · Post by **hielvc** » Tue May 09, 2006 12:22 am

slycordinator Good ! Did you try useing the "-q" with pretend to see how it changes the output format? I'm starting to like the more compact format.

hielvc · Post by **hielvc** » Tue May 09, 2006 2:07 am

I'm staring to feel like a post monkey. Found a minor glitch, exclude didnt work with " -W", world - system. Fixed also minor cleanup and additional comments added.

TIME STAMP 19K May 8 19:02 emtst.sh*

hielvc · Post by **hielvc** » Tue May 09, 2006 4:14 am

darkphader heres a revdep-rebuild wrapper for you

Code: Select all

#!/bin/bash
# By hielvc, A wrapper for revdep-rebuild that uses a "for loop" to feed the emerge
# stage. Suggested by darkphader in the emwrap.sh thread, 
# http://forums.gentoo.org/viewtopic-t-282474-start-400.html  pg 17

########### varible  setting  #######################
prtnd=""

# clean revdep-rebuild files. Keeps "rm" from complaining if nothing there
if [ -e /root/.revdep-rebuild* ]; then
	rm /root/.revdep-rebuild*
fi
function info(){
	cat <<END
	Usage "p" to run in pretend mode and "d" to do the deed.
	Runs revdep-rebuild -i -p to generate build list.
END
}
echo "   You have to be root to run this for real, with the "d" flag."
# Getargs
if [ "$#" == "0" ]; then
	# calling info
	info	
	exit
elif [ "$1" == "p" ];then
	prtnd="-p"
elif [ "$1" ==  "d" ]; then
	:  # SHORT HAND FOR TRUE
else 
	echo "   Wrong USAGE. \"$1\" is not a \"p\" or a \"d\" ya dummy ;^)"
	echo
	info
	exit
fi
	
# Run revdep-rebuild
revdep-rebuild -i -p

# Test and see if /root/.revdep-rebuild.4_ebuilds exist and has something in it.
if [[ -e /root/.revdep-rebuild.4_ebuilds && ! -s /root/.revdep-rebuild.4_ebuilds ]]; then
	for i in $(< /root/revdep-rebuild.4_ebuilds); do
		emerge -O1 $prtnd =$i || echo "$i">> revdep-failed
		grep -v $i /root/.revdep-rebuild.4_ebuilds>>tmp; mv tmp /root/.revdep-rebuild.4_ebuilds
	done
fi
if [ -e revdep-failed ]; then ehoc "These failed to build";cat revdep-failed ;fi

slycordinator · Post by **slycordinator** » Tue May 09, 2006 5:02 am

hielvc wrote:slycordinator Good ! Did you try useing the "-q" with pretend to see how it changes the output format? I'm starting to like the more compact format.

I like. Very succinct (sp?).

Randy R · Post by **Randy R** » Fri May 12, 2006 3:01 am

hielvc,

Thanks. I got your script and it works flawlessly.