An emerge wrapper for breaking emerges into chunks

Message

slycordinator · Post by **slycordinator** » Wed Jan 25, 2006 5:15 am

hielvc wrote:That idiot, I shall remain un-named, straighten out my port forwarding. But for 2 weeks now at about 10pm my internet goes down. so Im directly hooked the modem. I now have a new ip address so tomorrow mornig Ill have to redo everthing. In other words my links down. IN the mean time I wait and wait for cox service sigh.

FYI

You can use Dynamic DNS to have your (changing) IP associated with a web address name of your choice. And unless you need a bunch of sub-domains or something, it's free.

Some sites that do it:
http://www.dyndns.com/
http://www.no-ip.com/
(a bunch more sites do it)

Those companies just use a database that associates the web address with your REAL IP. You just use a script that tells the company's server your new IP address when it changes.

The updater script for both are in portage (net-dns/noip-updater for no-ip.com and net-dns/ddclient for dyndns) and ddclient might even be included in your router (though it'll be an older version than the one in portage).

So that avoids your worry about the different IP. I think (since I haven't used this service) that it takes a few minutes for the IP change to propogate to their servers when it does change but that's probably acceptable.

hielvc · Post by **hielvc** » Wed Jan 25, 2006 7:09 am

Actually my problem was that it was so stable. when my internel address changed, my son was fixing a freinds computer and got my ip. I forgot to ahange the router mappings. When I did, I didnt really check them out and I had crossed mappings form an old CS server that was no longer used. Anyway I cleaned that up and it worked. Tonight of course the net is working fine, so cox thinks I nuts. but has required about 6 resets of the cable modem about the same for the router which are buried behind another sons speakers and records. So Im back to my old internel ip, edeited apache, and the router tables and that part is good. I havent had a chance to fix emtst.sh though . Well theres tomorrow or the day after.

oh and thanks for the tips.

hielvc · Post by **hielvc** » Thu Jan 26, 2006 7:44 pm

Feld emtst.sh builds the tc now,agian.

hielvc · Post by **hielvc** » Thu Feb 02, 2006 4:22 am

Posted emwrap.sh version 4.0.0

Field thanks for testing

matobsk · Post by **matobsk** » Tue Feb 07, 2006 11:49 am

Emwrap.sh 4.0.0 (Datestamp Feb 1st 2006) still seems to want to build the TC when I do

Code: Select all

./emwrap.sh -W

This doesn't seem to be proper behavior from what I can tell in the docs. I thought it was the same as doing emerge -e world, but skips the TC and 'system' packages.
Maybe I missed some other flag I need to use to do that?

The rest of my system rebuild went great though!

darkphader · Post by **darkphader** » Tue Feb 07, 2006 12:59 pm

matobsk wrote:Emwrap.sh 4.0.0 (Datestamp Feb 1st 2006) still seems to want to build the TC when I do
Code: Select all
./emwrap.sh -W
This doesn't seem to be proper behavior from what I can tell in the docs. I thought it was the same as doing emerge -e world, but skips the TC and 'system' packages.
Maybe I missed some other flag I need to use to do that?

The rest of my system rebuild went great though!

Ditto. But

Code: Select all

emwrap.sh -S

seems to work as expected.

darkphader · Post by **darkphader** » Tue Feb 07, 2006 5:03 pm

The following patch (changes one line - easy to hand edit) to emwrap.sh 4.0.0 makes -W work like I expected it to:

Code: Select all

--- emwrap.sh.orig      2006-02-07 11:56:46.000000000 -0500
+++ emwrap.sh   2006-02-07 11:57:00.000000000 -0500
@@ -436,7 +436,7 @@
 # stops regerateing the build file if your resumeing build
 if [ "$resume" != "yes" ]; then
        # wrld_list already run for system files.
-       cat wrld.lst|tc_filter >sys.lst
+       cat wrld.lst >sys.lst
        # filter out sys files and puts the results into into the build list.
        if  [ "${do_wrld}" ==  "yes" ] ; then
                bclass="world";eargs="e"
@@ -494,4 +494,4 @@


 clean_up
-exit
\ No newline at end of file
+exit

Chris

hielvc · Post by **hielvc** » Wed Feb 08, 2006 5:54 am

Ok fixed.
darkphader your solution is almost correct, at least as I solved it. I removed the tc_fillter from both the" cat wrld.lst >sys.lst" and the from the wrld_lst >build. Anything thats common to both is removed including TC items.. Also added a Tc variable that is trigered off the " -t " so that I had a test condition instead of just a logic of order, that got broken whin I fixed something else, This prevents -W from running when its used to just build the TC, " -Wt ".

dagurasu · Post by **dagurasu** » Mon Feb 13, 2006 12:10 am

@hielvc

It's been awhile. I've been using version 3.0.1 of emwrap.sh since it was released, but decided to give the latest version a go especially to get the improved failed-package features. Anyway, the first thing I tried with 4.0.0 was an "emwrap.sh -setp"(edit: fixed typo). This listed both toolchain packages AND system packages . Should it do that?

There were some early versions where the -p option would list everything but when it was removed it would only emerge what was expected. I don't follow the script anymore since it's changed quite a bit, so I can't tell if this is this case now. So it seems either it wants to emerge more than I expected, or that -p doesn't show what will actually happen.

dagurasu · Post by **dagurasu** » Mon Feb 13, 2006 4:45 am

I found a(nother?) small glitch(not even bug) in emwrap.sh (and probably in most any emerge wrapper). I wanted to hit cntrl-c, do some stuff and then resume. Unfortunately if the cntrl-c comes during an emerge it only kills the emerge, not the script. The script goes on to the next emerge and so on until you get lucky enough to hit cntrl-c during the actual script. By then I'd already killed several emerges(even though I was holding down cntrl-c) and lost them from the build list for my planned resume. My guess is emerge doesn't exit properly and this is why this happens. Anyway, I made a test script with several emerge processes and found the same behavior. The same script run with just sleep processes had no problem exiting as expected. Anyway, I googled a little and found a fix. Put the following:

trap "exit 1" 1 2 3 15

as the first command in the script (or at least before an emerge). I've only tested it on my little test script, but it works like a charm there.

hielvc · Post by **hielvc** » Tue Feb 14, 2006 4:59 am

Thanks dagurasu fixed the prob with -sept running -se aand added "trap", so much for my quickie failed list generation techinic

So if in newer version CRTL-C doesnt stop the script hunt for the

# Main where the functions are called and things get done.
#trap "exit 1" 1 2 3 15
color....

commented out, ' #trap"exit 1" 1 2 3 15 ' at the top of main and uncomment it, remove the # sign. I not planing on doing it, but I might forget. Like I said CRTL-C is a great way to generate a failed list when I test

rommel · Post by **rommel** » Sun Feb 19, 2006 12:18 am

anyone get this when running the script

Code: Select all

 # emwrap -e world
/usr/bin/emwrap: line 464: 1: command not found
world

eandry · Post by **eandry** » Sun Feb 19, 2006 4:28 am

Thought it was time to do an update after 6 months. I check this thread to be sure I have the latest emwrap.sh so I downlaod from hielvc's website.

I check for the options by running as

Code: Select all

emwrap.sh -?

However, I don't get very helpful output:

Code: Select all

                                 Runnig emwrap.sh
         if emwrap.sh is run and it detects a "failed' or "build"
         file then it will offer to build them. You can ignor this by not doing  
         anything and what you wanted todo will start after 10 seconds.

It then returns back to the prompt. Nothing is written in /tmp/emwrap either.

What gives?

hielvc · Post by **hielvc** » Sun Feb 19, 2006 5:58 am

eandry fixed print_help. Its now posted at link and in a few on the front page. rommel are you running emwrap.sh or emwrap ? If its mine, emwrap.sh, what exactly did options did you use?

eandry · Post by **eandry** » Sun Feb 19, 2006 5:29 pm

hielvc,

Thanks for the quick update! Your script seems to be working now as it's finding packages that I need to have either unmasked or rebuilt. I will be attempting a full TC and system rebuild this afternoon.

Just an FYI, page one of this thread mentions the files sys-build.sh and basesys-build.lst both pointing to http://www.fi3nd.net/dads/ which doesn't allow directory browsing capability like yours does and that's where I found sys-build.sh.

Where can I find basesys-build.lst?

Are both of these files required for your emwrap.sh script?

hielvc · Post by **hielvc** » Sun Feb 19, 2006 6:48 pm

eandry Thanks for remainding me of those links to sys-build.sh. No those arent needed. Thats an old project that ended up proveing to me that portage does a very good job of building things in the correct order with the exception of linux-headers and glibc updates. But as I stated on the front page since most of the time they are minor revision updates they dont cause that many probs. Actually dictionary.lst is basesys-build.lst. I forgot to rename it.

As to you rebuilding your system, what I do is

Code: Select all

emwrap.sh -suDt   ==> to make sure gcc isnt makeing a major jump gcc3.3.# > gcc3.4
emwrap.sh -Sb1  or -seb1 ==> rebuilds entire TC once then the rest of the system files.
emwrap.sh -W

This is what I did. I havent really looked at it, but if xorg isnt in your world file then it now gets built when you do a system update. Im going to try to look at that today. It used to be ,for me least, that building the TC and the system once was fine. Then rebuilding the world files once usually worked, but if things were slow or there minor gatchs then I would do a second world rebuild. The world rebuild Im talking about is the emwrap.sh world minus system. Now unless you manually emerge xorg which puts it into your world file, the new virtuals have it built during the system update. I havent checked out what other files are pulled into system with this behavior. So what I did was TC+system once and world once if only 3 months or less had gone by twice if more. This should work still work fine but i need to check.

rommel · Post by **rommel** » Wed Feb 22, 2006 4:29 pm

hielvc, yeah i was running your emwrap.sh... the latest dated on the download page http://68.8.82.201:9002/ ... but something blew up.. i ran it with diff option comments listed in the script and it would just return me to a prompt... then i passed it -e world and after 20 mins started noticing econf errors scrolling by so i killed it and then nothing would emerge again... all failing with segmentation faults on configure

anyway i nuked that amd64 install and added its drive and another like it to a raid array for my users home on my ~x86 install.... i guess i better read this entire thread before i play with it more.

eandry · Post by **eandry** » Wed Feb 22, 2006 11:16 pm

rommel, I was having similar problems. After running doing manual 'emerge -vp world/system' and a python script (I can't remember the name at the moment) I found in one of the other threads which validates your portage configuration (e.g. package.mask, package.use, etc.).

I had blocks and a couple of use flags that were no longer valid. After fixing those problems I was able to eventually do a full rebuild of my system with hielvc's emwrap.sh script.

Thanks hielvc for such a wonderful way of rebuilding your entire system!

hielvc · Post by **hielvc** » Thu Feb 23, 2006 4:34 pm

eandry Im glad you figured out what was wrong. As you said and what I do is run emerge blaa first as emwrap.sh , all these scripts really, strip out usefull info that emerge will give. Here is a usefull little portage config checker script to help manage package.keywords,unmask and mask . They are several but since I havent used them for while I forgotten thier names.

. and gentoolkit has gotten quit good. Oh yes and Porthole-0.5.0 released is a nice tool if you run gnome and there is one for KDE but I cant find it now but Dol-san mentions it in his thread.

\

kohno · Post by **kohno** » Thu Feb 23, 2006 4:44 pm

I tried the emwrap.sh script in my new install,here's what I did:
I unpacked a stage1 tarball, edited /etc/make.conf, then

Code: Select all

# mkdir /etc/portage/profile
# echo "sys-devel/gcc-3.3.6" >> /etc/portage/profile/package.provided
# emerge -uav gcc
# gcc-config 6 (switch to 3.4.4)
# source /etc/profile

Then I run the emwrap.sh script:

Code: Select all

# /root/emwrap.sh -Sb

It works very well. Thank you, hielvc!

hielvc · Post by **hielvc** » Thu Feb 23, 2006 5:18 pm

Oh joy. kohno Im glad it worked but I hope all those nuts , me excluded of course, dont come runnig over here from What exactly happened to stage 1? And bootstrapping? and start yelling and screaming.

eandry · Post by **eandry** » Fri Feb 24, 2006 2:50 am

Incidently, here's portage_syntax_checker.py the portage flag checker that was nameless to me the other day.

Thanks for the reminder about portpeek. I had it downloaded but have never tried it.

Good idea kohno using emwrap.sh during a NEW install. It's been a while since I've done a brand new install of Gentoo since it's been pretty stable since I installed it the first time, so I would have never thought about it.

I have a new AMD64 box that I'm just waiting for the chance to install Gentoo on...so I'm your new fanboy, hielvc!

Treborius · Post by **Treborius** » Fri Feb 24, 2006 5:14 pm

great script, let me say thx first

but if you have a crosscompiler the script fails, cause

# emerge glibc

for example will fail with something like
: the package name glibc is ambious, please blah blah ....

could be solved by adding sys-libs/ to every glibc occurence and sys-devel to every gcc occurence and so on,

for now i just unmerged my crosscompiler

hielvc · Post by **hielvc** » Sat Feb 25, 2006 5:20 am

Treborius this is turning out to be a pain in the dinkus. I take it your useing portage-2.1_pre4-r1 ? Well I am and they have changed what is an acceptble atom. these are and can be used with an " =" , gcc, sys-devel/gcc-3.4.4, gcc-3.4.4.. But this is no longer an acceptabe atom with "=" sys-devel/gcc, but is fine with out the "=". So I put in a merge_tc function that doesnt use "emerge =". Unforgently that breaks the resume or failed build if its a TC item or list.
Joy Joy. Well =package-version works and shouldnt be abiguous.so I shall try changeing the list editing to one common format ,package-version..

carpman · Post by **carpman** » Mon Feb 27, 2006 7:27 pm

Hello having problems with blocked packages:

Code: Select all

/root/updating/emwrap.sh -uDwp



                         Runnig emwrap.sh

 Files made by emwrap.sh are located in /tmp/emwrap. Feel free
 to "rm -r /tmp/emwrap"

1 of 10
sys-libs/timezone-data-2006b

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-libs/timezone-data-2006b [2006a]
2 of 10
sys-apps/man-pages-2.23

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-apps/man-pages-2.23 [2.22]
3 of 10
sys-devel/flex-2.5.33-r1

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-devel/flex-2.5.33-r1 [2.5.4a-r6]
4 of 10
sys-libs/cracklib-2.8.5-r1

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-libs/cracklib-2.8.5-r1 [2.8.5]
5 of 10
app-admin/fam-2.7.0-r4

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] app-admin/fam-2.7.0-r4 [2.7.0-r2]
6 of 10
sys-apps/hal-0.5.5.1-r3

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-apps/hal-0.5.5.1-r3 [0.4.8]
7 of 10
sys-apps/pmount-0.9.6

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild  N    ] sys-apps/pmount-0.9.6
8 of 10
app-portage/ufed-0.40-r1

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] app-portage/ufed-0.40-r1 [0.36]
9 of 10
gnome-base/gnome-volume-manager-1.5.4

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] gnome-base/gnome-volume-manager-1.5.4 [1.2.2]
10 of 10
sys-apps/ivman-0.6.8

These are the packages that I would merge, in order:

Calculating dependencies  ...done!
[ebuild     U ] sys-apps/ivman-0.6.8 [0.5_pre2]
sys-fs/cryptsetup
(is
blocking
sys-fs/cryptsetup-luks-1.0.1-r1)
sys-fs/cryptsetup
(is
blocking
sys-fs/cryptsetup-luks-1.0.1-r1)
perl-core/Test-Simple-0.62
sys-kernel/gentoo-sources-2.6.15-r1
sys-apps/hal-0.5.5.1-r3
sys-apps/pmount-0.9.6
app-editors/nano-1.3.9
gnome-base/gnome-volume-manager-1.5.4
sys-apps/ivman-0.6.8
sys-process/psmisc-21.9
sys-libs/timezone-data-2006b
sys-apps/hal-0.5.5.1-r3
sys-apps/pmount-0.9.6
gnome-base/gnome-volume-manager-1.5.4
sys-apps/ivman-0.6.8

If i try regular emerge update i don't get this:

Code: Select all

 emerge -uDp world

These are the packages that I would merge, in order:

Calculating world dependencies                              ...done!
[ebuild     U ] sys-devel/binutils-2.16.1 [2.15.92.0.2-r7]
[ebuild     U ] sys-devel/gcc-3.4.5 [3.4.3.20050110-r2]
[ebuild     U ] sys-kernel/linux-headers-2.6.11-r2 [2.6.8.1-r2]
[ebuild     U ] sys-libs/timezone-data-2006b [2006a]
[ebuild     U ] sys-libs/glibc-2.3.6-r3 [2.3.5]
[ebuild     U ] sys-apps/man-pages-2.23 [2.22]
[ebuild     U ] sys-devel/flex-2.5.33-r1 [2.5.4a-r6]
[ebuild     U ] sys-libs/cracklib-2.8.5-r1 [2.8.5]
[ebuild     U ] app-admin/fam-2.7.0-r4 [2.7.0-r2]
[ebuild     U ] sys-apps/hal-0.5.5.1-r3 [0.4.8]
[ebuild  N    ] sys-apps/pmount-0.9.6
[ebuild     U ] app-portage/ufed-0.40-r1 [0.36]
[ebuild     U ] gnome-base/gnome-volume-manager-1.5.4 [1.2.2]
[ebuild     U ] sys-apps/ivman-0.6.8 [0.5_pre2]