Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
An emerge wrapper for breaking emerges into chunks
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5 ... 28, 29, 30  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Sun Mar 20, 2005 5:06 am    Post subject: Reply with quote

One of the nice things about MindEraser's script is that the "-p or --pretend" works with the other switchs. This allows you to see whats going to be done. To understand this script better, play with "emerge world -ep|wc " and then emerge system -ep|wc" the same with -uD. Redirect them into files and look at them.

evoweiss technically portage and gcc-config arent part of the basic toolchain but they are part of gentoos toolchain and toolchain config. I started this with the idea that most people who would be interested are staying on top keeping their system pretty well up to date already. and so including portage and gcc-config werent that important. When I check out what emerge wants do for me I usually will emerge portage by its self to get it out of the list and that is why MindEraser included it and it is part a gentoo system toolchain :lol:. I guess you can put it this way " gentoo without portage would be LFS, LInux From Scratch." and thats not a put down of LFS. At the time when Minderaser added gcc-config to tcupdate.sh it seemed to depend on what phase the moon was in as whehter gcc-config would change your profile for you or not. At least once I was half way through a long emerge before I thought to check and it hadnt switched of course. 8O
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
reteo
n00b
n00b


Joined: 26 Dec 2004
Posts: 62
Location: Phoenix, Arizona

PostPosted: Sun Mar 20, 2005 11:12 pm    Post subject: Reply with quote

MindEraser wrote:
:D

TODO:
...
Can't think of any way of overriding what emerge puts on xterm titles.


Doesn't "FEATURES=notitles" work?
_________________
Lampros Liontos (aka. Reteo Varala)

The Penguin Producer - Tips, tricks, and techniques for producing various forms of media with Linux tools.
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Mar 21, 2005 9:16 pm    Post subject: Reply with quote

Maedhros wrote:
Moved from Portage & Programming.

Unsupported Software!?! :!: :?: :!:

Congrats, Gents. you are now Officially members of the Lunatic Fringe. :lol:
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
kimchi_sg
Advocate
Advocate


Joined: 26 Nov 2004
Posts: 2955

PostPosted: Tue Mar 22, 2005 8:26 am    Post subject: Reply with quote

Bob P wrote:
Congrats, Gents. you are now Officially members of the Lunatic Fringe. :lol:

Come on... which sane user would want to touch (no, remerge) his toolchain with a 5-foot long wooden pole (this script) ? :P
Back to top
View user's profile Send private message
mlybarger
Guru
Guru


Joined: 04 Sep 2002
Posts: 473

PostPosted: Thu Mar 24, 2005 5:21 pm    Post subject: Reply with quote

could someone please give a 1000ft. overview of what these scripts are intended to do? it's a tad challenging to pick it out of the 4 pages of posts here and other related threads.

my impression from reading these threads is that emerge has some major issues with upgrading "system" type stuff, and sometimes it needs to emerge them twice?

basically my problem is that i'm getting tons of unresolved symbols when trying to start xorg. i've tried various versions of xorg in portage, each build and install ok, but none will start. this just started recently, perhaps after doing an "emerge -uD world". i've been considering just backing up /etc and /home and other areas, then re-installing, but would like to avoid that if possible just to keep as much stuff in tact. this seems like perhaps one solution to rebuild/clean the system w/o impacting other stuff.

any comments would be most appreciated.

Thanks~
~mark
Back to top
View user's profile Send private message
reteo
n00b
n00b


Joined: 26 Dec 2004
Posts: 62
Location: Phoenix, Arizona

PostPosted: Thu Mar 24, 2005 7:53 pm    Post subject: Reply with quote

It would work something like this:

Using tools: old-gcc, old-glibc, old-binutils, old-linux26-headers:

build new-linux26-heders
build new-gcc
build new-glibc
build new-binutils

New tools are now made, but they've been built using the logic of the old tools; this could pose problems if enough changes were introduced to confuse the optimizations, or especially if there is a bug in the old version of one of those tools. So we repeat the process so the new tools are remade, but using themselves rather than their previous versions.

Using tools: new-gcc, new-glibc, new-binutils, new-linux26-headers:

build new-linux26-heders
build new-gcc
build new-glibc
build new-binutils

They are now rebuilt, but using themselves as a base, thus we have a system toolchain in parity with itself. Since they all depend on each other to work properly, it's important all are remade when one of them is updated... and before they are used to build anything else.

I hope I've gotten all the necessary cogent points elaborated.
_________________
Lampros Liontos (aka. Reteo Varala)

The Penguin Producer - Tips, tricks, and techniques for producing various forms of media with Linux tools.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Fri Mar 25, 2005 3:18 am    Post subject: Reply with quote

the basic progs needed to build a TC and the order built : linux-headers glibc binutils gcc. For an explanation of these particularly linux-headers and glibc and why they should be installed first "emerge app-doc/linuxfromscratch-html " and read chapters 5 and 6.

Currently when portage builds the TC its done willy nilly. I just did an "emerge system -ep>>sys.lst and here is the order listed.
Quote:

10 [ebuild N ] sys-devel/binutils-config-1.8-r1
11 [ebuild N ] sys-devel/binutils-2.15.92.0.2-r7
......
45 [ebuild N ] sys-devel/gcc-config-1.3.10-r1
46[ebuild N ] sys-devel/gcc-3.4.3.20050110-r1
47[ebuild N ] sys-libs/libstdc++-v3-3.3.4
48[ebuild N ] sys-kernel/linux26-headers-2.6.8.1-r2
49[ebuild N ] sys-apps/sysvinit-2.86
50[ebuild N ] sys-apps/baselayout-1.11.10-r4
51[ebuild N ] sys-libs/glibc-2.3.4.20050125-r1


If you look in your /var/log/portage/linux-headers you'll see that at the end it even says that its recomended that you re-emerge glibc. Now is this build order super critical no its not. As I said in Bob_Ps Stage 1/3 Installation with GCC 343 on 2005.0-rc5 = Problems
hielvc wrote:
Bob tcupdate.sh uses the order " glibc binutils gcc && glibc binutils gcc". Both work, hell going by the forums 98 to 99% of the time "emerge -uD/or -e " work well other wise there would be tons of post complaining about " me broke after emerge -uD .. " and traceing it back to a broken TC. It happens, but a hugh majority of post are about newbei errors, this package is blocked or broken, I optimized the shit out my CFLAGS so that when I jump out the window with it we go faster but then it breaks :twisted: , and portage circular dependenceies that crop up and go away, and how to use emerge. If you go by the number of registered users 80747 and figure just a quater of them use gentoo full time and update once or twice a year, then a very small percentage of people have problems. Its mainly us nutters jumping out the window :lol:


The advantage of useing tcupdate.sh is that you have control of when and if you want to update your TC, preventing one set of pesky probs.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
eandry
n00b
n00b


Joined: 15 Nov 2004
Posts: 50

PostPosted: Mon Mar 28, 2005 5:00 am    Post subject: Reply with quote

Excellent work on this script. I use it everytime there's a change in toolchain components. And for the most part it works great!

After rebuilding a new TC, I rebuild 'system' and then 'world' packages to take advantage of any new TC features.

However when I build 'world', packages in 'system' are also rebuilt. Is there a way to keep this from happening?

Here's the tcupdate.sh commands I used in the order described above:

Code:
tcupdate.sh -et


Code:
tcupdate.sh -ew system


Code:
tcupdate.sh -ew world --exclude='system'


On the third use of tcupdate.sh, it just rebuilds the entire system including system and world packages. Unless I specify '-e' isn't there a chance that I won't get a clean rebuild of the entire system? From the help documentation looks like exclude is experimental. Is this still the case? I'm using version 2.2.0.
Back to top
View user's profile Send private message
mope
Apprentice
Apprentice


Joined: 23 Feb 2003
Posts: 206

PostPosted: Mon Mar 28, 2005 1:46 pm    Post subject: Reply with quote

When I do this:

Quote:

tcupdate.sh -ew --exclude='system' world


it looks like it excludes the system files.

Am I looking at it correctly?


Also, there doesn't appear to be linux26-headers anymore.
Is it time to update the scripts again?

(and which one do we use anyway? are they both essentially the same?)
Back to top
View user's profile Send private message
agent_jdh
Veteran
Veteran


Joined: 08 Aug 2002
Posts: 1783
Location: Scotland

PostPosted: Mon Mar 28, 2005 2:20 pm    Post subject: Reply with quote

mope wrote:
When I do this:

Quote:

tcupdate.sh -ew --exclude='system' world


it looks like it excludes the system files.

Am I looking at it correctly?


Also, there doesn't appear to be linux26-headers anymore.
Is it time to update the scripts again?

(and which one do we use anyway? are they both essentially the same?)


w.r.t. the 2nd point, it looks like with the 2005.0 release, linux26-headers has been consumed by linux-headers, which now uses 2.6 kernel headers by default; also development-sources is no more, vanilla-sources is now a 2.6 kernel. This is on x86 at least.
_________________
Jingle Jangle Jewellery
Back to top
View user's profile Send private message
mope
Apprentice
Apprentice


Joined: 23 Feb 2003
Posts: 206

PostPosted: Mon Mar 28, 2005 2:29 pm    Post subject: Reply with quote

agent_jdh wrote:
also development-sources is no more, vanilla-sources is now a 2.6 kernel. This is on x86 at least.


sweet!

only problem is that the latest gentoo-sources hangs my computer when I reboot.
don't know what that's about, but I was waiting for dev-sources to figure it out. bummer, now it looks like I just have to refrain from rebooting! :)
Back to top
View user's profile Send private message
wiktorw
n00b
n00b


Joined: 14 Dec 2004
Posts: 39
Location: Lodz, Poland

PostPosted: Mon Mar 28, 2005 10:24 pm    Post subject: First try - and success! (on amd64/2005.0 profile) Reply with quote

Hello,

Just wanted to say that I used the "tcupdate.sh" script with great success.
My intention was to use it as the preliminary step for a new Gentoo installation.
The method I used:

Code:
# tcupdate.sh -e -t -s

Of special interest may be the length of this process.
<irony mode="sweet"> Whew, it took only short 10 hours... </irony>
But at least I'm confident that the whole toolchain is correct.
Which bothered me sometime ago, actually.

Now it's time to follow the Stage 1/3 Installation - Gentoo 2005.0 & GCC 3.4.3 by Bob P :)
Even though I've done most of the steps myself already :D
(like: partitioning, untarring stage3 & portage snapshot, setting USE flags, fstab, etc.)
And a new 2.6.11-nitro2 awaits too :) (/me also wonders when 2.6.12 nitro comes out...)

A very fine piece of work! Thanks!

Machine:
939-pin Athlon64 3000+ on ASUS A8V Deluxe with 1GB dual-channel RAM and 120 GB SATA HDD.

Code:
# emerge info
Portage 2.0.51.19 (default-linux/amd64/2005.0, gcc-3.4.3, glibc-2.3.4.20050125-r1, 2.6.11-nitro1 x86_64)
=================================================================
System uname: 2.6.11-nitro1 x86_64 AMD Athlon(tm) 64 Processor 3000+
Gentoo Base System version 1.4.16
Python:              dev-lang/python-2.3.4-r1 [2.3.4 (#1, Mar  4 2005, 20:56:19)]
dev-lang/python:     2.3.4-r1
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.7.9-r1, 1.8.5-r3, 1.6.3, 1.9.4, 1.5
sys-devel/binutils:  2.15.92.0.2-r1
sys-devel/libtool:   1.5.10-r4
virtual/os-headers:  2.6.8.1-r4
ACCEPT_KEYWORDS="amd64"
AUTOCLEAN="yes"
CFLAGS="-O3 -march=k8 -ffast-math -fomit-frame-pointer -mfpmath=sse -funit-at-a-time -fpeel-loops -ftracer -funswitch-loops"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown/usr/kde/3/share/config /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O3 -march=k8 -ffast-math -fomit-frame-pointer -mfpmath=sse -funit-at-a-time -fpeel-loops -ftracer -funswitch-loops"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs autoconfig ccache distlocks sandbox"
GENTOO_MIRRORS="http://gentoo.prz.rzeszow.pl http://src.gentoo.pl http://gentoo.netnitco.net http://gentoo.inode.at"
LANG="pl_PL"
LC_ALL="pl_PL"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="amd64 acl acpi alsa bash-completion berkdb bitmap-fonts cdr crypt dio doc dvd dvdr font-server fortran gdbm gif gpm hal inifile ipv6 java jp2 jpeg lzw lzw-tiff mime mozilla mp3 ncurses nls nptl nptlonly opengl oss pam pcre perl pic png python readline ssl tcpd tiff truetype truetype-fonts type1-fonts unicode usb userlocales xml2 xpm xrandr xv zlib linguas_pl"
Unset:  ASFLAGS, CBUILD, CTARGET, LDFLAGS

_________________
Change is the only constant in the universe.
Heraclitus
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Mar 28, 2005 11:16 pm    Post subject: Reply with quote

you know, when i was writing the Stage 1/3 Guide for 2005.0, i thought very seriously about just turning it into a Stage 3 install, then after rebooting, having the user do something like this:

Code:
# tcupdate.sh -et --keeplog
# cp make.conf.343 make.conf
# tcupdate.sh -etw --keeplog


the only downside to doing it this way is that you trade-off by adding some compile time for the sake of command line convenience. you build some world packages twice this way, and you don't do that in the Stage 1/3 guide.

hiel, is this similar to what you did on your new install?
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Mar 29, 2005 5:03 am    Post subject: Reply with quote

Yes But I built the TC with a oneliner.

I just updated my script and you could use the oneliner or 2 liner for changing the CFLAGS. WIth mine you would then "emwrap.sh -se" to do a "emerge system -e" minus the TC. In other words the TC wouldnt get rebuilt but everything else would be updated saveing time. tcupdate.sh Im sure does the same but Id have to look at it again.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
eandry
n00b
n00b


Joined: 15 Nov 2004
Posts: 50

PostPosted: Tue Mar 29, 2005 5:13 am    Post subject: Reply with quote

mope wrote:
When I do this:

Quote:

tcupdate.sh -ew --exclude='system' world


it looks like it excludes the system files.

Am I looking at it correctly?


No you're not. :)

For example xorg-x11.

Code:
# tcupdate.sh -p -ew --exclude='system' world | grep xorg
[ebuild   R   ] x11-base/xorg-x11-6.8.2-r1


Code:
# tcupdate.sh -p -ew system | grep xorg
[ebuild   R   ] x11-base/xorg-x11-6.8.2-r1


It wants to be re-emerged if doing world or just system.

But I think though this is considered "normal" because when you use '-e' the package is removed from the repository. So of course it has to replace the package again because it's no longer there.

Am I right about this...script authors? :D
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Mar 29, 2005 4:41 pm    Post subject: Reply with quote

Updated my script "emwrap.sh" changed command line args to more closely follow portage short args. the big difference sicne last bight is added "-d and -r" options. "emwrap.sh -d" does a "emerge system -e" and no TC. After running it if you run "emwrap.sh -r" it builds the wrld list minus the system files from the "emwrap.sh -d" command. That saves you from re-emergeing 100+ files :wink:
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Mar 29, 2005 4:52 pm    Post subject: Reply with quote

If you were really quick and got the new emwrap.sh before the time of this post then the -r wont work unless you add "r " to this line, 210, in the script
Quote:
while getopts ":bdehoprstz" OPT; do

_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
eandry
n00b
n00b


Joined: 15 Nov 2004
Posts: 50

PostPosted: Tue Mar 29, 2005 5:10 pm    Post subject: Reply with quote

hielvc wrote:
If you were really quick and got the new emwrap.sh before the time of this...


Would have had I known you posted a link to the script somewhere! :)

Did I miss a new thread for "emwrap.sh"? I didn't see an update at the beginning of the thread for this new script.

Is it used in conjunction or replace tcupdate.sh?
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Mar 29, 2005 5:27 pm    Post subject: Reply with quote

MIndEraser wrote tcupdate.sh and thats the link. I finally updated my script which Im calling emwrap.sh now, and it is the one that you have to cut and paste from the listing.
Ive got to get on the road but just though of more error. in lines 155 and 156 change the
Quote:

155 tc_filter>>wrld.tst from to tc_filter>wrld.tst
156 tc_filter>>sys.tst from to tc_filter>sys.tst

Change " >> " to ">" other wise the files grow and grow or just copy it again.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Sat Apr 02, 2005 9:27 pm    Post subject: Reply with quote

have either of you guys considered adding a --fetchonly option to the script?

personally, i'd like to cut down on the total rebuild time by fetching concurrently with rebuilding (running tasks in separate consoles), instead of fetching every ebuild, pausing during the download, and sequentially building it. concurrent fetching instead of sequential fetching could dramatically cut down on the total build time by essentially removing the sum of the download times from the total build time.

even though this would help peeps like me with broadband connections, the fetching concept would really help dial-up users more than anyone else.

just a thought.
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
zendal
n00b
n00b


Joined: 30 Nov 2002
Posts: 23
Location: Olympia, WA

PostPosted: Fri Apr 08, 2005 4:42 am    Post subject: --fetchonly in the script Reply with quote

What is to stop them from just running
Code:
emerge -f system
emerge -f world

to get the packages then to run the script.

It would download all the files and put them in /usr/portage/distfiles
Back to top
View user's profile Send private message
powerman
n00b
n00b


Joined: 21 Dec 2003
Posts: 19
Location: Kharkov

PostPosted: Sat Apr 09, 2005 9:37 pm    Post subject: Reply with quote

Information in this thread help me a lot, thanks!!

You try to solve two different problems: compile toolchain in right order and minimize compilation time. I just don't understand why not use internal portage feature (buildpkg) to speedup compilation without complex scripts? Please explain why this example is wrong or not-so-fast as your scripts:

Code:
rm -r $(portageq pkgdir)/All/    # just a dirty example of cleanup
emerge      linux-headers glibc binutils gcc-config gcc
gcc-config YOUR_SELECTION && source /etc/profile    # ONLY if major gcc upgrade
emerge -b   glibc binutils gcc portage
emerge -bke system
emerge -bke world


For me this is sort of resume for all 4 pages in this thread (excluding some crazy ideas like emerging linux-headers twice).
_________________
WBR, Alex
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 09, 2005 10:13 pm    Post subject: Reply with quote

that's exactly what the script does, except you're building packages on the second pass. throw "buildpkg" in to FEATURES in make.conf and it does the same thing.

but how does that speed up compile times?

edit: this would give the same result as yours:
Code:
# tcupdate --nodeps -etw system && tcupdate -ew --exclude=system world


and in my .bashrc i alias that to "overhaul". :wink:
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
powerman
n00b
n00b


Joined: 21 Dec 2003
Posts: 19
Location: Kharkov

PostPosted: Sat Apr 09, 2005 10:23 pm    Post subject: Reply with quote

dirtyepic wrote:
but how does that speed up compile times?

Speedup mean don't recompile packages when this isn't needed. My `emerge -bke system` will not recompile glibc, binutils and gcc - it will just unpack them (unpacking will take some additional time, but I think we don't notice it because compilation of other packages take much more time).
_________________
WBR, Alex
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 09, 2005 10:30 pm    Post subject: Reply with quote

oh. no, the script actually takes that into account. if you use the -t flag it builds only the toolchain, if you use -w system, it builds the system packages MINUS the toolchain. :D
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5 ... 28, 29, 30  Next
Page 4 of 30

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum