Installing Gentoo 2004.3: Stage 1 NPTL on a Stage 3 Tarball

[jack.stay]

Given the change in the tutorial to the stable branch, I'm struggling to understand one subtlety related to the installation:

Why does the system compile GCC 3. 4.3 under step 7.1 of the tutorial? Given that the make.conf now includes ACCEPT_KEYWORDS="x86" I was expecting to see something like sys-devel/gcc ~x86 in package.keywords prior to emerging gcc in this step.

I am a complete new comer to Linux, but I have been following the progress of this tutorial with great interest. It has been a wonderful learning experience.

[Insanity5902]

i am going to take a shot in the dark but ...

with a ACCEPT_KEYWORDS="x86" you end up an extremely stable system and aren't testing out packages. What you need to do to still use gcc 3.4.3 and other select packages is add them to your /etc/portage/package.keywords

so a simple command like this

Code: Select all

echo sys-devel/gcc ~x86 >> /etc/portage/package.keywords

This is what I startedto do one my laptop, added only packages that have advanced features I want, or ones that are masked with the ~x86. It will create a truly customized and a hell of a lot more stable system then a pure ~x86.

The reason we are going to go ahead and emerge gcc3.4.3 b/c it can provide some big speed imporvments for you system.

I am built my server very close to this guide before it came out and I am going to rebuild my desktop to this. And I am going to update my laptop to gcc 3.4.x this summer (after this semster of college).

I honestly think this guid along with a x86 system utlizing package.keywords should be the standard for those wanting a very fast system, yet stable as possble. IMO, this guide gives you everybit of performance and still provides a lot of stabilty. The simple fact it isn't a pure x86 could mean you are giving up some stabliity

[lockfree]

when i "emerge gcc-config glibc binutils gcc", it takes a really long time - 14 hours and still ongoing...

My server is a pentiumpro 200MHz with 10k rpm scsi harddisk.

Is this normal? How long did you take to emerge the line above?

[Iron_DragonLord]

200 MHz...? Sounds normal to me.

Took 48 hours straight or more (I think, I was of course asleep at night when I let it run) on my 500 MHz AMD K-3 with just a normal stage 1 install. This was on my server, thus no KDE or anything for graphics, too.

[lockfree]

It has taken abt 10hours before that. Running that line of command alone is > 14hrs and counting.. and after choosing the default C compiler to 3.4.3, i need to run that command again.. man! that's a pretty long time!

Anyway, how to have xcfe as the default windows manager, instead of kde, gnome, etc. At which stage in this manual should it be configured and how?

Thanks.

[hielvc]

My old p166 would take at least 5,may have been 7, days to build through gnome and mozilla it had 128Megs ram.

I agree with jack.stay that sys-devel/gcc needs to be added to /etc/portage/package.keywords as all versions of gcc above gcc-3.3.4-r1 are "~x86".

[Bob P]

Ever heard of distcc there Bob

well, yes i have... but its not doing me any good as i've i've got all 3 of my machines compiling their own stuff right now!

i also have to assume that some of your ribbing was in jest, as we all know how distcc will b0rk up compilation when the two machines are running different versions of gcc! the fact we're migrating from 3.3.4 to 3.4.3 makes distributed compilation a tricky implementation.

i am also somewhat biased against using distcc when testing modifications to the tutorial -- i want to limit failures to failures in the tutorial itself, and i don't want to be subject to distcc-induced complications. unfortuantely, distcc adds its own set of potential problems. sometimes its just "cleaner" not to use distcc.

[Bob P]

Given the change in the tutorial to the stable branch, I'm struggling to understand one subtlety related to the installation:

Why does the system compile GCC 3. 4.3 under step 7.1 of the tutorial? Given that the make.conf now includes ACCEPT_KEYWORDS="x86" I was expecting to see something like sys-devel/gcc ~x86 in package.keywords prior to emerging gcc in this step.

I am a complete new comer to Linux, but I have been following the progress of this tutorial with great interest. It has been a wonderful learning experience.

Jack, if you're a newcomer, you're doing a good job and you're catching on quickly.

yes, GCC 3.4.3 is in the testing branch, and by changing the keywords in make.conf to use the stable branch instead of the testing branch, we'd torpedo our ability to use GCC 3.4.3. the key to success, as both you and Insanity pointed out, is to explicitly specify the testing branch for GCC in /etc/portage/package.keywords.

Unfortunately, the tutorial doesn't yet reflect the necessary changes to implement the stable branch -- I am midstream in prototyping the changes on a testbed PC and I haven't gotten far enough to update the tut yet. From a theoretical standpoint, if you use the step that Insanity recommended (just before updating the Portage tree in section 6.6.x) you should be good to go. don't forget to also add libstdc++-v3 as it is a required dependency.

Insanity, I've been spending the day working on the testbed PC, fixing a b0rked Windows box (dammit!) and updating the PDF along the way. Once that's finished I'll update the tut and let you know that the hardcopy is updated as well.

lockfree, if you're doing this install on a P200, you are one serious glutton for punishment. the steps in section 7.1 alone will proably take you 36 hours on that hardware. for a full installation, you're probably looking at at least a week of compilation! didn't you take me seriously when you read the warning at the beginning of the howto?

WARNING: This is an ADVANCED installation method. The amount of time required for you to complete this type of installation will equal or exceed that of any other Gentoo installation method. Your time will be well invested, though, as the result will be a very stable Gentoo system. This installation method may prove to be somewhat difficult and cumbersome for users who are new to Gentoo. It will prove especially painful for users who plan to install Gentoo old hardware.

I really need to pick-up a couple of more testbed PCs -- after writing this tut it became painfully obvious that I need some faster testbeds.

[mtamizi]

does a normal person really need to be running xinetd?

Since there was no update in the tutorial, I recommend everyone who will add xinetd to their default run level read Securing Services in the Gento Security Guide. As a general rule, if you don't know what xinetd or inetd are, you don't need them. Although I do recommend learning what xinetd is and what it is used for.

[gnough]

I would like to request that users who do not fully understand why certain critical steps in the guide are being taken should refrain from making recommendations to remove them just because they fail to understand them.

The post I wrote here was a complete garbage.
Sorry for making such a stupid comment.

[hielvc]

Heres a automated wrapper for emerge that will emerge world -uD unless linux26-headers or glibc are being updated in which case it rebuilds the toolchain first and then with an edited list emerges the rest of package list.

This script is meant for use after you've built your system and want to quarantee a stable toolchain.

Code: Select all

#!/bin/bash
# 
# Date 1-12-05 version 1.0, Hiel Van Campen
# Use at yee own risk. It works for me, but then I wrote the damn thing.
# A wraper to run emerge world -uD
# It chechs if linux26-headers or glibc are being updated and if so runs the 
# toolchain thang. It then uses an edited -uDp generated list wtih toolchain items removed to
# emerge the rest of the list
# 
# Toolchain thang
# emerge linux26-headers glibc && emerge glibc binutils gcc && emerge binutils gcc
#   
## Changed the to use awk as it removes toolchain items in one pass
#           
#   cut -f2 -d "]" -s wrld.lst|awk '!/glibc|glibc|binutils|gcc/ {print $1}'
##
##   
##  The build script for the edited wrld.lst-->build.lst. Anything that fails to emerge
##  is copied to failed.lst  To view the build.lst or failed.lst use "less failed.lst"
##  for i in $(< wrld1.lst);do emerge --oneshot  =$i || echo $i>> failed.lst;done 
##   

a=0  # If 0 then update doesnt have linux-headers or glibc update and continue emerge. If set to 1 then
     # rebuild toolcahin and then emerge from the build list after running toolchain thang. 

:>build.lst   # empty build.lst and wrld.lst other wise the next time they are run theyll still have 
:>wrld.lst    # the old stuff in them plus any new stuff, could becaome a very long list.
:>failed.lst  # empty failed.lst so that only those that failed this time are listed  


emerge world -uDp>>wrld.lst

if echo "linux26" | grep -q linux26 wrld.lst
	then a=1
elif echo "glibc" | grep -q glibc wrld.lst
	then a=1
fi

if [ "$a" -ge "1" ]
   then	
	for line in $(cut -f2 -d "]" -s wrld.lst|awk '!/glibc|glibc|binutils|gcc/ {print $1}')
	do echo $line>>build.lst
	done
	
	echo "Found glibc or linux26-headers rebuilding toolchain"
	
	emerge linux26-headers glibc && emerge glibc binutils gcc && emerge binutils gcc \
	&& for i in $(< build.lst);do emerge --oneshot  =$i || echo $i>> failed.lst;done
	
			
   else 
	echo "No glibc or linux26-headers update so procedeing with regular emerge world -uD."
	emerge world -uD
   
fi

exit 0

Cut and paste into a text editor, save as emerge_wrapper.sh. then "chmod 774 emerge_wrapper.sh. You can then copy to /usr/local/bin/ and run it.

[Bob P]

I think what Bob wants to do in emerge -e system with rsync-restriction stuff is to exclude recompilation of glibc, binutils, gcc, linuxheaders26.

No, let me clarify that. In following this guide, you should not restrict rebuilding of the toolkit in any way. The steps in rebuilding the toolkit have been carefully planned and there are no unnecessary steps in them. Each iteration of the system build is absolutely necessary to ensure system stability, and modifying the sequence of events will only cause problems to pop-up farther down the road.

My only recommendations about restricting rebuilds of the toolkit were made in the context of updating a completed Gentoo installation on subsequent world emerges/updates, not system emerges. There's a huge difference between them and its important to understand the difference.

Sorry if a failure to clarify things on my part led to this misunderstanding.

I'm going to rant a little bit on this subject in my next post. I've had alot of people contact me via PMs with questions like these, so although I'll be using your case as an example, please don't get the impression that this is directed at you.

edit: changed "redundant" to "unnecessary" in second sentence.

[Bob P]

ok. This is my first stage1 equivalent installation..

I followed up until:

Code: Select all

# env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge gcc-config glibc binutils gcc

and:

Code: Select all

# gcc-config 2 && env-update && source /etc/profile && emerge glibc binutils gcc portage && emerge -e system

I think what Bob wants to do in emerge -e system with rsync-restriction stuff is to exclude recompilation of glibc, binutils, gcc, linuxheaders26.

I didn't want them to be recompiled either becuase it takes so much time and time is money and with that money, I can upgrade my system and save compilation time, which is equivalent to saving time because time = money.
/me kids and hides

That approach may save you some time, but saving time by using a half-baked approach to rebuild your toolkit after switching C compilers is absolute false economy. You are going to seriously undermine the stability of your Gentoo box by selectively choosing to follow some sections of the tutorial and not others, especially if you decide to throw away some of the sections about building the toolkit. The stability of the system absolutely requires that you take the extra steps to build the toolkit properly, and the stability of your toolkit absolutely requies that you follow ALL of the steps, not just the ones that look good at a casual glance.

A Stage 1 on 3 installation is significantly more complex than a Stage 1 installation method because the contents of the tarballs differ. The difference in the content of the tarballs requires that additional steps be taken to change the system headers, compilers and libraries, AND to redundantly rebuild the system toolkit after making such a change.

It makes absolutely NO SENSE that anyone would choose to use THIS tutorial on how to build a Stage 1 system on a Stage 3 tarball if they should plan to excise critial steps along the way. Even limited experience with Stage 1 type installations, the GCC compiler, and rebuilding Gentoo system toolkits should prevent the reader from contemplating such an approach unless he doesn't really understand the focus of this installation guide.

As I had mentioned in the header to this guide, this is an alternative method to building Gentoo that is guaranteed to take longer than any other currently published method of building Gentoo. It makes no sense that anyone should choose to follow this installation guide if they do not have sufficient time to complete the installtion procedures. If time is a critical factor, there are other ways to build Gentoo that will take less time, that will not require emasculation of the procedures to achieve a speedier finish for the impatient.

There is absolutely NO GOOD REASON for skipping the steps where the toolchain is rebuilt. Rebuilding the toolchain in a redundant fashion is the focal point of this installation method. Everything in the tutorial up to that point is just series of supporting steps to bring the reader to the point where they can safely rebuild their toolkit. This entire installation method was designed around this step! Skipping this step will seriously undermine the robust nature of this Gentoo installation method. Readers who do not intend to perform this step have been wasting an awful lot of time (time is money!) by following this installation guide -- by deleting some of the apparently "unnecessary" steps, the "bulletproof" nature of this installation method will be seriously undermined.

While I appreciate the helfpul suggestions that have been made in an effort to refine the content of this guide, I would like to request that users who do not fully understand why certain critical steps in the guide are being taken should refrain from making recommendations to remove them just because they fail to understand them. Other less experienced people may read such comments and choose to follow such ill-advised methods without understanding the perils that would lie in store for them.


On the subject of skipping a toolkit rebuild on subsequent world updates, yes, control over rebuilding of the toolkit is an important consideration for maintaining the stability of your Gentoo system -- but you should NOT be avoiding a complete rebuild of the toolkit because it takes too much time, as that is absolute false economy!

Unplanned rebuilding of isolated toolkit components in a single pass is to be avoided because a more throrough approach (think robmoss) is required to assure toolkit stability and total system stability after a major toolkit component is changed. Masking of the toolkit components may be useful for those with time constraints who wish to delay toolkit rebuilds until such time as they can appropriately addressed. I have done it, and I have written a tutorial on how to do it. I've decided not to publish it though. Everyone familiar with Portage should have no problem performing such a task, while those who aren't sufficiently familiar with Portage may be best advised not to bother.

[Clyde]

(I post this not as a newcomer seeking support, but in the spirit of proofing/improving the tutorial. )

1. Which, if any, of the USE flags in section 6.5 are considered required before building the toolchain? I ask because several of them are not in the global use.desc, specifically:

freetype hal ide ithreads pthreads userlocales

although some are in use.local.desc for certain packages.

2. 9.3 should read "Gensplash"

[Insanity5902]

these are all used to to build the system

As I understand it, on a very basic level ...

Global use flags are for more the one package, hence the global tagging.

Local use flags are for specific apps.

What some people do is do a look up on all the use flags they use. If it is global that add it to the make.conf if it is local they put add it to /etc/portage/package.use

Code: Select all

echo <package-level>/<package-name> <useflags> >> /etc/portage/package.use

[mtamizi]

I'm finishing up an install using this tutorial and I came across a problem when emerging ttmkfdir -- a dependency for x11. I fixed the problem by first doing the following:

Code: Select all

ACCEPT_KEYWORDS=~x86 emerge libtool

[kimchi_sg]

I fixed the problem by first doing the following:

Code: Select all

ACCEPT_KEYWORDS=~x86 emerge libtool

Uh huh. I wouldn't induce ACCEPT_KEYWORDS=~x86 so easily.

Simply emerging libtool twice in a row should solve the problem.

Code: Select all

emerge libtool && emerge libtool

[thecookie]

If anyone was wondering;

http://forums.gentoo.org/viewtopic.php?p=1976103

How does this affect the recompiling of the toolchain? Is it needed if gcc dosn't take all the new flags?

Is the newer gcc more optimized in general compiling too?

[mtamizi]

Simply emerging libtool twice in a row should solve the problem.

Code: Select all

emerge libtool && emerge libtool

Why would that work?

[Bob P]

I'm finishing up an install using this tutorial and I came across a problem when emerging ttmkfdir -- a dependency for x11. I fixed the problem by first doing the following:

Code: Select all

ACCEPT_KEYWORDS=~x86 emerge libtool

well, you've brought up an interesting question. installing x11 is somewhat beyond the scope of this thread, but since its a very popular program that alot of people will be installing, i'd like to hear more -- especially since others have done it without problems while you seem to have hit an obstacle. unfortunately, you haven't given us any specific information to go on. we need more information from you. there are going to be alot of questions specific to your application/installation that may not be relevant for everyone else. could you start a support thread in the Desktops forum so that we can pursue the subject there?

[Bob P]

Which, if any, of the USE flags in section 6.5 are considered required before building the toolchain? I ask because several of them are not in the global use.desc, specifically:

freetype hal ide ithreads pthreads userlocales

although some are in use.local.desc for certain packages.

unfortunately, the state of documentation for use flags is somewhat behind the times. there are already over 100 undocumented use flags, which makes the subject of use flags a difficult one to address.

hal is a use flag that is invoked by one of the gnome packages. if you don't use gnome (and none of your packages have gnome dependencies), you can remove it.

userlocales is used by glibc. this should be evident, as its discussed in the tutorial.

to be honest, i don't remember exactly where i picked up freetype, but iirc it came from X11 or some X program.

ithreads and pthreads are necessary for the system. they are also poorly documented. ithreads is specifically needed to fully support nptl by perl/libperl. i had never heard of the flag until i saw it roll across the screen as i watched perl/libperl compile. if you don't use ithreads, perl/libperl issue screen output asking you to set that flag and then recompile them.

i guess that the moral of the story is that somebody expects us to sit at our computers and actually read the text output that's displayed when programs compile!

[DrWoland]

Worked fine here aswell, it's stable but it doesn't feel much more 'optimised' than it did with gcc 3.3 But apps do seem to compile faster than in gcc 3.3.

Code: Select all

CFLAGS="-O2 -mtune=athlon-xp -march=athlon-xp -fomit-frame-pointer -fforce-addr -momit-leaf-frame-pointer -frename-registers -fweb -ftracer -pipe"
CXXFLAGS="${CFLAGS} -fvisibility-inlines-hidden"

I used these same flags and so far so good - booted fine, everything works fine. I'll see when I get home from work if nvidia and xorg emerged properly.

[Bob P]

ithreads and pthreads are necessary for the system. they are also poorly documented. ithreads is specifically needed to fully support nptl by perl/libperl. i had never heard of the flag until i saw it roll across the screen as i watched perl/libperl compile. if you don't use ithreads, perl/libperl issue screen output asking you to set that flag and then recompile them.

i left out the part about NPTL. if you compile perl/libperl without NPTL support, you won't see the ithreads message. IIRC, if you compile perl/libperl with the nptl use flag set, then they look for the status of the ithreads flag and issue an error warning if both ntpl and ithreads are not both set. at least that's the way i remember it.

[jerickson314]

There is already a thread open for the libtools/ttmkfdir error at http://forums.gentoo.org/viewtopic.php?p=1977938.

[kimchi_sg]

Simply emerging libtool twice in a row should solve the problem.

Code: Select all

emerge libtool && emerge libtool

Why would that work?

Honestly, I do not know. But I know it works. I had that problem, and that was the fix.

If I want total system stability, a libtool error would NEVER be an excuse to use ACCEPT_KEYWORDS=~x86 so easily. Unless a dev in the bugzilla tells me to, I suppose.