A more robust installation method

Message

rac · Post by **rac** » Thu Nov 18, 2004 11:13 pm

All installs done with stageballs other than stage 3 have two fundamental flaws: they are susceptible to circular dependencies within the base system, and they have the potential for unwanted files from the stageball to remain on the install, because they have incomplete /var/db/pkg.

There are some 80+ packages in a stage1ball that are not listed in /var/db/pkg. Why? When you do your "emerge system", you would want your new toolchain to be used to compile all software. If portage sees that a particular version from the stageball is still current, it will omit it. The solution that somebody apparently chose was to make portage forget that most of this software is installed at all, which has the unfortunate side effect of making portage be unable to clean it when your "emerge system" finishes.

The solution here is to add an -e to that first "emerge system", which will recompile everything even if it is current. Since we're going to use emerge -e system, we might as well use a stage3ball, which has a proper /var/db/pkg. This has the pleasant side-effect of making us immune to circular dependencies as well, which crop up occasionally.

To summarize, use a stage3ball, run bootstrap.sh if you wish to do what you would normally think of as a "stage 1 install", or omit this step if you are satisfied with the toolchain in the stageball. Then "emerge -e system". I think it would be great if this install method made it into the handbook, and would allow /var/db/pkg to be added to all the stageballs, or, in the interests of space and simplicity, only what we now know as "stage3balls" really need to be distributed.

You will find both in forum posts, bugs, and on IRC, various ad hoc methods of resolving circular dependencies. Some are good, some are bad, most are somewhere in between. Almost all of them run the risk of you ending up with core system packages being built in a half-baked state, which is why I recommend that you bypass this entire class of headaches entirely by using this install method.

curtis119 · Post by **curtis119** » Fri Nov 19, 2004 12:17 am

rac,
Thank you for posting this great tip. I have used this very method to get a gentoo system up and running when time was of the essence (saving the 'emerge -e system' for a later, more convienent time). After reading about circular dependencies and the incomplete /var/db/pkg, I will use this method for all of my gentoo installs in the future.

I wanted to add that since an emerge -e system will recompile the entire system anyways, now would be a perfect time to change the chost, mcpu, mtune and any other c/cxxflags to suit your platform.

Genone · Post by **Genone** » Fri Nov 19, 2004 10:00 am

Hmm, if you changed USE flags you probably want a `emerge --depclean` too.

rac · Post by **rac** » Tue Nov 23, 2004 7:30 am

I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this:

Code: Select all

# emerge os-headers glibc binutils gcc

I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world.

Primera · Post by **Primera** » Mon Nov 29, 2004 5:44 am

Rac,
We spoke briefly about his thread in IRC, and I was hoping you might be able to go into a bit more detail for us less familiar. Could you possibly outline the steps in a bit more detail? Such as...do you have to install the portage snapshot from the livecd before

Code: Select all

emerge -e system

? How does

Code: Select all

emerge os-headers gcc glibc binutils

fit into all this? When would we change USE settings and such, as well as when would we do an emerge sync? Thanks.

seank · Post by **seank** » Mon Nov 29, 2004 6:07 am

rac wrote:I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this:
Code: Select all
# emerge os-headers glibc binutils gcc
I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world.

If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands?

seank · Post by **seank** » Mon Nov 29, 2004 6:14 am

Primera wrote:Such as...do you have to install the portage snapshot from the livecd before
Code: Select all
emerge -e system
?

Yes, you have to install portage before emerging system.

Primera wrote:How does
Code: Select all
emerge os-headers gcc glibc binutils
fit into all this?

Like rac said, if you aren't satisfied with the stage3's bootstrap job, then you would run that emerge command twice before running the emerge -e system.

Primera wrote:When would we change USE settings and such, as well as when would we do an emerge sync?

Modify your USE flags in /etc/make.conf before emergeing system and if you're going to re-bootstrap, do it before then too (I think, someone correct me if I'm wrong). Same goes for syncing portage.

I wrote this small howto page for my personal reference on rac's way of installing Gentoo, but he has NOT YET APPROVED it. So don't come balling to me or rac if it doesn't work.

rac · Post by **rac** » Tue Nov 30, 2004 5:38 am

If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands?

No need. Since you are starting from a stage3ball, you don't need to avail yourself of the trickery involved with those USE. They are designed to inform ebuilds that they must tread carefully, crippling themselves if necessary, to function in the limited environment of a stage1ball.

rac · Post by **rac** » Tue Nov 30, 2004 5:41 am

To clarify some stuff Primera was asking about, think of the "bootstrap" process as having two halves:

Code: Select all

emerge os-headers glibc portage binutils gcc

...and

Code: Select all

emerge glibc binutils gcc

The second half doesn't do os-headers again simply because I have a hard time imagining a situation in which changes in glibc/binutils/gcc versions would affect the result. You of course may include it for completeness. Between these two halves is a nice place to adjust CFLAGS and gcc-config profile, to take advantage of new features in your new compiler version.

Those wishing to do what has historically has been named a "stage 1" install do both halves of the bootstrap process. Those wishing to do a "stage 2" omit the second half, and those wishing to do a "stage 3" skip both halves.

While I'm at it, a few people who were very helpful in helping me diagnose the various issues and discuss options: ferringb, jstubbs, and ciaranm.

EDIT: We should do portage early, lest it restart the package count in the emerge -e system. It's not absolutely critical where it goes, since it will get rebuild eventually, so let's stick it after glibc in pass 1.

rac · Post by **rac** » Tue Nov 30, 2004 10:15 pm

For those of you wishing to use linux26-headers, you have a couple of choices. Where we mention "os-headers", you can explicitly say "linux26-headers", or you can add "nptl" (and "nptlonly", if you don't want multiple glibcs and don't need linuxthreads) to your USE, which will shift this virtual.

Since you can only have one set of userland kernel headers, you would need to emerge -C linux-headers before emerging linux26-headers.

rhill · Post by **rhill** » Sun Dec 05, 2004 3:18 am

hey rac, i was wondering if this could play off of another idea i was having. i've noticed that a lot of the time when building a gentoo system, you're installing things twice, whether it be "emerge -e system, emerge -e system, emerge -e world, emerge -e world" like robmoss suggests, or building the toolchain twice which is what you're suggesting (and i think is a little less overkill). what would be the validity of doing the first pass without optimizations, to save compile time? does this invalidate the whole point of building the toolchain twice? i understand that the first build is done to create a complete (possibly upgraded) toolchain. the second build is to build the toolchain using the complete (possibly upgraded) toolchain. the second build produces an optimized toolchain, but does it have to be build _with_ an optimized toolchain to acheive that?

just in case my question isn't clear, here's the jist:

Code: Select all

(w/ CFLAGS="-O0 -march=<arch> -pipe)
emerge -v linux26-headers binutils glibc gcc

(change CFLAGS="-O2 -march=<arch> -fomit-frame-pointers -fetc... -pipe)
emerge -v linux26-headers binutils glibc gcc

or does this defeat the purpose of dual-building in the first place?

tcostigl · Post by **tcostigl** » Sun Dec 05, 2004 5:46 pm

when i do:
emerge -pv kinux26-headers glibc binutils gcc

I see linux-headers-...
and linux26-headers-...

should both be emerged? ( I did emerge -C linux-headers)

Also, in sean_mkcken's howto he says to add the versions that will be emerged to /etc/portage/package.mask

what exactly is that for?

Are the '>' pat of the lines? Is so, then we want to mask everything higher than the version that will be emerged? why is this necassary?

tcostigl · Post by **tcostigl** » Sun Dec 05, 2004 6:26 pm

Ok, looks like when there is no "headers" package installed glibc wants to emerge linux-headers by default. I just emerged linux26-headers separately first ( not sure if this is even necassary). The continued with emerge glibc ...

I still would like to know why we add these package versions to package.mask as in sean_mickens howto?

thanks

rhill · Post by **rhill** » Tue Dec 07, 2004 2:36 am

hrm. considering i just noticed glibc sets its own opt flags and gcc does two pass profiling on itself as it compiles, i submit my previous post to the Stupid Question Bin.

Deranger · Post by **Deranger** » Tue Dec 07, 2004 10:06 am

I just tried it inside a chroot jail. Grabbed stage3ball, bootstrapped and did emerge -e system twice. This is going to be my new playground

Thanks for the tip, rac!

hielvc · Post by **hielvc** » Thu Dec 09, 2004 7:46 pm

For anyone who finds this post and is interested in trying it See ali3nx's Installing Gentoo - The Developers Method - Stage1 and NPTL and then go to the bottom of pg16 for the supper dudduper onliner that incorperates this approch.

rac · Post by **rac** » Fri Dec 10, 2004 7:21 am

dirtyepic wrote:does it have to be build _with_ an optimized toolchain to acheive that?

No. You could use -O0 for the first pass if you want, assuming that all those packages will actually build with that flag, which I haven't checked. gcc must generate the same code regardless of what flags itself was built with, otherwise it will fail its internal consistency checks. I would say that any change in how binutils behaves depending on CFLAGS would likewise be a binutils bug.

Fleta · Post by **Fleta** » Tue Dec 14, 2004 7:08 pm

Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning?

Deranger · Post by **Deranger** » Tue Dec 14, 2004 7:11 pm

Fleta wrote:Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning?

emerge --resume

Ateo · Post by **Ateo** » Wed Dec 22, 2004 8:32 pm

Thanks for clarifying this difference between the stages. This is an excellent tip.

hielvc · Post by **hielvc** » Tue Jan 04, 2005 3:14 am

The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install

kimchi_sg

Code: Select all

env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc && gcc-config 2 && env-update && . /etc.profile && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default

From pg21
Edit one liner But I would suggest that you go to this Howto by Bob_P Installing Gentoo: Stage 1 NPTL on a Stage 3 Tarball

Bob P · Post by **Bob P** » Wed Jan 05, 2005 2:45 am

hielvc wrote:The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install

kimchi_sg

Code: Select all

env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc portage && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default

From pg21

there are two things that i still don't understand about this proposed one-liner. maybe i'm missing something, or maybe i'm seeing something that nobody else is seeing. maybe this isn't evident because we're used to thinking about this in the context of a Stage 1 tarball, but with a Stage 3 tarball the context is different.

assume that you have GCC 3.3.4 on your system, as you would have if you installed from a 2004.3 Stage 3 tarball. after performing "emerge gcc" as depicted in the one-liner, you have successfully emerged GCC 3.4.3 onto your system, but GCC 3.3.4 remains your default compiler. the next sequence of steps involves rebuilding the entire system via the "emerge-e system" command without taking advantage of the newly emerged compiler!

to successfully install GCC 3.4.3 as the default compiler, you have to break the sequence of events in the one-liner and insert "gcc-config 2" in order to set 3.4.3 as the default compiler. if you fail to do this, you're going to waste an awful lot of time running through that one liner. although you'll be working under the assumption that you're using 3.4.3 to rebuild your system, you're actually using 3.3.4 -- which amounts to a COLOSSAL waste of time! as soon as you've emerged GCC 3.4.3 you need to actually install it to get any benefit from it.

to test this theory i started building a Stage 1 on 3 system from scratch the other day, and this is exactly what happened! a quick look at the output of "gcc-config -l" after i had emerged gcc 3.4.3 confirmed my suspicions.

the second question that i have also relates to GCC 3.4.3. after emerging the latest GCC compiler, you should really stop to re-assess your USE flags and CFLAGS statements in make.conf. if you really want to gain anything useful from rebuilding your toolchain, you should stop to peruse the new CFLAGS options offered in 3.4.3, and make any pertinent changes to /etc/make.conf. after doing that, it would be time to resume the one-liner, picking up where it left off at "emerge-e system."

since we're interrupting the flow of the one-liner to take advantage of GCC, it would seem to make sense to recompile glibc, binutils, gcc and portage using the new 3.4.3-built-with-3.3.4 compiler. this will result in a compiler tooklit that is 3.4.3-built-with-3.4.3. THAT is what you really want to use to perform your "emerge -e system", isn't it?

logon · Post by **logon** » Sat Jan 08, 2005 9:33 am

Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here?

Genone · Post by **Genone** » Sun Jan 09, 2005 5:27 pm

logon wrote:Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here?

- You don't necessarily get the same versions
- Depending on USE flags (and maybe other configuration data) the package might produce different files
- It breaks some sanity checks like collision-protect
- It's generally a bad idea to castrate vardb

hielvc · Post by **hielvc** » Thu Jan 20, 2005 4:27 am

Posted an emerge wrapper tha does this for emerge updates since portage will build vour toolchain updates ass backwards . An emerge wrapper for correctly building the toolchain