View previous topic :: View next topic |
Author |
Message |
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Thu Nov 18, 2004 11:13 pm Post subject: A more robust installation method |
|
|
All installs done with stageballs other than stage 3 have two fundamental flaws: they are susceptible to circular dependencies within the base system, and they have the potential for unwanted files from the stageball to remain on the install, because they have incomplete /var/db/pkg.
There are some 80+ packages in a stage1ball that are not listed in /var/db/pkg. Why? When you do your "emerge system", you would want your new toolchain to be used to compile all software. If portage sees that a particular version from the stageball is still current, it will omit it. The solution that somebody apparently chose was to make portage forget that most of this software is installed at all, which has the unfortunate side effect of making portage be unable to clean it when your "emerge system" finishes.
The solution here is to add an -e to that first "emerge system", which will recompile everything even if it is current. Since we're going to use emerge -e system, we might as well use a stage3ball, which has a proper /var/db/pkg. This has the pleasant side-effect of making us immune to circular dependencies as well, which crop up occasionally.
To summarize, use a stage3ball, run bootstrap.sh if you wish to do what you would normally think of as a "stage 1 install", or omit this step if you are satisfied with the toolchain in the stageball. Then "emerge -e system". I think it would be great if this install method made it into the handbook, and would allow /var/db/pkg to be added to all the stageballs, or, in the interests of space and simplicity, only what we now know as "stage3balls" really need to be distributed.
You will find both in forum posts, bugs, and on IRC, various ad hoc methods of resolving circular dependencies. Some are good, some are bad, most are somewhere in between. Almost all of them run the risk of you ending up with core system packages being built in a half-baked state, which is why I recommend that you bypass this entire class of headaches entirely by using this install method. _________________ For every higher wall, there is a taller ladder |
|
Back to top |
|
|
curtis119 Bodhisattva
Joined: 10 Mar 2003 Posts: 2160 Location: Toledo, Ohio,USA, North America, Earth, SOL System, Milky Way, The Universe, The Cosmos, and Beyond.
|
Posted: Fri Nov 19, 2004 12:17 am Post subject: |
|
|
rac,
Thank you for posting this great tip. I have used this very method to get a gentoo system up and running when time was of the essence (saving the 'emerge -e system' for a later, more convienent time). After reading about circular dependencies and the incomplete /var/db/pkg, I will use this method for all of my gentoo installs in the future.
I wanted to add that since an emerge -e system will recompile the entire system anyways, now would be a perfect time to change the chost, mcpu, mtune and any other c/cxxflags to suit your platform. _________________ Gentoo: it's like wiping your ass with silk. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9554 Location: beyond the rim
|
Posted: Fri Nov 19, 2004 10:00 am Post subject: |
|
|
Hmm, if you changed USE flags you probably want a `emerge --depclean` too. |
|
Back to top |
|
|
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Tue Nov 23, 2004 7:30 am Post subject: |
|
|
I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this: Code: | # emerge os-headers glibc binutils gcc |
I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world. _________________ For every higher wall, there is a taller ladder |
|
Back to top |
|
|
Primera n00b
Joined: 05 Nov 2003 Posts: 22 Location: New Orleans, LA
|
Posted: Mon Nov 29, 2004 5:44 am Post subject: |
|
|
Rac,
We spoke briefly about his thread in IRC, and I was hoping you might be able to go into a bit more detail for us less familiar. Could you possibly outline the steps in a bit more detail? Such as...do you have to install the portage snapshot from the livecd before ? How does Code: | emerge os-headers gcc glibc binutils | fit into all this? When would we change USE settings and such, as well as when would we do an emerge sync? Thanks. _________________ All it takes for evil to triumph is for good men to do nothing. |
|
Back to top |
|
|
seank l33t
Joined: 08 Jul 2004 Posts: 686
|
Posted: Mon Nov 29, 2004 6:07 am Post subject: |
|
|
rac wrote: | I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this: Code: | # emerge os-headers glibc binutils gcc |
I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world. |
If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands? |
|
Back to top |
|
|
seank l33t
Joined: 08 Jul 2004 Posts: 686
|
Posted: Mon Nov 29, 2004 6:14 am Post subject: |
|
|
Primera wrote: | Such as...do you have to install the portage snapshot from the livecd before ? | Yes, you have to install portage before emerging system.
Primera wrote: | How does Code: | emerge os-headers gcc glibc binutils | fit into all this? | Like rac said, if you aren't satisfied with the stage3's bootstrap job, then you would run that emerge command twice before running the emerge -e system.
Primera wrote: | When would we change USE settings and such, as well as when would we do an emerge sync? | Modify your USE flags in /etc/make.conf before emergeing system and if you're going to re-bootstrap, do it before then too (I think, someone correct me if I'm wrong). Same goes for syncing portage.
I wrote this small howto page for my personal reference on rac's way of installing Gentoo, but he has NOT YET APPROVED it. So don't come balling to me or rac if it doesn't work. |
|
Back to top |
|
|
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Tue Nov 30, 2004 5:38 am Post subject: |
|
|
Quote: | If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands? |
No need. Since you are starting from a stage3ball, you don't need to avail yourself of the trickery involved with those USE. They are designed to inform ebuilds that they must tread carefully, crippling themselves if necessary, to function in the limited environment of a stage1ball. _________________ For every higher wall, there is a taller ladder |
|
Back to top |
|
|
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Tue Nov 30, 2004 5:41 am Post subject: |
|
|
To clarify some stuff Primera was asking about, think of the "bootstrap" process as having two halves: Code: | emerge os-headers glibc portage binutils gcc |
...and Code: | emerge glibc binutils gcc |
The second half doesn't do os-headers again simply because I have a hard time imagining a situation in which changes in glibc/binutils/gcc versions would affect the result. You of course may include it for completeness. Between these two halves is a nice place to adjust CFLAGS and gcc-config profile, to take advantage of new features in your new compiler version.
Those wishing to do what has historically has been named a "stage 1" install do both halves of the bootstrap process. Those wishing to do a "stage 2" omit the second half, and those wishing to do a "stage 3" skip both halves.
While I'm at it, a few people who were very helpful in helping me diagnose the various issues and discuss options: ferringb, jstubbs, and ciaranm.
EDIT: We should do portage early, lest it restart the package count in the emerge -e system. It's not absolutely critical where it goes, since it will get rebuild eventually, so let's stick it after glibc in pass 1. _________________ For every higher wall, there is a taller ladder
Last edited by rac on Wed Dec 01, 2004 7:04 am; edited 1 time in total |
|
Back to top |
|
|
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Tue Nov 30, 2004 10:15 pm Post subject: |
|
|
For those of you wishing to use linux26-headers, you have a couple of choices. Where we mention "os-headers", you can explicitly say "linux26-headers", or you can add "nptl" (and "nptlonly", if you don't want multiple glibcs and don't need linuxthreads) to your USE, which will shift this virtual.
Since you can only have one set of userland kernel headers, you would need to emerge -C linux-headers before emerging linux26-headers. _________________ For every higher wall, there is a taller ladder |
|
Back to top |
|
|
rhill Retired Dev
Joined: 22 Oct 2004 Posts: 1629 Location: sk.ca
|
Posted: Sun Dec 05, 2004 3:18 am Post subject: |
|
|
hey rac, i was wondering if this could play off of another idea i was having. i've noticed that a lot of the time when building a gentoo system, you're installing things twice, whether it be "emerge -e system, emerge -e system, emerge -e world, emerge -e world" like robmoss suggests, or building the toolchain twice which is what you're suggesting (and i think is a little less overkill). what would be the validity of doing the first pass without optimizations, to save compile time? does this invalidate the whole point of building the toolchain twice? i understand that the first build is done to create a complete (possibly upgraded) toolchain. the second build is to build the toolchain using the complete (possibly upgraded) toolchain. the second build produces an optimized toolchain, but does it have to be build _with_ an optimized toolchain to acheive that?
just in case my question isn't clear, here's the jist:
Code: | (w/ CFLAGS="-O0 -march=<arch> -pipe)
emerge -v linux26-headers binutils glibc gcc
(change CFLAGS="-O2 -march=<arch> -fomit-frame-pointers -fetc... -pipe)
emerge -v linux26-headers binutils glibc gcc |
or does this defeat the purpose of dual-building in the first place? |
|
Back to top |
|
|
tcostigl Tux's lil' helper
Joined: 28 Jul 2004 Posts: 97
|
Posted: Sun Dec 05, 2004 5:46 pm Post subject: |
|
|
when i do:
emerge -pv kinux26-headers glibc binutils gcc
I see linux-headers-...
and linux26-headers-...
should both be emerged? ( I did emerge -C linux-headers)
Also, in sean_mkcken's howto he says to add the versions that will be emerged to /etc/portage/package.mask
what exactly is that for?
Are the '>' pat of the lines? Is so, then we want to mask everything higher than the version that will be emerged? why is this necassary? |
|
Back to top |
|
|
tcostigl Tux's lil' helper
Joined: 28 Jul 2004 Posts: 97
|
Posted: Sun Dec 05, 2004 6:26 pm Post subject: |
|
|
Ok, looks like when there is no "headers" package installed glibc wants to emerge linux-headers by default. I just emerged linux26-headers separately first ( not sure if this is even necassary). The continued with emerge glibc ...
I still would like to know why we add these package versions to package.mask as in sean_mickens howto?
thanks |
|
Back to top |
|
|
rhill Retired Dev
Joined: 22 Oct 2004 Posts: 1629 Location: sk.ca
|
Posted: Tue Dec 07, 2004 2:36 am Post subject: |
|
|
hrm. considering i just noticed glibc sets its own opt flags and gcc does two pass profiling on itself as it compiles, i submit my previous post to the Stupid Question Bin. |
|
Back to top |
|
|
Deranger Veteran
Joined: 26 Aug 2004 Posts: 1215
|
Posted: Tue Dec 07, 2004 10:06 am Post subject: |
|
|
I just tried it inside a chroot jail. Grabbed stage3ball, bootstrapped and did emerge -e system twice. This is going to be my new playground
Thanks for the tip, rac! |
|
Back to top |
|
|
hielvc Advocate
Joined: 19 Apr 2002 Posts: 2805 Location: Oceanside, Ca
|
|
Back to top |
|
|
rac Bodhisattva
Joined: 30 May 2002 Posts: 6553 Location: Japanifornia
|
Posted: Fri Dec 10, 2004 7:21 am Post subject: |
|
|
dirtyepic wrote: | does it have to be build _with_ an optimized toolchain to acheive that? |
No. You could use -O0 for the first pass if you want, assuming that all those packages will actually build with that flag, which I haven't checked. gcc must generate the same code regardless of what flags itself was built with, otherwise it will fail its internal consistency checks. I would say that any change in how binutils behaves depending on CFLAGS would likewise be a binutils bug. _________________ For every higher wall, there is a taller ladder |
|
Back to top |
|
|
Fleta n00b
Joined: 12 Dec 2004 Posts: 68
|
Posted: Tue Dec 14, 2004 7:08 pm Post subject: |
|
|
Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning? |
|
Back to top |
|
|
Deranger Veteran
Joined: 26 Aug 2004 Posts: 1215
|
Posted: Tue Dec 14, 2004 7:11 pm Post subject: |
|
|
Fleta wrote: | Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning? |
emerge --resume |
|
Back to top |
|
|
Ateo Advocate
Joined: 02 Jun 2003 Posts: 2021 Location: Republic of California
|
Posted: Wed Dec 22, 2004 8:32 pm Post subject: |
|
|
Thanks for clarifying this difference between the stages. This is an excellent tip. |
|
Back to top |
|
|
hielvc Advocate
Joined: 19 Apr 2002 Posts: 2805 Location: Oceanside, Ca
|
Posted: Tue Jan 04, 2005 3:14 am Post subject: |
|
|
The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install Quote: | kimchi_sg Code: | env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc && gcc-config 2 && env-update && . /etc.profile && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default |
|
From pg21
Edit one liner But I would suggest that you go to this Howto by Bob_P Installing Gentoo: Stage 1 NPTL on a Stage 3 Tarball _________________ An A-Z Index of the Linux BASH command line
Last edited by hielvc on Thu Jan 06, 2005 12:18 am; edited 2 times in total |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Wed Jan 05, 2005 2:45 am Post subject: |
|
|
hielvc wrote: | The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install Quote: | kimchi_sg Code: | env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc portage && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default |
| From pg21 |
there are two things that i still don't understand about this proposed one-liner. maybe i'm missing something, or maybe i'm seeing something that nobody else is seeing. maybe this isn't evident because we're used to thinking about this in the context of a Stage 1 tarball, but with a Stage 3 tarball the context is different.
assume that you have GCC 3.3.4 on your system, as you would have if you installed from a 2004.3 Stage 3 tarball. after performing "emerge gcc" as depicted in the one-liner, you have successfully emerged GCC 3.4.3 onto your system, but GCC 3.3.4 remains your default compiler. the next sequence of steps involves rebuilding the entire system via the "emerge-e system" command without taking advantage of the newly emerged compiler!
to successfully install GCC 3.4.3 as the default compiler, you have to break the sequence of events in the one-liner and insert "gcc-config 2" in order to set 3.4.3 as the default compiler. if you fail to do this, you're going to waste an awful lot of time running through that one liner. although you'll be working under the assumption that you're using 3.4.3 to rebuild your system, you're actually using 3.3.4 -- which amounts to a COLOSSAL waste of time! as soon as you've emerged GCC 3.4.3 you need to actually install it to get any benefit from it.
to test this theory i started building a Stage 1 on 3 system from scratch the other day, and this is exactly what happened! a quick look at the output of "gcc-config -l" after i had emerged gcc 3.4.3 confirmed my suspicions.
the second question that i have also relates to GCC 3.4.3. after emerging the latest GCC compiler, you should really stop to re-assess your USE flags and CFLAGS statements in make.conf. if you really want to gain anything useful from rebuilding your toolchain, you should stop to peruse the new CFLAGS options offered in 3.4.3, and make any pertinent changes to /etc/make.conf. after doing that, it would be time to resume the one-liner, picking up where it left off at "emerge-e system."
since we're interrupting the flow of the one-liner to take advantage of GCC, it would seem to make sense to recompile glibc, binutils, gcc and portage using the new 3.4.3-built-with-3.3.4 compiler. this will result in a compiler tooklit that is 3.4.3-built-with-3.4.3. THAT is what you really want to use to perform your "emerge -e system", isn't it? _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
logon Apprentice
Joined: 22 May 2002 Posts: 182 Location: Bonn / NRW Germany
|
Posted: Sat Jan 08, 2005 9:33 am Post subject: |
|
|
Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here? |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9554 Location: beyond the rim
|
Posted: Sun Jan 09, 2005 5:27 pm Post subject: |
|
|
logon wrote: | Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here? |
- You don't necessarily get the same versions
- Depending on USE flags (and maybe other configuration data) the package might produce different files
- It breaks some sanity checks like collision-protect
- It's generally a bad idea to castrate vardb |
|
Back to top |
|
|
hielvc Advocate
Joined: 19 Apr 2002 Posts: 2805 Location: Oceanside, Ca
|
|
Back to top |
|
|
|