Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
A more robust installation method
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Thu Nov 18, 2004 11:13 pm    Post subject: A more robust installation method Reply with quote

All installs done with stageballs other than stage 3 have two fundamental flaws: they are susceptible to circular dependencies within the base system, and they have the potential for unwanted files from the stageball to remain on the install, because they have incomplete /var/db/pkg.

There are some 80+ packages in a stage1ball that are not listed in /var/db/pkg. Why? When you do your "emerge system", you would want your new toolchain to be used to compile all software. If portage sees that a particular version from the stageball is still current, it will omit it. The solution that somebody apparently chose was to make portage forget that most of this software is installed at all, which has the unfortunate side effect of making portage be unable to clean it when your "emerge system" finishes.

The solution here is to add an -e to that first "emerge system", which will recompile everything even if it is current. Since we're going to use emerge -e system, we might as well use a stage3ball, which has a proper /var/db/pkg. This has the pleasant side-effect of making us immune to circular dependencies as well, which crop up occasionally.

To summarize, use a stage3ball, run bootstrap.sh if you wish to do what you would normally think of as a "stage 1 install", or omit this step if you are satisfied with the toolchain in the stageball. Then "emerge -e system". I think it would be great if this install method made it into the handbook, and would allow /var/db/pkg to be added to all the stageballs, or, in the interests of space and simplicity, only what we now know as "stage3balls" really need to be distributed.

You will find both in forum posts, bugs, and on IRC, various ad hoc methods of resolving circular dependencies. Some are good, some are bad, most are somewhere in between. Almost all of them run the risk of you ending up with core system packages being built in a half-baked state, which is why I recommend that you bypass this entire class of headaches entirely by using this install method.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
curtis119
Bodhisattva
Bodhisattva


Joined: 10 Mar 2003
Posts: 2160
Location: Toledo, Ohio,USA, North America, Earth, SOL System, Milky Way, The Universe, The Cosmos, and Beyond.

PostPosted: Fri Nov 19, 2004 12:17 am    Post subject: Reply with quote

rac,
Thank you for posting this great tip. I have used this very method to get a gentoo system up and running when time was of the essence (saving the 'emerge -e system' for a later, more convienent time). After reading about circular dependencies and the incomplete /var/db/pkg, I will use this method for all of my gentoo installs in the future.

I wanted to add that since an emerge -e system will recompile the entire system anyways, now would be a perfect time to change the chost, mcpu, mtune and any other c/cxxflags to suit your platform.
_________________
Gentoo: it's like wiping your ass with silk.
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Fri Nov 19, 2004 10:00 am    Post subject: Reply with quote

Hmm, if you changed USE flags you probably want a `emerge --depclean` too.
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Nov 23, 2004 7:30 am    Post subject: Reply with quote

I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this:
Code:
# emerge os-headers glibc binutils gcc

I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
Primera
n00b
n00b


Joined: 05 Nov 2003
Posts: 22
Location: New Orleans, LA

PostPosted: Mon Nov 29, 2004 5:44 am    Post subject: Reply with quote

Rac,
We spoke briefly about his thread in IRC, and I was hoping you might be able to go into a bit more detail for us less familiar. Could you possibly outline the steps in a bit more detail? Such as...do you have to install the portage snapshot from the livecd before
Code:
emerge -e system
? How does
Code:
emerge os-headers gcc glibc binutils
fit into all this? When would we change USE settings and such, as well as when would we do an emerge sync? Thanks.
_________________
All it takes for evil to triumph is for good men to do nothing.
Back to top
View user's profile Send private message
seank
l33t
l33t


Joined: 08 Jul 2004
Posts: 686

PostPosted: Mon Nov 29, 2004 6:07 am    Post subject: Reply with quote

rac wrote:
I also became a bit disillusioned with bootstrap.sh when it refused to allow me to build without USE=multilib, so I used this:
Code:
# emerge os-headers glibc binutils gcc

I run this twice to lock down my toolchain, making sure it was all built with itself, and then go for the emerge -e world.

If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands?
Back to top
View user's profile Send private message
seank
l33t
l33t


Joined: 08 Jul 2004
Posts: 686

PostPosted: Mon Nov 29, 2004 6:14 am    Post subject: Reply with quote

Primera wrote:
Such as...do you have to install the portage snapshot from the livecd before
Code:
emerge -e system
?
Yes, you have to install portage before emerging system.

Primera wrote:
How does
Code:
emerge os-headers gcc glibc binutils
fit into all this?
Like rac said, if you aren't satisfied with the stage3's bootstrap job, then you would run that emerge command twice before running the emerge -e system.

Primera wrote:
When would we change USE settings and such, as well as when would we do an emerge sync?
Modify your USE flags in /etc/make.conf before emergeing system and if you're going to re-bootstrap, do it before then too (I think, someone correct me if I'm wrong). Same goes for syncing portage.


I wrote this small howto page for my personal reference on rac's way of installing Gentoo, but he has NOT YET APPROVED it. So don't come balling to me or rac if it doesn't work.
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Nov 30, 2004 5:38 am    Post subject: Reply with quote

Quote:
If I bootstrap my system this way, should I put USE="-* build bootstrap" before the emerge commands?

No need. Since you are starting from a stage3ball, you don't need to avail yourself of the trickery involved with those USE. They are designed to inform ebuilds that they must tread carefully, crippling themselves if necessary, to function in the limited environment of a stage1ball.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Nov 30, 2004 5:41 am    Post subject: Reply with quote

To clarify some stuff Primera was asking about, think of the "bootstrap" process as having two halves:
Code:
emerge os-headers glibc portage binutils gcc

...and
Code:
emerge glibc binutils gcc

The second half doesn't do os-headers again simply because I have a hard time imagining a situation in which changes in glibc/binutils/gcc versions would affect the result. You of course may include it for completeness. Between these two halves is a nice place to adjust CFLAGS and gcc-config profile, to take advantage of new features in your new compiler version.

Those wishing to do what has historically has been named a "stage 1" install do both halves of the bootstrap process. Those wishing to do a "stage 2" omit the second half, and those wishing to do a "stage 3" skip both halves.

While I'm at it, a few people who were very helpful in helping me diagnose the various issues and discuss options: ferringb, jstubbs, and ciaranm.

EDIT: We should do portage early, lest it restart the package count in the emerge -e system. It's not absolutely critical where it goes, since it will get rebuild eventually, so let's stick it after glibc in pass 1.
_________________
For every higher wall, there is a taller ladder


Last edited by rac on Wed Dec 01, 2004 7:04 am; edited 1 time in total
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Nov 30, 2004 10:15 pm    Post subject: Reply with quote

For those of you wishing to use linux26-headers, you have a couple of choices. Where we mention "os-headers", you can explicitly say "linux26-headers", or you can add "nptl" (and "nptlonly", if you don't want multiple glibcs and don't need linuxthreads) to your USE, which will shift this virtual.

Since you can only have one set of userland kernel headers, you would need to emerge -C linux-headers before emerging linux26-headers.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sun Dec 05, 2004 3:18 am    Post subject: Reply with quote

hey rac, i was wondering if this could play off of another idea i was having. i've noticed that a lot of the time when building a gentoo system, you're installing things twice, whether it be "emerge -e system, emerge -e system, emerge -e world, emerge -e world" like robmoss suggests, or building the toolchain twice which is what you're suggesting (and i think is a little less overkill). what would be the validity of doing the first pass without optimizations, to save compile time? does this invalidate the whole point of building the toolchain twice? i understand that the first build is done to create a complete (possibly upgraded) toolchain. the second build is to build the toolchain using the complete (possibly upgraded) toolchain. the second build produces an optimized toolchain, but does it have to be build _with_ an optimized toolchain to acheive that?

just in case my question isn't clear, here's the jist:

Code:
(w/ CFLAGS="-O0 -march=<arch> -pipe)
emerge -v linux26-headers binutils glibc gcc

(change CFLAGS="-O2 -march=<arch> -fomit-frame-pointers -fetc... -pipe)
emerge -v linux26-headers binutils glibc gcc


or does this defeat the purpose of dual-building in the first place?
Back to top
View user's profile Send private message
tcostigl
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jul 2004
Posts: 97

PostPosted: Sun Dec 05, 2004 5:46 pm    Post subject: Reply with quote

when i do:
emerge -pv kinux26-headers glibc binutils gcc

I see linux-headers-...
and linux26-headers-...

should both be emerged? ( I did emerge -C linux-headers)

Also, in sean_mkcken's howto he says to add the versions that will be emerged to /etc/portage/package.mask

what exactly is that for?

Are the '>' pat of the lines? Is so, then we want to mask everything higher than the version that will be emerged? why is this necassary?
Back to top
View user's profile Send private message
tcostigl
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jul 2004
Posts: 97

PostPosted: Sun Dec 05, 2004 6:26 pm    Post subject: Reply with quote

Ok, looks like when there is no "headers" package installed glibc wants to emerge linux-headers by default. I just emerged linux26-headers separately first ( not sure if this is even necassary). The continued with emerge glibc ...

I still would like to know why we add these package versions to package.mask as in sean_mickens howto?

thanks
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Tue Dec 07, 2004 2:36 am    Post subject: Reply with quote

hrm. considering i just noticed glibc sets its own opt flags and gcc does two pass profiling on itself as it compiles, i submit my previous post to the Stupid Question Bin. :oops:
Back to top
View user's profile Send private message
Deranger
Veteran
Veteran


Joined: 26 Aug 2004
Posts: 1215

PostPosted: Tue Dec 07, 2004 10:06 am    Post subject: Reply with quote

I just tried it inside a chroot jail. Grabbed stage3ball, bootstrapped and did emerge -e system twice. This is going to be my new playground ;)

Thanks for the tip, rac!
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Dec 09, 2004 7:46 pm    Post subject: Reply with quote

For anyone who finds this post and is interested in trying it See ali3nx's Installing Gentoo - The Developers Method - Stage1 and NPTL and then go to the bottom of pg16 for the supper dudduper onliner that incorperates this approch.
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Fri Dec 10, 2004 7:21 am    Post subject: Reply with quote

dirtyepic wrote:
does it have to be build _with_ an optimized toolchain to acheive that?

No. You could use -O0 for the first pass if you want, assuming that all those packages will actually build with that flag, which I haven't checked. gcc must generate the same code regardless of what flags itself was built with, otherwise it will fail its internal consistency checks. I would say that any change in how binutils behaves depending on CFLAGS would likewise be a binutils bug.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
Fleta
n00b
n00b


Joined: 12 Dec 2004
Posts: 68

PostPosted: Tue Dec 14, 2004 7:08 pm    Post subject: Reply with quote

Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning?
Back to top
View user's profile Send private message
Deranger
Veteran
Veteran


Joined: 26 Aug 2004
Posts: 1215

PostPosted: Tue Dec 14, 2004 7:11 pm    Post subject: Reply with quote

Fleta wrote:
Let's say I typed "emerge -e system" before I went to bed and found that one of the packages failed next morning. How do I resume the process when I've fixed the problem? If type "emerge -e system" again, would it start rebuilding from the very beginning?

emerge --resume
Back to top
View user's profile Send private message
Ateo
Advocate
Advocate


Joined: 02 Jun 2003
Posts: 2021
Location: Republic of California

PostPosted: Wed Dec 22, 2004 8:32 pm    Post subject: Reply with quote

Thanks for clarifying this difference between the stages. This is an excellent tip.
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Tue Jan 04, 2005 3:14 am    Post subject: Reply with quote

The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install
Quote:
kimchi_sg
Code:
env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc && gcc-config 2 && env-update && . /etc.profile && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default


From pg21
Edit one liner But I would suggest that you go to this Howto by Bob_P Installing Gentoo: Stage 1 NPTL on a Stage 3 Tarball
_________________
An A-Z Index of the Linux BASH command line


Last edited by hielvc on Thu Jan 06, 2005 12:18 am; edited 2 times in total
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Wed Jan 05, 2005 2:45 am    Post subject: Reply with quote

hielvc wrote:
The latest one liner from Installing Gentoo - The Developers Method - Stage1 and NPTL modified for a stage3 install
Quote:
kimchi_sg
Code:
env-update && source /etc/profile && emerge -C linux-headers && emerge linux26-headers && emerge --nodeps --oneshot gcc-config && emerge --oneshot glibc binutils gcc portage && emerge -e system && emerge syslog-ng xinetd grub hotplug coldplug vixie-cron reiserfsprogs reiser4progs sysfsutils udev dhcpcd && emerge --nodeps acpid ntp && rc-update add syslog-ng default && rc-update add net.eth0 default && rc-update add vixie-cron default && rc-update add xinetd default && rc-update add sshd default && rc-update add coldplug default && rc-update add hotplug default && rc-update acpid default
From pg21


there are two things that i still don't understand about this proposed one-liner. maybe i'm missing something, or maybe i'm seeing something that nobody else is seeing. maybe this isn't evident because we're used to thinking about this in the context of a Stage 1 tarball, but with a Stage 3 tarball the context is different.

assume that you have GCC 3.3.4 on your system, as you would have if you installed from a 2004.3 Stage 3 tarball. after performing "emerge gcc" as depicted in the one-liner, you have successfully emerged GCC 3.4.3 onto your system, but GCC 3.3.4 remains your default compiler. the next sequence of steps involves rebuilding the entire system via the "emerge-e system" command without taking advantage of the newly emerged compiler!

to successfully install GCC 3.4.3 as the default compiler, you have to break the sequence of events in the one-liner and insert "gcc-config 2" in order to set 3.4.3 as the default compiler. if you fail to do this, you're going to waste an awful lot of time running through that one liner. although you'll be working under the assumption that you're using 3.4.3 to rebuild your system, you're actually using 3.3.4 -- which amounts to a COLOSSAL waste of time! as soon as you've emerged GCC 3.4.3 you need to actually install it to get any benefit from it.

to test this theory i started building a Stage 1 on 3 system from scratch the other day, and this is exactly what happened! a quick look at the output of "gcc-config -l" after i had emerged gcc 3.4.3 confirmed my suspicions.

the second question that i have also relates to GCC 3.4.3. after emerging the latest GCC compiler, you should really stop to re-assess your USE flags and CFLAGS statements in make.conf. if you really want to gain anything useful from rebuilding your toolchain, you should stop to peruse the new CFLAGS options offered in 3.4.3, and make any pertinent changes to /etc/make.conf. after doing that, it would be time to resume the one-liner, picking up where it left off at "emerge-e system."

since we're interrupting the flow of the one-liner to take advantage of GCC, it would seem to make sense to recompile glibc, binutils, gcc and portage using the new 3.4.3-built-with-3.3.4 compiler. this will result in a compiler tooklit that is 3.4.3-built-with-3.4.3. THAT is what you really want to use to perform your "emerge -e system", isn't it?
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
logon
Apprentice
Apprentice


Joined: 22 May 2002
Posts: 182
Location: Bonn / NRW Germany

PostPosted: Sat Jan 08, 2005 9:33 am    Post subject: Reply with quote

Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here?
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9501
Location: beyond the rim

PostPosted: Sun Jan 09, 2005 5:27 pm    Post subject: Reply with quote

logon wrote:
Well one question came up in my mind and I can't find the answer myself. You say, there are some packages that are not in the /var/db/pkg list when doing stage1. But aren't they not in the list, so that portage doesn't complain when compiling the same version again? And where is the problem, if the same version gets compiled it should produce the same files, which overwrite the other ones, and then are listed in the package list. Or did i get something wrong here?


- You don't necessarily get the same versions
- Depending on USE flags (and maybe other configuration data) the package might produce different files
- It breaks some sanity checks like collision-protect
- It's generally a bad idea to castrate vardb
Back to top
View user's profile Send private message
hielvc
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca

PostPosted: Thu Jan 20, 2005 4:27 am    Post subject: Reply with quote

Posted an emerge wrapper tha does this for emerge updates since portage will build vour toolchain updates ass backwards . An emerge wrapper for correctly building the toolchain
_________________
An A-Z Index of the Linux BASH command line
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum