| View previous topic :: View next topic |
| Author |
Message |
ElleStone
Tux's lil' helper
Joined: 12 Mar 2013
Posts: 106
|
 Posted: Tue Mar 19, 2013 1:23 pm Post subject: -O#, "-pipe", time to build packages on older comp |
 |
|
Before installing Gentoo on my main computer I have some general questions.
The Gentoo Installation Guide says don't use "-pipe" if the system doesn't have enough RAM. My Acer laptop has 1GB of ram. Should I use "-pipe"?
Building software on my Acer laptop is ***very*** slow, I mean really ridiculously slow. The 32-bit Acer laptop has 1 Intel dual-core (T2080) 1.8 GHz cpu and 1 GB ram.
My 64-bit workstation computer has 12 GB ram and two AMD (Opteron 256) 3.0 GHz single-core processors. Will building Gentoo software on my workstation also be very slow?
I have the option of upgrading the workstation to two Opteron 2.6 GHz dual-core processors. The $cost is low (Ebay used server parts). Would such an upgrade make a perceptible difference in time to build and/or speed of running various CPU-intensive programs? We built this workstation circa 2005. It has a Tyan Thunder K8WE server board, if that is relevant.
At least some parts of the Gentoo Installation Guide date back to 2005. Is "-O2" still recommended as the right choice for make.conf? I found some documentation for Linux from Scratch that recommended "-O3", so I was just wondering.
Thanks! for any input/advice.
Elle |
|
| Back to top |
|
 |
eyoung100
Veteran
Joined: 23 Jan 2004
Posts: 1428
|
| Posted: Tue Mar 19, 2013 2:40 pm Post subject: |
|
|
To offset Low RAM use High Swap. Use the following rule of thumb:
<4GB RAM = Double Swap, so in your Case, 1GB RAM = 2GB Swap
Re: 12GB
This would necessitate nearly no swap. My Machine only has 4GB and a Dual Core Athlon X2 3.2 Ghz. I just switched package managers, so I emereged the emptytree of world(1488 packages) and only took 2 days.
Again as a general rule I use
>=4GB = 1GB for every GB of RAM
12 may be overkill
Compiler Optimizations
_________________
The Birth and Growth of Science is the Death and Atrophy of Art -- Unknown
Registerd Linux User #363735
Adopt a Post | Strip Comments| Emerge Wrapper |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Tue Mar 19, 2013 2:53 pm Post subject: Re: -O#, "-pipe", time to build packages on older |
|
|
| ElleStone wrote: | | The Gentoo Installation Guide says don't use "-pipe" if the system doesn't have enough RAM. My Acer laptop has 1GB of ram. Should I use "-pipe"? |
ElleStone ... using -pipe does not require that much additional ram, but it will improve build time. The reason being that is it uses 'pipes' in place of writing to the filesystem, with the obvious benefit to throughput. 1GB may not seem like a lot of ram, at least not by modern standards, but linux MM is fairly good at managing it, and so I don't see there really being an issue with having '-pipe' set. That said, the firefox and *Office builds can have issues when there isn't masses of ram (and disk space) at their disposal, and so for these you may want to use the *-bin packages.
| ElleStone wrote: | | Building software on my Acer laptop is ***very*** slow, I mean really ridiculously slow. The 32-bit Acer laptop has 1 Intel dual-core (T2080) 1.8 GHz cpu and 1 GB ram. |
I have a T2500 @ 2.00GHz which is in a similar range and actually under utilise the processor when compiling by using only '-j1 -l 1.00', ionice, etc ... and really, I don't notice the 'speed' or lack thereof. I guess it depends on what your expectations are ... speed being a relative phenomena. Anyhow, there may be other reasons for the actual/percieved slowness ... the speed of disk for instance, or the driver sellected for SATA (ie, using ATA_PIIX in stead of native AHCI), and other considerations such as how the disk is tuned.
| ElleStone wrote: | | My 64-bit workstation computer has 12 GB ram and two AMD (Opteron 256) 3.0 GHz single-core processors. Will building Gentoo software on my workstation also be very slow? |
That might somewhat explain the above :) ... but, really, thats not a question that can be answerd objectively.
| ElleStone wrote: | | I have the option of upgrading the workstation to two Opteron 2.6 GHz dual-core processors. The $cost is low (Ebay used server parts). Would such an upgrade make a perceptible difference in time to build and/or speed of running various CPU-intensive programs? We built this workstation circa 2005. It has a Tyan Thunder K8WE server board, if that is relevant. |
Yes, it would ... in comparison with the 3.0 GHz single-core.
| ElleStone wrote: | | At least some parts of the Gentoo Installation Guide date back to 2005. Is "-O2" still recommended as the right choice for make.conf? I found some documentation for Linux from Scratch that recommended "-O3", so I was just wondering. |
Yes, -O2 is recommended. -O3 will not provide any significant increase ... its more likely to cause issues. The recommended CFLAGS are '-O2 -march=native -pipe'.
best ... khay |
|
| Back to top |
|
|
destroyedlolo
l33t
Joined: 17 Jun 2011
Posts: 846
Location: Close to Annecy (France)
|
| Posted: Tue Mar 19, 2013 5:39 pm Post subject: Re: -O#, "-pipe", time to build packages on older |
|
|
| ElleStone wrote: | | The Gentoo Installation Guide says don't use "-pipe" if the system doesn't have enough RAM. My Acer laptop has 1GB of ram. Should I use "-pipe"? |
My previous main system has 1.5 Go of RAM and I never seen it swapping even doing massive compilation and having Firefox launched at the same time ... but it depends also on your DE : I'm using LXDE and it helps.
Part of my distcc network, I have also some 512 Mo machine and they are able to participate to distcc even if LXDE is started. To be fair, my only problem is with old Dell P4 with only 256 Mo where I have to stop X before my distcc is compiling heavy application like Firefox.
| khayyam wrote: | | The recommended CFLAGS are '-O2 -march=native -pipe'. |
Hum, bad idea because it will break Distcc attempt. Instead of -march=native, it's better to use CPU's own optimization flags : they will be propagated to all distcc participant even if they aren't using the same CPU. Obviously, the generated binaries will be optimized only for the distcc master.
Bye
Laurent |
|
| Back to top |
|
|
ElleStone
Tux's lil' helper
Joined: 12 Mar 2013
Posts: 106
|
| Posted: Tue Mar 19, 2013 5:55 pm Post subject: |
|
|
eyoung100, thanks! for the tip about Swap. I gave the laptop 4 GB Swap, twice the recommended amount, but I'm guessing "too much swap" doesn't cause any issues. If "-pipe" runs out of RAM, will it start using Swap? Or will things break?
khayyam, by "slow", for example it took two hours to reinstall gcc. Per your advice (thanks!), I modified make.conf to read '-O2 -march=native -pipe' and also '-j2' (too much?), and now things seem to be compiling faster. My main point of comparison for "slow" is compiling programs (eg babl/gegl/Gimp) on the workstation computer with 4gb of ram and 1 processor; the extra 8gb ram and second processor were only added last week.
The extra RAM, the processor upgrades, and the planned switch from openSUSE to Gentoo all have one goal: get babl/gegl/Gimp from git to run faster. On openSUSE, after compiling babl/gegl/Gimp with CFLAGS="-march=native -Ofast", Gimp ran approximately twice as fast (rather, half as slow) as no optimization at all. CFLAGS="-O3 -ffast-math -ftree-vectorize" was recommended as being a safer set of optimizations for atk/glib/pango/cairo/babl/gegl/gimp. "Per package" settings for items compiled from git rather than portage are an option with Gentoo?
Elle |
|
| Back to top |
|
|
eyoung100
Veteran
Joined: 23 Jan 2004
Posts: 1428
|
| Posted: Tue Mar 19, 2013 6:50 pm Post subject: |
|
|
| ElleStone wrote: | eyoung100, thanks! for the tip about Swap. I gave the laptop 4 GB Swap, twice the recommended amount, but I'm guessing "too much swap" doesn't cause any issues. If "-pipe" runs out of RAM, will it start using Swap? Or will things break?
Elle |
Too much swap never hurt anyone, although you have to remember on smaller systems, you may be trading off hard drive space for increased performance.
A few comments on the other sections before the others reply.
GCC is a PITA to recompile, but you'll get used to it, as believe it or not once compiled most users don't tweak packages in the toolchain on a regular basis. May I recommend the Graphite USE Flag.
Per package USE Flags are allowed, but not every package has an ebuild that will grab from git. Normally packages that come from git contain 9999 as the version number and must be installed from an overlay. See:
Working with Portage
The Gentoo Overlay Guide
in that order as you must learn portage before installing an overlay
_________________
The Birth and Growth of Science is the Death and Atrophy of Art -- Unknown
Registerd Linux User #363735
Adopt a Post | Strip Comments| Emerge Wrapper |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Tue Mar 19, 2013 9:26 pm Post subject: |
|
|
| ElleStone wrote: | | khayyam, by "slow", for example it took two hours to reinstall gcc. Per your advice (thanks!), I modified make.conf to read '-O2 -march=native -pipe' and also '-j2' (too much?), and now things seem to be compiling faster. |
ElleStone ... no, -j2 isn't too much, its about right for that CPU (jobs are normally set equal to the number of processors ... though I've heard say 'processors+1' from some quarters, and 'man make.conf' suggests one or other). This will certainly improve the speed as it run two jobs concurrently. You can also tweek --load-average via MAKEOPTS (eg, "-j2 -l 1.75") which is a method of informing the build process how much it can greedily occupy (obviously the lower value the more is left for other processes). '2.00' would be equivelant of occupying the full extent of both cpu's.
| ElleStone wrote: | | The extra RAM, the processor upgrades, and the planned switch from openSUSE to Gentoo all have one goal: get babl/gegl/Gimp from git to run faster. On openSUSE, after compiling babl/gegl/Gimp with CFLAGS="-march=native -Ofast", Gimp ran approximately twice as fast (rather, half as slow) as no optimization at all. CFLAGS="-O3 -ffast-math -ftree-vectorize" was recommended as being a safer set of optimizations for atk/glib/pango/cairo/babl/gegl/gimp. "Per package" settings for items compiled from git rather than portage are an option with Gentoo? |
Last question first ... when you say "compiled from git" I don't know if you mean *-9999 (or what are sometimes called 'live ebuilds') or if you mean 'git clone git://' and compiled on the system. As far as the later goes its pretty much the same as any distibution, with the understanding that dependencies (libraries, etc), linked to will/may be portage provided and so some level of targeted optimisation involved. As for the former, they are available for some packages but are generally 'masked' due to possible stability issues, if you want the latest and greatest that is possible, but it brings with it some risk, and some need to understand how to manage the system (and any possible breakage).
I'd dispute the claim that '-O3 etc' are 'a safer set of optimizations', infact I'd be inclined to think that this would produce little marked improvement over '-O2', it might even have the opposite effect. This is of course a tricky area because it depends on specific app, version of gcc, etc, but given that gentoo has been compiled hundreds of time with various combinations of CFLAGS, I'd think there would be somekind of concensus here about any quantifiable gain. Other CFLAGS, -ffast-math for instance, may produce some improvement for applications which are particularly math intensive, but the issue is how to quantify "improvement", benchmarks are difficult to apply system wide.
As for the question re "per package settings", yes, this possible, see 6.b. Per-Package Environment Variables
best ... khay |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Tue Mar 19, 2013 9:38 pm Post subject: Re: -O#, "-pipe", time to build packages on older |
|
|
| destroyedlolo wrote: | | khayyam wrote: | | The recommended CFLAGS are '-O2 -march=native -pipe'. |
Hum, bad idea because it will break Distcc attempt. Instead of -march=native, it's better to use CPU's own optimization flags : they will be propagated to all distcc participant even if they aren't using the same CPU. Obviously, the generated binaries will be optimized only for the distcc master. |
destroyedlolo ... when I say 'reccommended' I'm going by the Compilation Optimization Guide its fairly clear under what circumstances -march=native is appropriate ... and not everyone is using distcc ... and in this case the OP is not, so no, its not a "bad idea".
best ... khay |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Tue Mar 19, 2013 11:43 pm Post subject: |
|
|
| khayyam wrote: | | ElleStone wrote: | | khayyam, by "slow", for example it took two hours to reinstall gcc. Per your advice (thanks!), I modified make.conf to read '-O2 -march=native -pipe' and also '-j2' (too much?), and now things seem to be compiling faster. |
ElleStone ... no, -j2 isn't too much, its about right for that CPU (jobs are normally set equal to the number of processors ... though I've heard say 'processors+1' from some quarters, and 'man make.conf' suggests one or other). |
The $(($num_cpus + 1)) thing seems to originate as an attempt to keep the CPUs busy while processes are stuck on IO. It doesn't help as much any more because most desktop systems nowadays have enough RAM to cache the entire workdir and make compiling CPU-bound. Similar reasoning applies to compiling in tmpfs; it helped to prioritise the portage directory in RAM over page cache but it's not a terribly big gain any more. |
|
| Back to top |
|
|
ElleStone
Tux's lil' helper
Joined: 12 Mar 2013
Posts: 106
|
| Posted: Wed Mar 20, 2013 1:41 pm Post subject: |
|
|
Laurent, "distcc" is for compiling packages for a different computer? As perhaps I could compile Gentoo for my laptop, using my main computer, if my main computer were already running Gentoo? Is there a link for how to do this? I have a couple of P-III/512Mb-RAM laptops I'd like to resurrect.
Khay, what is the recommended "-j2 -l 1.75" '-l' value? I didn't see anything about '-l' in the installation guide.
| Quote: | | when you say "compiled from git" I don't know if you mean *-9999 (or what are sometimes called 'live ebuilds') or if you mean 'git clone git://' and compiled on the system. |
By "compiled from git" I mean "git clone git://".
| Quote: | | I'd dispute the claim that '-O3 etc' are 'a safer set of optimizations', in fact I'd be inclined to think that this would produce little marked improvement over '-O2', it might even have the opposite effect. Other CFLAGS, -ffast-math for instance, may produce some improvement for applications which are particularly math intensive, but the issue is how to quantify "improvement", benchmarks are difficult to apply system wide. |
Sorry! I wasn't being clear enough - the claim that '-O3 etc' was safer than '-Ofast' was only with respect to installing atk/glib/pango/cairo/babl/gegl/gimp. I'm not sure how much testing/benchmarking was done, and some of the testing was on a windows system. I'm happy with -O2 (stability is good!), except perhaps with babl/gegl/gimp and other imaging software, as image processing software by definition is math intensive. This link: http://openbenchmarking.org/result/1210138-RA-GCCAMDBUL77 convinced me that -O3/-Ofast isn't always faster than -O2. But often, for image processing, it's faster.
Elle |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Wed Mar 20, 2013 2:22 pm Post subject: |
|
|
| ElleStone wrote: | | "distcc" is for compiling packages for a different computer? As perhaps I could compile Gentoo for my laptop, using my main computer, if my main computer were already running Gentoo? Is there a link for how to do this? I have a couple of P-III/512Mb-RAM laptops I'd like to resurrect. |
ElleStone ... yes, it a package for distributed compilations ... as Laurant noted, you will need to specify the -march=. See the Gentoo Distcc Documentation.
| ElleStone wrote: | | Khay, what is the recommended "-j2 -l 1.75" '-l' value? I didn't see anything about '-l' in the installation guide. |
There isn't a recommended value that I know of, as I stated above I under utilise the cpu with MAKEOPTS so that my GPU (which is ... ummm ... crappy .... shared graphics) remains responsive while building. You might look at this thread and this blogpost for some explanation of how --load-average works. As I stated with 2 cpu's '2.00' would be the maximal value.
| ElleStone wrote: | | khayyam wrote: | | when you say "compiled from git" I don't know if you mean *-9999 (or what are sometimes called 'live ebuilds') or if you mean 'git clone git://' and compiled on the system. |
By "compiled from git" I mean "git clone git://". |
I see ... in which case it's the same as any other distribution in that regard, portage is not involved, CFLAGS are provided by you at buildtime. You should keep anything installed via this method segregated from package installed software in /usr by using /usr/local.
| ElleStone wrote: | | Sorry! I wasn't being clear enough - the claim that '-O3 etc' was safer than '-Ofast' was only with respect to installing atk/glib/pango/cairo/babl/gegl/gimp. I'm not sure how much testing/benchmarking was done, and some of the testing was on a windows system. I'm happy with -O2 (stability is good!), except perhaps with babl/gegl/gimp and other imaging software, as image processing software by definition is math intensive. This link: http://openbenchmarking.org/result/1210138-RA-GCCAMDBUL77 convinced me that -O3/-Ofast isn't always faster than -O2. But often, for image processing, it's faster. |
Again, it somewhat depends on the version of gcc ... gentoo's gcc is old by some standards, as it has to be able to compile the whole tree, and dependably so. If you wanted to take a more experimental appraoch you could use ~arch gcc ... but again, you have to be able to cope with any possible issues, particularly if its used as the default gcc. So, with a more recent gcc, and using -O3 -ffast-math etc, some improvement may be found, but I can't really say for sure as its not something I've tried. You might want to look into LTO and this thread. As eyoung100 suggtested the 'graphite' useflag might also improve things ... there are numerious avenues you might take to improve speed.
I should note in passing that for me using gentoo is not simply about optimisation, its more about the other *features*, and though some level of optimisation can be achieved I would play down the oft heard claims that gentoo is just ... well ... *faster*.
HTH & best ... khay |
|
| Back to top |
|
|
cach0rr0
Bodhisattva
Joined: 13 Nov 2008
Posts: 4123
Location: Houston, Republic of Texas
|
| Posted: Thu Mar 21, 2013 7:10 am Post subject: Re: -O#, "-pipe", time to build packages on older |
|
|
| ElleStone wrote: | Before installing Gentoo on my main computer I have some general questions.
The Gentoo Installation Guide says don't use "-pipe" if the system doesn't have enough RAM. My Acer laptop has 1GB of ram. Should I use "-pipe"?
Building software on my Acer laptop is ***very*** slow, I mean really ridiculously slow. The 32-bit Acer laptop has 1 Intel dual-core (T2080) 1.8 GHz cpu and 1 GB ram.
|
way back when, i think '03 or '04, i remember doing a gentoo install on an Athlon XP 1800+ TBRED-A system with 256MB RAM
I used -pipe back then. At the time mysql was not happy with -pipe, but everything else seemed just fine.
Things seem to be heavier nowadays, but i would imagine -pipe would still be liveable. Back then it was basically, fire off an emerge of X, go to sleep, wake up, hope it built, if not fix the error and fire it off before work, ssh in from work, check status. Then fire off xfce4 build, hope for no error, let sit overnight. And those were your time sucks, nothing else seemed to be too bad. All told everything built and configured took a good week (going from stage1). Things have of course changed, but not so much that 1GB RAM on a 32bit single-core would be unbearable (though you will not likely be upgrading your browser every night!)
it will not take you this long. But larger things like chromium, firefox, thunderbird, libreoffice, will be tedious to build if going with the source package.
_________________
Lost configuring your system?
dump lspci -n here | see Pappy's guide | Link Stash |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9708
Location: almost Mile High in the USA
|
| Posted: Thu Mar 21, 2013 4:29 pm Post subject: |
|
|
When linking firefox, it will take more than 1GB of virtual memory. Prepare to swap. -pipe doesn't help link as far as I know.
I still build on my eeePC netbook, which uses a single core (dual thread) 1.6GHz Atom. I find it about the speed of a 1.3GHz P3 and maybe slower if it's running certain tasks. I don't recall it taking 2 hours to build gcc (it has 2GB RAM however.)...
And yes I've used -pipe since forever... It's purely a speed gain, unless you can't keep both gcc and the assembler in RAM at the same time. All modern machines can do this (I'd say anything with more than 128MB or so)
I'd look into some other issues with the laptop if it's really "ridiculously" slow. That is, unless you're comparing it to an i7 or something...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
ElleStone
Tux's lil' helper
Joined: 12 Mar 2013
Posts: 106
|
| Posted: Fri Mar 22, 2013 1:10 pm Post subject: |
|
|
Thanks very much to all of you for responding to my questions. Having gotten the '-pipe' and '-j2' straightened out, the laptop compiles at what seems to me to be a very reasonable speed, considerably faster than before.
| Quote: | | for me using gentoo is not simply about optimisation, its more about the other *features*, and though some level of optimisation can be achieved I would play down the oft heard claims that gentoo is just ... well ... *faster*. |
How do you get the author's name in a quote, as "khayyam wrote"? Anyway, faster is one reason I'm switching to Gentoo. Another is more control over what gets installed. I usually have a terminal open running htop or top to keep track of what's doing what, and I'm tired of seeing consolekit in my list of running processes.
I've been reading up on graphite and LTO and I will try both out on the laptop, after upgrading gcc to 4.7.2. I'm sure I'll have questions, but I'll post them in a different forum as upgrading gcc isn't really an "install" question.
So much to learn!
Elle |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Fri Mar 22, 2013 2:00 pm Post subject: |
|
|
| ElleStone wrote: | | Quote: | | for me using gentoo is not simply about optimisation, its more about the other *features*, and though some level of optimisation can be achieved I would play down the oft heard claims that gentoo is just ... well ... *faster*. |
How do you get the author's name in a quote, as "khayyam wrote"? |
ElleStone ... you use quote="khayyam" within square braces, and end the quote block with '/quote' (also within square braces). This should happen automatically if you use 'quote' (top right of post) rather than 'post reply' (bottom left of thread).
| ElleStone wrote: | | Anyway, faster is one reason I'm switching to Gentoo. Another is more control over what gets installed. |
Targeted builds are generally faster, but it all really comes down to how well gcc can optimise the code. As the stable gcc tends to have stricter Q&A it can lag behind upstream releases, with other distributions having more recent version as the default. None the less, gcc is "slotted" and so multiple versions can be installed simultaniously.
| ElleStone wrote: | | I've been reading up on graphite and LTO and I will try both out on the laptop, after upgrading gcc to 4.7.2. I'm sure I'll have questions, but I'll post them in a different forum as upgrading gcc isn't really an "install" question. |
Well, some are already testing LTO with 4.8.x (as in the above linked thread) ... note however that some packages won't build and so you need to use /etc/portage/env to filter these packages ...
best ... khay |
|
| Back to top |
|
|
cwr
Veteran
Joined: 17 Dec 2005
Posts: 1969
|
| Posted: Sun Mar 24, 2013 1:44 pm Post subject: |
|
|
FWIW, here is a summary of some build times, building from scratch using
a current Portage snapshot and Stage 3 files in 2011. The Asus is a
2.2 GHz / 2 GB Dual Core desktop, and the T23 a 1.1 GHz / 512 MB PIII laptop.
X11 is the packages needed to get an X11 desktop, Gnome is the packages needed
to get a Gnome desktop, and World is the "standard" bunch of packages I
generally use.
Summary of build times:
| Code: |
Asus 16:45 17:53 X11
17:25 22:28 Gnome
0:40 4:35 World
T23 12:12 15:00 X11
14:21 06:25 Gnome
2:09 15:25 World
|
Obviously such times aren't especially consistent, especially since HD speed isn't taken
into account, but they give a starting point
Will |
|
| Back to top |
|
|
Mr.EVIL
n00b
Joined: 22 Mar 2013
Posts: 10
|
| Posted: Sun Mar 24, 2013 3:31 pm Post subject: |
|
|
how did you mount /tmp ?
also I will recommend use -Os instead -O2
and if you workstantion with two dualcore processors you must use -j4 or even -j5 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Installing Gentoo
