Big increase in compile times for new versions of software

Message

linux_matt · Post by **linux_matt** » Sun Jun 07, 2015 9:45 am

I have noticed large increases in compile times for some software. Below is the output of genlop -t for webkit-gtk. Is this to be expected with the increasing complexity of software or is it due to system age/performance (this is fairly old single-processor AMD system).

GCC compile flags are: CFLAGS="-march=k8-sse3 -O2 -pipe -fstack-protector-strong -mmmx -m3dnow -msse -msse2 -mssse3 -msahf --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=1024"

Processor is: AMD Athlon(tm) 64 Processor 4000+

Fri May 25 03:53:27 2012 >>> net-libs/webkit-gtk-1.6.3-r200
merge time: 1 hour, 20 minutes and 6 seconds.

Mon Jul 16 00:15:47 2012 >>> net-libs/webkit-gtk-1.8.1-r301
merge time: 2 hours, 43 minutes and 6 seconds.

Sat Dec 29 08:55:04 2012 >>> net-libs/webkit-gtk-1.10.2-r300
merge time: 5 hours, 3 minutes and 9 seconds.

Sat Dec 29 13:08:01 2012 >>> net-libs/webkit-gtk-1.8.3-r200
merge time: 4 hours, 12 minutes and 57 seconds.

Mon Nov 25 04:54:32 2013 >>> net-libs/webkit-gtk-2.0.4
merge time: 5 hours, 36 minutes and 45 seconds.

Mon Dec 23 19:14:04 2013 >>> net-libs/webkit-gtk-1.8.3-r201
merge time: 2 hours, 7 minutes and 37 seconds.

Thu Jan 30 04:58:51 2014 >>> net-libs/webkit-gtk-2.0.4
merge time: 4 hours, 19 minutes.

Sun Mar 9 06:16:44 2014 >>> net-libs/webkit-gtk-2.2.5
merge time: 4 hours, 43 minutes and 12 seconds.

Sun Mar 9 10:23:40 2014 >>> net-libs/webkit-gtk-2.2.5-r200
merge time: 3 hours, 58 minutes and 53 seconds.

Mon Aug 25 09:29:54 2014 >>> net-libs/webkit-gtk-2.4.4
merge time: 8 hours, 24 minutes and 19 seconds.

Mon Feb 2 01:54:02 2015 >>> net-libs/webkit-gtk-2.4.8-r200
merge time: 6 hours, 54 minutes and 23 seconds.

Mon Feb 2 11:25:42 2015 >>> net-libs/webkit-gtk-2.4.8
merge time: 8 hours, 35 minutes and 52 seconds.

Mon Feb 2 20:15:31 2015 >>> net-libs/webkit-gtk-2.6.5
merge time: 8 hours, 19 minutes and 34 seconds.

Sun Jun 7 09:22:58 2015 >>> net-libs/webkit-gtk-2.6.6
merge time: 16 hours, 26 minutes and 54 seconds.

krinn · Post by **krinn** » Sun Jun 07, 2015 12:19 pm

I'm afraid your answer is : both.

That's webkit-gtk-2.4.9 with a model name : Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz
Sat Jun 6 12:23:15 2015 >>> net-libs/webkit-gtk-2.4.9
merge time: 3 hours, 8 minutes and 58 seconds.
Not really the oldest cpu you can find, but it show its age already, but the dual core makes really a diff.

But seeing my other (the daily usage one) computer has done, you can also tell the complexity increase.

franzf · Post by **franzf** » Sun Jun 07, 2015 12:47 pm

The three steps to decrease compile time for webkit-gtk:
* Use clang
* Use ld.gold
* set CMAKE_MAKEFILE_GENERATOR="ninja" in your make.conf - you need to install dev-util/ninja for this.
Especially the last point reduced build time by 30 minutes (!!) on my system. ld.gold uses less memory (no swapping, hence faster) and also seems to be faster in general. clang compiles faster on my system than gcc-4.8 and 4.9 - 4.7 was just fine.

padoor · Post by **padoor** » Sun Jun 07, 2015 2:21 pm

Code: Select all

 Mon Oct 22 14:05:17 2012 >>> net-libs/webkit-gtk-1.8.3-r200
       merge time: 2 hours, 57 minutes and 47 seconds.

     Thu Jun 27 15:26:22 2013 >>> net-libs/webkit-gtk-1.8.3-r201
       merge time: 2 hours, 40 minutes and 31 seconds.

     Sun Jan  5 14:07:35 2014 >>> net-libs/webkit-gtk-1.8.3-r201
       merge time: 3 hours, 26 minutes and 17 seconds.

     Mon Jan 27 16:14:23 2014 >>> net-libs/webkit-gtk-1.8.3-r201
       merge time: 3 hours, 2 minutes and 18 seconds.

     Fri Apr 18 13:19:50 2014 >>> net-libs/webkit-gtk-2.2.6-r200
       merge time: 4 hours, 47 minutes and 51 seconds.

     Fri May 30 18:48:00 2014 >>> net-libs/webkit-gtk-2.2.6-r200
       merge time: 4 hours, 33 minutes and 21 seconds.

     Tue Jun  2 18:10:47 2015 >>> net-libs/webkit-gtk-2.4.8-r200
       merge time: 7 hours, 4 minutes and 3 seconds.

not only webkit-gtk almost all emerges have gone atleast 50% more time for merging if not double the older time
i thought it is because of multilib installation
seems some thing from portage or use flags

Post by **John R. Graham** » Sun Jun 07, 2015 2:24 pm

franzf wrote:The three steps to decrease compile time for webkit-gtk:
* Use clang
* Use ld.gold
* set CMAKE_MAKEFILE_GENERATOR="ninja" in your make.conf - you need to install dev-util/ninja for this.
Especially the last point reduced build time by 30 minutes (!!) on my system. ld.gold uses less memory (no swapping, hence faster) and also seems to be faster in general. clang compiles faster on my system than gcc-4.8 and 4.9 - 4.7 was just fine.

There are trade-offs, though. Last I checked, clang-compiled code on average benchmarked slower than gcc-compiled code. ld is indeed a memory hog; however, if you have 4GiB or more of memory, my intuition is that you won't see a big advantage with Gold: after all, compile time dominates the build time.

Thanks for the tip on Ninja. I'm going to have to check that out.

- John

linux_matt · Post by **linux_matt** » Sun Jun 07, 2015 3:10 pm

Thanks to all for the useful feedback

John R. Graham wrote:
franzf wrote:The three steps to decrease compile time for webkit-gtk:
* Use clang
* Use ld.gold
* set CMAKE_MAKEFILE_GENERATOR="ninja" in your make.conf - you need to install dev-util/ninja for this.
Especially the last point reduced build time by 30 minutes (!!) on my system. ld.gold uses less memory (no swapping, hence faster) and also seems to be faster in general. clang compiles faster on my system than gcc-4.8 and 4.9 - 4.7 was just fine.
There are trade-offs, though. Last I checked, clang-compiled code on average benchmarked slower than gcc-compiled code. ld is indeed a memory hog; however, if you have 4GiB or more of memory, my intuition is that you won't see a big advantage with Gold: after all, compile time dominates the build time.

Thanks for the tip on Ninja. I'm going to have to check that out.

- John

Post by **toralf** » Sun Jun 07, 2015 3:30 pm

franzf wrote:Especially the last point reduced build time by 30 minutes (!!) on my system.

How long does it need before ?

franzf · Post by **franzf** » Sun Jun 07, 2015 5:50 pm

toralf wrote:
franzf wrote:Especially the last point reduced build time by 30 minutes (!!) on my system.
How long does it need before ?

by not to 30 minutes

(reduziert UM 30 Minuten)
I now need 43 minutes (mostly thx to distcc

), before that it were ~43 + 30 = 73 minutes. Those 30 minutes really just came from calculating dependencies - cmake spun @100% doing - don't know

@ John R. Graham: Yeah, clang is not THAT fast. If you believe benchmarks it might be 5-10% depending on what you compile. I have other problems with gcc I can't track down: compiling certain c++ files takes great amounts of RAM. It gets unusable when boost is involved: one process can occupy 80++ % of RAM - I have 4GB. Result: Excessive swapping. I see that in webkit, too. As I mentioned: On MY system, with gcc-4.8 and 4.9, 4.7 didn't show that behaviour and also was faster.
ld: Even the webkit-gtk-DEVs say that 4GB isn't enough for using ld.bfd to link webkit. I currently can't remember if the minimum was 8 or 16 GB. With ld.gold I really don't notice that some (potentially) heavy linking is going on.
Of course you are right: If you don't have any resource (RAM) constraints you won't notice such big differences - probably besides ninja vs emake. But I really can't see a reason for those enormous jumps - ~4 hours to ~16 hours! I didn't see such compile times here, that's why I think on those systems ARE some constraints and ld (+clang) could be the solution.

krinn · Post by **krinn** » Sun Jun 07, 2015 5:57 pm

that might be because you're using x86-64 my poor core2 have 2g and it has build it just fine (without distcc help). if you have <4g running it x86-64 is not helping it and it would be better to use x86 then.
(that might be a 3rd choice for op, dropping 64bits on its cpu)
op: remember also -pipe, while it rocks if you have ram, when low of it, it's a bad idea to use it.

Post by **toralf** » Sun Jun 07, 2015 5:58 pm

16GB sounds too much IMO - I spent 10 GB tmpfs together for my all of my tinderbox chroot images which usually was enough for webkit-gtk (except corner cases)

franzf · Post by **franzf** » Sun Jun 07, 2015 6:17 pm

@krinn: I actually have 4GB RAM available (minus what my intel iGPU needs). 64bit runs fine, and as long as ld.gold+clang stay with that RAM usage I can't see why I should restrict myself to 32bit.

@toralf: how much RAM is left after putting your chroot into the tmpfs?

Post by **toralf** » Sun Jun 07, 2015 8:28 pm

franzf wrote:@toralf: how much RAM is left after putting your chroot into the tmpfs?

I don't put the whole chroot image into tmpfs, just /var/tmp/portage is mounted onto tmpfs. Furthermore in the past I set the max size to 10 GB for 4 independent chroot images - hoped, that not all would need it at the same time (unfortunately that wasn't the case for about 2 times within last few weeks, resulted in the moronic-question-of-the-week in this forum BTW)
Currently I spend each of the the chroot images an own tmpfs with a size of up to 8 GB - at a 16 GB system - works fine so far.

cwr · Post by **cwr** » Sun Jun 07, 2015 8:55 pm

As a datapoint, an Intel i686 Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz takes an hour to compile
a 32-bit webkit-gtk-1.8.3. It looks as if webkit has got a lot more complicated recently.

Will

Princess Nell · Post by **Princess Nell** » Sun Jun 07, 2015 9:53 pm

I masked >1.8.3-r300 a good while back because I don't want ruby on my system. In particular if the only purpose of its presence is to support the build system of webkit-gtk.

padoor · Post by **padoor** » Tue Jun 09, 2015 2:40 am

i have tried using ninja gold and clang yesterday hoping it would reduce te compile time of llvm but i ended up using 3 times long merge.

Code: Select all

ramaswamy@tux ~ $ genlop -t llvm
 * sys-devel/llvm

     Thu Oct 18 12:40:45 2012 >>> sys-devel/llvm-3.0-r2
       merge time: 31 minutes and 18 seconds.

     Sat Nov 17 11:17:04 2012 >>> sys-devel/llvm-3.1-r2
       merge time: 43 minutes and 51 seconds.

     Fri Mar 28 10:54:26 2014 >>> sys-devel/llvm-3.3-r3
       merge time: 46 minutes and 8 seconds.

     Wed May 27 18:57:45 2015 >>> sys-devel/llvm-3.5.0
       merge time: 2 hours, 17 minutes and 52 seconds.

     Tue Jun  9 00:59:01 2015 >>> sys-devel/llvm-3.5.0
       merge time: 7 hours, 22 minutes and 36 seconds.

and to my disappointment todays update wants to merge it again with -clang -gold
i could not find anything about trouble with use of ninja
wasted 5 hrs just 6 hrs before on merging llvm and probably another 7 hours to day for same purpose

krinn · Post by **krinn** » Tue Jun 09, 2015 10:41 am

padoor: don't you think it would just be worther to waste the 10 minutes you need to learn portage?
-k and package.use could had saved you, but that's your time...

padoor · Post by **padoor** » Tue Jun 09, 2015 11:55 am

krinn wrote:padoor: don't you think it would just be worther to waste the 10 minutes you need to learn portage?
-k and package.use could had saved you, but that's your time...

i did not understand -k and package.use
some link you can give me to read up on it
excuse my ignorance

krinn · Post by **krinn** » Tue Jun 09, 2015 12:04 pm

Code: Select all

man emerge
man portage
man make.conf

steveL · Post by **steveL** » Tue Jun 09, 2015 12:25 pm

linux_matt wrote:I have noticed large increases in compile times for some software. Below is the output of genlop -t for webkit-gtk. Is this to be expected with the increasing complexity of software or is it due to system age/performance (this is fairly old single-processor AMD system).

make-4 apparently slows things down a lot. (the new one with GUILE built-in; cos what every shellscripter wants is a LISP..;)

kernelOfTruth · Post by **kernelOfTruth** » Tue Jun 09, 2015 10:41 pm

How about ccache or icecream ?

No one mentioned it yet

krinn · Post by **krinn** » Wed Jun 10, 2015 2:01 am

Because there are solves that are good for some user and some that aren't good for some other.
i really don't think padoor is the kind of user that should use ccache ; it won't help him, it would made it worst.

mv · Post by mv » Wed Jun 10, 2015 2:16 am

krinn wrote:i really don't think padoor is the kind of user that should use ccache ; it won't help him, it would made it worst.

I don't agree: The recompilation with changed USE-flags has quite some probability to be much faster with ccache.

mv · Post by mv » Wed Jun 10, 2015 2:24 am

steveL wrote:(the new one with GUILE built-in; cos what every shellscripter wants is a LISP..;)

Although I don't know why the possibility to use GUILE should slows things down, I never understood why GNU people are so crazy about LISP. It was a horrible idea to use it for an editor, it is an even worse idea for a make system. I fail to see any rational reasons for this choice.

krinn · Post by **krinn** » Wed Jun 10, 2015 2:52 am

mv wrote:
krinn wrote:i really don't think padoor is the kind of user that should use ccache ; it won't help him, it would made it worst.
I don't agree: The recompilation with changed USE-flags has quite some probability to be much faster with ccache.

Where did i said ccache won't faster it? That's not what i'm speaking of.

franzf · Post by **franzf** » Wed Jun 10, 2015 4:43 am

I used ccache in the past as I wanted to speedup kde updates. It didn't really help, mostly because one of the central includes in kde was version.h, and it changed with every release. So only Gentoo rev bumps were built faster. Don't know if that has changed and how other projects handle that.

Big increase in compile times for new versions of software

Big increase in compile times for new versions of software

Re: Big increase in compile times for new versions of softwa

Re: Big increase in compile times for new versions of softwa