Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Save 30-50% off Chromium compile time with one simple trick!
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4707

PostPosted: Tue Jan 02, 2018 2:19 pm    Post subject: Save 30-50% off Chromium compile time with one simple trick! Reply with quote

</clickbait>

It's real though!

The build system in newer versions of chromium has support for source file concatenation, like what sqlite3 does (or to a lesser extent, the USE=pch flag). It's not enabled by default, but you can try it out yourself by doing this:
/etc/portage/package.env:
www-client/chromium chromium.conf

/etc/portage/env/chromium.conf:
EXTRA_GN="use_jumbo_build=true jumbo_file_merge_limit=25"

Feel free to increase the number if you have the RAM to handle it - 25 is just a safe number I picked because it worked on my 2GB netbook. The default is 100, which has some gcc processes eating 2.5GB or more.

The improvements can range from several hours to half a day or more -
Code:
$ qlop -vgH chromium
chromium-63.0.3239.59: Tue Nov 28 11:35:50 2017: 7 hours, 29 minutes, 8 seconds
chromium-64.0.3282.24: Fri Dec 22 04:12:14 2017: 5 hours, 57 minutes, 19 seconds

$ lscpu | grep P
CPU op-mode(s):      32-bit, 64-bit
CPU(s):              4
On-line CPU(s) list: 0-3
CPU family:          16
Model name:          AMD Phenom(tm) II X4 20 Processor
CPU MHz:             800.000
CPU max MHz:         2800.0000
CPU min MHz:         800.0000
BogoMIPS:            5625.36


Code:
$ qlop -vgH chromium
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-62.0.3202.18: Sat Sep 30 00:34:10 2017: 1 day, 2 hours, 50 minutes, 43 seconds
chromium-64.0.3282.24: Fri Dec 22 19:10:06 2017: 21 hours, 18 minutes, 46 seconds
# (not pictured: a regular compile of 64.0 that OOMed at the linker stage after 36 hours)

$ lscpu | grep P
CPU op-mode(s):      32-bit
CPU(s):              2
On-line CPU(s) list: 0,1
CPU family:          6
Model name:          Intel(R) Atom(TM) CPU N270   @ 1.60GHz
CPU MHz:             800.000
CPU max MHz:         1600.0000
CPU min MHz:         800.0000
BogoMIPS:            3193.13


See also: bug 641786
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Juippisi
Apprentice
Apprentice


Joined: 30 Sep 2005
Posts: 197
Location: /home

PostPosted: Tue Jan 02, 2018 2:29 pm    Post subject: Reply with quote

Thanks for bringing this out in the open, I wouldnt have found that bug report on my own. This seems to work, and is a huge improvement in compiling Chromium.
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1274

PostPosted: Wed Jan 03, 2018 6:04 am    Post subject: Reply with quote

Yes it works.
At 100 it crashed but 75 gives:
Code:

 # genlop -t chromium
 * www-client/chromium

     Wed Dec 20 08:13:44 2017 >>> www-client/chromium-64.0.3282.24
       merge time: 4 hours, 17 minutes and 54 seconds.

     Mon Dec 25 02:38:39 2017 >>> www-client/chromium-64.0.3282.39
       merge time: 4 hours, 22 minutes and 55 seconds.

     Wed Jan  3 02:22:42 2018 >>> www-client/chromium-64.0.3282.39
       merge time: 1 hour, 59 minutes and 58 seconds.


Can you explain to me in simple words how this works?

Ma

EDIT
Is it possible to do that with other programs?
Or in other words. Can it be used as a standard for everything?
_________________
Thks 4 ur attention, interest & support.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4707

PostPosted: Thu Jan 04, 2018 9:05 am    Post subject: Reply with quote

It concatenates 25/75/100 source files together at a time and compiles the result with a single GCC process. Apart from saving 99% of the overhead of starting up the compiler, it also means header files with multiple-include guards won't have to be processed over and over, and a lot more inter-function and dead code optimisation can be done at compile-time (the stuff usually reserved for -flto). Probably the latter thing is where most of the RAM usage comes from, but it also takes some off the final linking step.

It's hardcoded in chromium's build scripts, and is only possible because they already rewrote what autoconf/make does from scratch and control the entire compilation process. Making it work anywhere else would be pretty hard, definitely not something you can throw in make.conf and forget about.
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1274

PostPosted: Thu Jan 04, 2018 9:21 am    Post subject: Reply with quote

Thanks Ant. P. I wanted to know that.
This tip is a fantastic time saver.

Thanks again
Ma
_________________
Thks 4 ur attention, interest & support.
Back to top
View user's profile Send private message
wxqc11h
n00b
n00b


Joined: 18 Sep 2007
Posts: 24

PostPosted: Tue Jan 23, 2018 11:32 am    Post subject: Reply with quote

Does it work for webkit-gtk too? Man merging it takes ages.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4707

PostPosted: Wed Jan 24, 2018 6:01 am    Post subject: Reply with quote

Nope, chromium-only. I don't recall Webkit-GTK having compile times as dire as this last time I had it installed though... was closer to firefox (minus rust) if anything.
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Princess Nell
l33t
l33t


Joined: 15 Apr 2005
Posts: 721

PostPosted: Sun Jan 28, 2018 9:09 pm    Post subject: Reply with quote

Off-topic here, but qlop must have serious bugs. Looking at emerge.log, genlop is quite correct.
Code:

# genlop -t webkit-gtk |tail -n 3
     Tue Jan 16 00:04:25 2018 >>> net-libs/webkit-gtk-2.18.5
       merge time: 2 hours, 40 minutes and 40 seconds.

# qlop -vgH webkit-gtk |tail -n 3
webkit-gtk-2.18.4: Thu Dec 28 22:31:26 2017: 2 hours, 39 minutes, 55 seconds
webkit-gtk-2.18.5: Sun Jan 14 18:22:37 2018: 1 day, 5 hours, 41 minutes, 48 seconds
webkit-gtk: 19 times
#
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4707

PostPosted: Sun Jan 28, 2018 11:37 pm    Post subject: Reply with quote

Yeah, I noticed the same. It works if you give it a very narrow date range but seems to drop recent entries otherwise.
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Princess Nell
l33t
l33t


Joined: 15 Apr 2005
Posts: 721

PostPosted: Wed Jan 31, 2018 7:59 am    Post subject: Reply with quote

Impressive. jumbo_file_merge_limit=60 (100/90/80/70 crashed).
Code:

     Tue Jan 30 06:08:52 2018 >>> www-client/chromium-64.0.3282.123
       merge time: 8 hours, 20 minutes and 21 seconds.

     Wed Jan 31 03:18:11 2018 >>> www-client/chromium-64.0.3282.123
       merge time: 4 hours, 7 minutes and 32 seconds.
Back to top
View user's profile Send private message
walkingcorpse
n00b
n00b


Joined: 24 Apr 2017
Posts: 10
Location: Europe

PostPosted: Fri Feb 09, 2018 7:04 pm    Post subject: Reply with quote

jumbo_file_merge_limit=10 8GB ram and -j4. :?

Code:
     Sat Jan 20 20:30:37 2018 >>> www-client/chromium-65.0.3315.3
       merge time: 4 hours, 10 minutes and 39 seconds.

     Fri Feb  9 05:26:53 2018 >>> www-client/chromium-65.0.3315.3
       merge time: 2 hours, 5 minutes and 23 seconds


About the jumbo_file_merge_limit from the Chromium-dev google forum:

Quote:
Eventually we could have 10 or 20 for people with many cores, 50 for people with few cores, 100 or infinity for single core/full compilation.
Back to top
View user's profile Send private message
Section_8
Guru
Guru


Joined: 22 May 2004
Posts: 552
Location: Arlington, TX, US

PostPosted: Fri Feb 09, 2018 9:02 pm    Post subject: Reply with quote

I noticed the chromium ebuild has a jumbo-build use flag that must implement this:
Code:
john@officepc ~ $ grep jumbo `equery which chromium`
IUSE="component-build cups gnome-keyring +hangouts jumbo-build kerberos neon pic +proprietary-codecs pulseaudio selinux +suid +system-ffmpeg +system-icu +system-libvpx +tcmalloc widevine"
        # https://chromium.googlesource.com/chromium/src/+/lkcr/docs/jumbo.md
        myconf_gn+=" use_jumbo_build=$(usex jumbo-build true false)"

I wonder what file merge limit it uses.
Back to top
View user's profile Send private message
walkingcorpse
n00b
n00b


Joined: 24 Apr 2017
Posts: 10
Location: Europe

PostPosted: Fri Feb 09, 2018 9:12 pm    Post subject: Reply with quote

It should be 200:
Quote:
By default at most 200 files are merged at a time. The more files are merged, the less total CPU time is needed, but parallelism is reduced. This can be changed by setting jumbo_file_merge_limit to something else than 200.

https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum