Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Chromium is taking a long time to compile
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Zucca
Moderator
Moderator


Joined: 14 Jun 2007
Posts: 3345
Location: Rasi, Finland

PostPosted: Sat Oct 27, 2018 7:52 pm    Post subject: Reply with quote

It's sad to see the most complicated (and bloated?) programs are web browsers.

I used to say "You can install Linux on that old PC of yours and use it for office programs and web surfing.".

To surf the web one used to need only a light surf board. Nowdays one needs an aircraft carrier.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote:
I am NaN! I am a man!
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Sun Oct 28, 2018 7:45 am    Post subject: Reply with quote

nowhereman wrote:
I also tried jumbo-build, but it didn’t really help much either.

Hello nowhereman,
On my Intel Core 2 duo E6300 1,86GHz and 6 GB RAM, values of file merge limit near 10 makes a big difference. It brings the build time from 37-something hours to 8-9. Yet the problem is the system is mostly unusable. It used to be around 30% of the non-jumboo build, but it increased over time to nearly 40%. Jumboo build is vulnerable to out of memory. If your RAM is limited, use smaller values for file merge limit. I used to use 25 but it started breaking at one moment.
There's a link to how you can configure your build environment for chrome anove, similar to what I've done.

What I've done is to mask those two packages to a certain version, so that I can build them every few months. There were some tips to tweak ccache given above by user mv, but I don't feel like spending the time to test it. I'm just trying to get by till I get my next job and buy a new i7 laptop.

Zucca wrote:

I used to say "You can install Linux on that old PC of yours and use it for office programs and web surfing.".


Hello Zucca,
My computer is more than 10 years old. I remember buying it at the end of 2005. Since then I've replaced the Pentium4 630 processor and the motherboard, but they are now no-newer than 2007 make. Then when I had to return back to using it at the end of 2017, I upgraded the memory from 3 to 6 GB. It's just not convenient to compile on it, but it still does a very good job.
But let's not go the off-topic path.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Sun Nov 18, 2018 8:24 am    Post subject: Reply with quote

Building of Chromium 70.0.3538.67 with file merge limit of 15 on 6 gigs of RAM and 2 cores went well. I remember trying with 25 but it didn't go well. This value depends on RAM, but also varies from build to build. I think Chromium 69 didn't go well with 15, but it's nearly that value for 6 gigs of RAM.

It took like 12-13 hours, but for you that depends on your CPU. Most people have faster CPU's than mine, so I guess everybody will get better compile times with a few exceptions.

If you have 4 gigs of RAM, per core I think it's safely to go for file merge limit of 10 per core or, if you have some spare hours to experiment and don't mind to lose them due to out of memory errors, test values 15 or 20. If the build succeeds, that will spare you at least 10-20% of the time.

Regards,
Georgi
Back to top
View user's profile Send private message
gnumber9
n00b
n00b


Joined: 25 Jan 2014
Posts: 4

PostPosted: Mon Aug 05, 2019 1:23 am    Post subject: Chromium isn't worth this. Reply with quote

On my i7-950, all cores maxed at 100% running around 3075MHz, using up to 8GB at times out of 24GB, makopts at -j8, SSD. Chromium update is now around 4 hours of compile time.

Debating next time to either:
emerge -avuDN @world --exclude=chromium
or to
emerge --depclean chromium
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1591

PostPosted: Sat Dec 21, 2019 12:10 pm    Post subject: Reply with quote

Hi there,

does anyone know what has changed since version chromium-79.0.3945.74. ? Since then the compilation time has increased drastically.
Code:

genlop -t chromium
 * www-client/chromium

     Sun May 13 12:33:27 2018 >>> www-client/chromium-67.0.3396.40
       merge time: 1 hour, 9 minutes and 59 seconds.

     Fri May 18 18:31:27 2018 >>> www-client/chromium-67.0.3396.40
       merge time: 1 hour, 8 minutes and 54 seconds.

     Wed May 30 01:24:44 2018 >>> www-client/chromium-67.0.3396.56
       merge time: 1 hour, 10 minutes and 55 seconds.

     Thu May 31 18:23:58 2018 >>> www-client/chromium-67.0.3396.62
       merge time: 1 hour, 14 minutes and 4 seconds.

     Sun Jun 10 14:35:42 2018 >>> www-client/chromium-67.0.3396.79
       merge time: 1 hour, 13 minutes and 38 seconds.

     Sat Jun 16 02:37:46 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 1 hour, 13 minutes and 12 seconds.

     Thu Jun 21 01:23:27 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 1 hour, 8 minutes and 6 seconds.

     Fri Jun 29 14:13:18 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 1 hour, 15 minutes and 18 seconds.

     Fri Jul  6 12:24:48 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 1 hour, 15 minutes and 4 seconds.

     Mon Jul  9 23:10:23 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 1 hour, 17 minutes and 9 seconds.

     Fri Jul 13 20:06:35 2018 >>> www-client/chromium-68.0.3440.25
       merge time: 2 hours, 45 minutes and 18 seconds.

     Mon Jul 23 05:45:37 2018 >>> www-client/chromium-68.0.3440.68
       merge time: 1 hour, 14 minutes and 50 seconds.

     Mon Jul 30 11:18:01 2018 >>> www-client/chromium-68.0.3440.75
       merge time: 1 hour, 26 minutes and 24 seconds.

     Mon Aug  6 12:31:47 2018 >>> www-client/chromium-69.0.3497.23
       merge time: 1 hour, 24 minutes and 31 seconds.

     Mon Aug 13 03:11:59 2018 >>> www-client/chromium-69.0.3497.32
       merge time: 1 hour, 14 minutes and 24 seconds.

     Sun Aug 19 09:35:21 2018 >>> www-client/chromium-69.0.3497.42
       merge time: 1 hour, 22 minutes and 12 seconds.

     Tue Aug 28 00:03:34 2018 >>> www-client/chromium-69.0.3497.57
       merge time: 1 hour, 31 minutes and 30 seconds.

     Tue Sep 11 20:01:44 2018 >>> www-client/chromium-69.0.3497.81
       merge time: 1 hour, 31 minutes and 18 seconds.

     Sun Sep 23 12:52:57 2018 >>> www-client/chromium-70.0.3538.22
       merge time: 1 hour, 16 minutes and 22 seconds.

     Mon Sep 24 18:55:38 2018 >>> www-client/chromium-70.0.3538.22
       merge time: 1 hour, 11 minutes and 56 seconds.

     Mon Oct  1 05:01:12 2018 >>> www-client/chromium-70.0.3538.35
       merge time: 1 hour, 21 minutes and 47 seconds.

     Mon Oct  1 17:01:23 2018 >>> www-client/chromium-70.0.3538.35
       merge time: 1 hour, 23 minutes and 8 seconds.

     Mon Oct  8 20:21:30 2018 >>> www-client/chromium-70.0.3538.45
       merge time: 1 hour, 41 minutes and 19 seconds.

     Sun Oct 14 01:58:20 2018 >>> www-client/chromium-70.0.3538.45
       merge time: 1 hour, 10 minutes and 14 seconds.

     Mon Oct 15 09:14:26 2018 >>> www-client/chromium-70.0.3538.54
       merge time: 1 hour, 12 minutes and 23 seconds.

     Fri Oct 19 08:59:15 2018 >>> www-client/chromium-70.0.3538.67
       merge time: 1 hour, 18 minutes and 36 seconds.

     Tue Oct 23 17:09:29 2018 >>> www-client/chromium-70.0.3538.67-r1
       merge time: 1 hour, 19 minutes and 50 seconds.

     Fri Oct 26 02:46:12 2018 >>> www-client/chromium-70.0.3538.67-r1
       merge time: 1 hour, 19 minutes and 40 seconds.

     Fri Oct 26 12:24:33 2018 >>> www-client/chromium-70.0.3538.67-r1
       merge time: 1 hour, 19 minutes and 38 seconds.

     Mon Nov  5 23:16:54 2018 >>> www-client/chromium-71.0.3578.30
       merge time: 1 hour, 33 minutes and 30 seconds.

     Wed Nov 21 12:08:13 2018 >>> www-client/chromium-71.0.3578.53
       merge time: 3 hours, 30 minutes and 44 seconds.

     Mon Nov 26 03:52:22 2018 >>> www-client/chromium-71.0.3578.62
       merge time: 1 hour, 41 minutes and 18 seconds.

     Fri Dec  7 15:04:17 2018 >>> www-client/chromium-71.0.3578.80
       merge time: 1 hour, 25 minutes and 21 seconds.

     Fri Dec 14 04:41:57 2018 >>> www-client/chromium-71.0.3578.98
       merge time: 1 hour, 35 minutes and 46 seconds.

     Fri Dec 28 01:59:49 2018 >>> www-client/chromium-72.0.3626.28
       merge time: 1 hour, 47 minutes and 38 seconds.

     Fri Dec 28 20:19:57 2018 >>> www-client/chromium-72.0.3626.28-r1
       merge time: 1 hour, 26 minutes and 4 seconds.

     Sun Feb 10 15:55:40 2019 >>> www-client/chromium-72.0.3626.96
       merge time: 3 hours, 28 minutes and 42 seconds.

     Thu Feb 14 00:13:32 2019 >>> www-client/chromium-72.0.3626.96
       merge time: 3 hours, 17 minutes and 8 seconds.

     Sat Feb 23 11:58:00 2019 >>> www-client/chromium-73.0.3683.27
       merge time: 1 hour, 24 minutes and 25 seconds.

     Sun Mar 17 13:03:37 2019 >>> www-client/chromium-73.0.3683.75
       merge time: 1 hour, 31 minutes and 22 seconds.

     Sat Mar 23 02:06:39 2019 >>> www-client/chromium-73.0.3683.86
       merge time: 2 hours, 9 minutes and 50 seconds.

     Tue May 28 21:34:55 2019 >>> www-client/chromium-74.0.3729.169
       merge time: 2 hours, 44 minutes and 47 seconds.

     Wed May 29 01:20:18 2019 >>> www-client/chromium-74.0.3729.169
       merge time: 2 hours, 39 minutes and 16 seconds.

     Thu Jun  6 19:31:38 2019 >>> www-client/chromium-74.0.3729.169
       merge time: 2 hours, 23 minutes and 48 seconds.

     Sun Jun 16 19:32:37 2019 >>> www-client/chromium-75.0.3770.90
       merge time: 1 hour, 29 minutes and 38 seconds.

     Tue Jun 18 01:18:41 2019 >>> www-client/chromium-75.0.3770.90
       merge time: 1 hour, 37 minutes and 24 seconds.

     Tue Jun 18 08:03:42 2019 >>> www-client/chromium-75.0.3770.90
       merge time: 1 hour, 33 minutes and 15 seconds.

     Tue Jun 18 23:56:16 2019 >>> www-client/chromium-76.0.3809.25
       merge time: 1 hour, 26 minutes and 26 seconds.

     Fri Jun 21 02:22:17 2019 >>> www-client/chromium-76.0.3809.36
       merge time: 3 hours, 6 minutes and 26 seconds.

     Fri Jun 21 06:06:56 2019 >>> www-client/chromium-76.0.3809.36
       merge time: 1 hour, 14 minutes and 39 seconds.

     Fri Jun 21 09:26:25 2019 >>> www-client/chromium-76.0.3809.36
       merge time: 1 hour, 14 minutes and 5 seconds.

     Fri Jun 21 11:06:18 2019 >>> www-client/chromium-76.0.3809.36
       merge time: 1 hour, 37 minutes and 32 seconds.

     Fri Jun 28 23:13:51 2019 >>> www-client/chromium-76.0.3809.46
       merge time: 1 hour, 41 minutes.

     Fri Jul 12 20:14:07 2019 >>> www-client/chromium-76.0.3809.62
       merge time: 1 hour, 29 minutes and 59 seconds.

     Fri Jul 19 23:45:57 2019 >>> www-client/chromium-76.0.3809.71
       merge time: 1 hour, 19 minutes and 38 seconds.

     Thu Jul 25 20:07:54 2019 >>> www-client/chromium-76.0.3809.80
       merge time: 1 hour, 29 minutes and 34 seconds.

     Wed Jul 31 21:52:18 2019 >>> www-client/chromium-76.0.3809.87
       merge time: 1 hour, 23 minutes and 28 seconds.

     Wed Aug 14 00:43:10 2019 >>> www-client/chromium-76.0.3809.100
       merge time: 1 hour, 18 minutes and 36 seconds.

     Wed Aug 14 21:09:13 2019 >>> www-client/chromium-77.0.3865.19
       merge time: 1 hour, 22 minutes and 24 seconds.

     Fri Aug 16 22:03:38 2019 >>> www-client/chromium-77.0.3865.35
       merge time: 1 hour, 31 minutes and 48 seconds.

     Tue Aug 20 04:41:17 2019 >>> www-client/chromium-77.0.3865.35
       merge time: 2 hours, 23 minutes and 31 seconds.

     Sat Aug 24 12:06:26 2019 >>> www-client/chromium-77.0.3865.42
       merge time: 1 hour, 21 minutes and 50 seconds.

     Tue Aug 27 06:50:00 2019 >>> www-client/chromium-77.0.3865.42
       merge time: 1 hour, 23 minutes.

     Fri Aug 30 23:52:15 2019 >>> www-client/chromium-77.0.3865.56
       merge time: 1 hour, 38 minutes and 14 seconds.

     Sun Sep  8 10:05:43 2019 >>> www-client/chromium-77.0.3865.70
       merge time: 1 hour, 22 minutes and 43 seconds.

     Wed Sep 11 19:56:47 2019 >>> www-client/chromium-77.0.3865.75
       merge time: 1 hour, 28 minutes and 21 seconds.

     Tue Sep 17 23:51:31 2019 >>> www-client/chromium-77.0.3865.75
       merge time: 1 hour, 21 minutes and 3 seconds.

     Thu Sep 26 19:14:56 2019 >>> www-client/chromium-77.0.3865.90
       merge time: 1 hour, 18 minutes and 54 seconds.

     Thu Oct  3 00:43:09 2019 >>> www-client/chromium-78.0.3904.34
       merge time: 1 hour, 30 minutes and 8 seconds.

     Fri Oct  4 21:33:36 2019 >>> www-client/chromium-78.0.3904.34
       merge time: 1 hour, 25 minutes and 41 seconds.

     Sun Oct  6 19:00:20 2019 >>> www-client/chromium-78.0.3904.34
       merge time: 1 hour, 42 minutes and 2 seconds.

     Mon Oct  7 10:04:55 2019 >>> www-client/chromium-78.0.3904.34
       merge time: 1 hour, 48 minutes and 32 seconds.

     Thu Oct 17 01:37:56 2019 >>> www-client/chromium-78.0.3904.50
       merge time: 1 hour, 35 minutes and 44 seconds.

     Mon Oct 21 12:58:06 2019 >>> www-client/chromium-78.0.3904.63
       merge time: 1 hour, 21 minutes and 23 seconds.

     Fri Oct 25 18:05:30 2019 >>> www-client/chromium-78.0.3904.70
       merge time: 1 hour, 39 minutes.

     Sat Nov  2 02:08:11 2019 >>> www-client/chromium-78.0.3904.87
       merge time: 1 hour, 34 minutes and 33 seconds.

     Fri Nov  8 01:36:42 2019 >>> www-client/chromium-79.0.3945.16
       merge time: 1 hour, 33 minutes and 54 seconds.

     Tue Nov 12 21:37:39 2019 >>> www-client/chromium-79.0.3945.29
       merge time: 1 hour, 32 minutes and 48 seconds.

     Mon Nov 18 19:03:36 2019 >>> www-client/chromium-79.0.3945.36
       merge time: 1 hour, 31 minutes and 29 seconds.

     Sat Nov 23 16:45:21 2019 >>> www-client/chromium-79.0.3945.45
       merge time: 1 hour, 34 minutes and 24 seconds.

     Thu Nov 28 20:56:31 2019 >>> www-client/chromium-79.0.3945.56
       merge time: 1 hour, 43 minutes and 21 seconds.

     Mon Dec  9 00:45:46 2019 >>> www-client/chromium-79.0.3945.74   <-----------
       merge time: 3 hours, 47 minutes and 31 seconds.

     Wed Dec 11 23:06:44 2019 >>> www-client/chromium-79.0.3945.79
       merge time: 3 hours, 57 minutes and 48 seconds.

     Thu Dec 19 23:40:52 2019 >>> www-client/chromium-79.0.3945.88
       merge time: 3 hours, 28 minutes and 25 seconds.

Is there anything new to speed up the process?

Thank you
Ma
_________________
------------------------------------------------------------------
http://radio.garden/
Back to top
View user's profile Send private message
zsitvaij
n00b
n00b


Joined: 11 Jan 2006
Posts: 74

PostPosted: Sat Dec 21, 2019 3:53 pm    Post subject: Reply with quote

Upstream dropped support for jumbo-build and it was removed.
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1591

PostPosted: Sat Dec 21, 2019 4:53 pm    Post subject: Reply with quote

Oh yes, jumbo builds. That’s it.

Thanks zsitvaij,
I remember the many discussions about it.
_________________
------------------------------------------------------------------
http://radio.garden/
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Sat Dec 21, 2019 6:52 pm    Post subject: Reply with quote

zsitvaij wrote:
Upstream dropped support for jumbo-build and it was removed.


Did they give a proper reason for that?
Back to top
View user's profile Send private message
fedeliallalinea
Administrator
Administrator


Joined: 08 Mar 2003
Posts: 30917
Location: here

PostPosted: Sat Dec 21, 2019 6:57 pm    Post subject: Reply with quote

logrusx wrote:
Did they give a proper reason for that?

See https://forums.gentoo.org/viewtopic-p-8388710.html#8388710
_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Sat Dec 21, 2019 7:14 pm    Post subject: Reply with quote

fedeliallalinea wrote:
logrusx wrote:
Did they give a proper reason for that?

See https://forums.gentoo.org/viewtopic-p-8388710.html#8388710


Thanks!

That kind of sounds like good news, but I'm afraid what that GOMA would be...
Back to top
View user's profile Send private message
hariskar
Apprentice
Apprentice


Joined: 29 Jan 2017
Posts: 168

PostPosted: Mon Feb 10, 2020 3:03 pm    Post subject: Reply with quote

Betweeen the 2 last compiles the only difference is that in the last one I installed and use ccache. Is the expected difference so big?

Code:
genlop -t ungoogled-chromium
 * www-client/ungoogled-chromium

     Sun Jan 12 19:15:53 2020 >>> www-client/ungoogled-chromium-79.0.3945.117_p1
       merge time: 10 hours, 56 minutes and 53 seconds.

     Wed Feb  5 01:56:04 2020 >>> www-client/ungoogled-chromium-79.0.3945.117_p1
       merge time: 11 hours, 21 minutes and 54 seconds.

     Thu Feb  6 18:57:20 2020 >>> www-client/ungoogled-chromium-79.0.3945.117_p1
       merge time: 11 hours, 29 minutes and 27 seconds.

     Mon Feb 10 06:53:30 2020 >>> www-client/ungoogled-chromium-80.0.3987.87_p2
       merge time: 5 hours, 25 minutes and 6 seconds.


Intel(R) Core(TM) i3-4330 CPU @ 3.50GHz
-j4 8MB RAM
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Mon Feb 10, 2020 8:18 pm    Post subject: Reply with quote

This is the situation with me:
Code:
     Thu Jan 23 09:06:12 2020 >>> www-client/chromium-79.0.3945.130
       merge time: 1 day, 10 hours, 6 minutes and 33 seconds.

     Fri Jan 24 20:23:31 2020 >>> www-client/chromium-80.0.3987.53
       merge time: 1 day, 10 hours, 7 minutes and 38 seconds.

     Tue Jan 28 03:46:15 2020 >>> www-client/chromium-80.0.3987.66
       merge time: 6 hours, 34 minutes and 18 seconds.


I enabled ccache for 79, as you see the first time around 80 there was no difference - ~3.5% cache hits rate and the significant speedup was the second time with ~30% cache hits rate. I'm waiting for 81 to see if it really makes a difference after jumboo build removal, but I'm not very optimistic about that.

Good news is with regular updates it can be build in only one night without interrupting my daily activities :-)
Unfortunately that might not hold true between major version updates.

p.s. I'm pretty sure you didn't build it on 8 Megs of RAM :-P

Regards,
Georgi
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Thu Feb 13, 2020 7:45 am    Post subject: Reply with quote

Unfortunately no luck with 81 either.

These are the ccache statistics prior to starting compilation
Code:

cache hit (direct)                 23044
cache hit (preprocessed)           6218
cache miss                         56773

and these are after finishing:
Code:

cache hit (direct)                 24850
cache hit (preprocessed)           7175
cache miss                         83491


There is some insignificant number of cache hits. I'll stay with 81 until at least 82 gets stabilized.

Regards,
Georgi
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Thu Apr 23, 2020 5:11 pm    Post subject: Reply with quote

Hu wrote:
kharitonow wrote:
Hi Tom,
you can enable ccache feature in your make.conf (see wiki) to boost future builds of chromium.
This is a common claim, but the circumstances where it holds true are fairly narrow. Using ccache only helps when the cache is big enough to hold all the object files and when all the sources for a given object are unchanged. When you upgrade to a new version of Chromium, it is very likely that most or all of the sources will be changed enough that ccache will not help you.


Tested out on regular Chromium package updates, I can say that the time-saving is totally worth it.

A first-time compile (to warm up ccache) took > 8 hours (I can't do jumbo build because I only have 16Gb of RAM -- jumbo build makes the machine unusable during build). An immediate rebuild took 12 minutes. An upgrade from 81.0.4044.113 to 81.0.4044.122 (stable targets as at time of writing) took around 15 minutes. And that's with ccache's cache dir on a spinning-rust drive. 15 minutes is _way_ better than 8 hours. I can heartily encourage everyone to enable ccache if they're tired of incremental updates to Chromium tying up their machines for hours at a time.
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2575
Location: Here and Away Again

PostPosted: Thu Apr 23, 2020 5:48 pm    Post subject: Reply with quote

davydm wrote:
(I can't do jumbo build because I only have 16Gb of RAM -- jumbo build makes the machine unusable during build)

Even with more RAM, you probably couldn't since 'jumbo-build' has been removed from Chromium.

It can of course be reverted, which is what Qt had done [1], but we shall see how long they will want to maintain it.

1. https://code.qt.io/cgit/qt/qtwebengine-chromium.git/commit/?h=80-based&id=02ee3e81
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Thu Apr 23, 2020 7:00 pm    Post subject: Reply with quote

Chiitoo wrote:

Even with more RAM, you probably couldn't since 'jumbo-build' has been removed from Chromium.


shows how long it's been since I last even tried; there was a time when jumbo built on this machine, but a threshold was crossed somewhere and I found this machine reaching a few gig into swap, so I disabled the feature.

ccache turns out to be a major win here for me -- all these incremental releases for security patches and the like can be built in a fraction of the full build time, at the cost (currently) of around a gig of drive space -- a fair trade-off in my book.

I wouldn't mind an hour or two -- even my prior 4-hour jumbo build times were tolerable overnight; but at > 8 hours, I'm kicking off a world update at night and it's not completing until the next evening, largely because of chromium; I use firefox as my main browser, so chromium is really just there for testing and as a backup.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Thu Apr 23, 2020 7:20 pm    Post subject: Reply with quote

[quote="davydm"][quote="Hu"]
kharitonow wrote:

Tested out on regular Chromium package updates, I can say that the time-saving is totally worth it.


Time savings only happen on minor updates. You can see the cache hit and miss rates a few posts above.

Regards,
Georgi
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Thu Apr 23, 2020 7:49 pm    Post subject: Reply with quote

Of course the most major time savings will be in minor updates - the point is that there have been (and will continue to be) many minor updates. If I can save 95% of the build time 80% of the time, that's a win. The argument that ccache isn't worth it is clearly wrong - I could have saved over a day of build time on the last few updates alone. In addition, much of Chromium is 3rd party packages, which often remain stable. So even major version updates are likely to score from ccache - and the next time one comes along, I'll be sure to time and post.
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Thu Apr 23, 2020 9:09 pm    Post subject: Reply with quote

davydm wrote:
Of course the most major time savings will be in minor updates - the point is that there have been (and will continue to be) many minor updates. If I can save 95% of the build time 80% of the time, that's a win. The argument that ccache isn't worth it is clearly wrong - I could have saved over a day of build time on the last few updates alone. In addition, much of Chromium is 3rd party packages, which often remain stable. So even major version updates are likely to score from ccache - and the next time one comes along, I'll be sure to time and post.


Run some tests and you'll see for yourself :)
p.s. most of the minor updates make no sense.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21635

PostPosted: Fri Apr 24, 2020 1:48 am    Post subject: Reply with quote

He already did test it, and has two posts in this thread describing how it was helpful. I am glad to be wrong about this. Given all the other bad things about the Chromium build system, I fully expected that most files would be including something that would defeat the cache, since ccache is quite cautious about what it considers to be an equivalent input.
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Fri Apr 24, 2020 3:15 am    Post subject: Reply with quote

logrusx wrote:
p.s. most of the minor updates make no sense.


I'd expect most of the minor updates to be security-related, for example I remember reading a day or two ago about some vulnerability that Google wouldn't even disclose the nature of, so the author of that article was speculating that it was quite serious. Even though chromium isn't my daily driver for browsing, it's still nice to be able to get security patches in a lot quicker.
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Fri Apr 24, 2020 3:35 am    Post subject: Reply with quote

Hu wrote:
He already did test it, and has two posts in this thread describing how it was helpful. I am glad to be wrong about this. Given all the other bad things about the Chromium build system, I fully expected that most files would be including something that would defeat the cache, since ccache is quite cautious about what it considers to be an equivalent input.


I think it's a case of ymmv -- quite considerabily. Whilst I did record a good time above, it looks like 81.0.4044.113 -> 81.0.4044.122 is taking the full time:

Code:
* www-client/chromium-81.0.4044.122

       current merge time: 7 hours, 15 minutes and 59 seconds.
       ETA: any time now.


although
- I'm not sure if perhaps I just have a misconfiguration with ccache
- if it has anything to do with PYTHON_TARGETS changing (as per news item) and chromium using python as part of the build chain).

Again though, if ccache only reduces build times on 2/3 or 3/4 of builds by 95%, that's still a significant win for me. Also the fact that I can effectively resume a build is useful -- I live in South Africa where the state-run power company has done such a poor job of looking after their infrastructure that we have enforced "load-shedding", where power is simply cut for a few hours at a time. I've been in the situation where the same chromium build was interrupted more than once and had to be restarted ):

I'd like to learn more about ccache, particularly to find out why the cache appeared to not be working for the build above, though it's definitely in use if the build is interrupted and restarted -- I killed the build above and restarted with `emerge -1 chromium` and I can see it hitting cache (from the killed build above, no doubt)
Back to top
View user's profile Send private message
jserink
Veteran
Veteran


Joined: 30 Jan 2004
Posts: 1008

PostPosted: Fri Apr 24, 2020 3:57 am    Post subject: Dumped chromium Reply with quote

Hi All:

I left my machine on the other night to emerge chromium-81.0.4044.113, took nearly 6 hours.

The next day my son tried to use my computer for E-Learning for school, he used my computer two days ago worked fine, the new chromium wouldn't play the videos.

So that is that. 6 hours to compile so videos won't play? I don't have time for this.
Am on a i7, SSD, tmpfs and 32G of ram.

Chromium is gone, got rid of it. 6 hour compile times are over the top.

I'll use Brave and Chrome, they work for me.

Onward and forward.

Cheers,
john
Back to top
View user's profile Send private message
logrusx
Veteran
Veteran


Joined: 22 Feb 2018
Posts: 1535

PostPosted: Fri Apr 24, 2020 6:37 am    Post subject: Reply with quote

davydm wrote:
- I'm not sure if perhaps I just have a misconfiguration with ccache
- if it has anything to do with PYTHON_TARGETS changing (as per news item) and chromium using python as part of the build chain).

There's nothing much you can configure/misconfigure about ccache. Also ccache is not a good restore mechanism, as a lot of the process is executing python scripts.
davydm wrote:
Again though, if ccache only reduces build times on 2/3 or 3/4 of builds by 95%, that's still a significant win for me. Also the fact that I can effectively resume a build is useful

Keepwork feature allows you to resume builds. You can enable it per package

In conjunction with ccache it'll do a good job, although sometimes it may cost you failed builds and starting over(clearing the cache), but you need to cleanup /var/tmp/portage directory manually. For an example how to do it, see one of my older posts.

Regards,
Georgi
Back to top
View user's profile Send private message
davydm
n00b
n00b


Joined: 06 Jan 2017
Posts: 73

PostPosted: Fri Apr 24, 2020 7:20 am    Post subject: Reply with quote

logrusx wrote:

Keepwork feature allows you to resume builds. You can enable it per package

In conjunction with ccache it'll do a good job, although sometimes it may cost you failed builds and starting over(clearing the cache), but you need to cleanup /var/tmp/portage directory manually. For an example how to do it, see one of my older posts.

Regards,
Georgi


Thanks, I didn't know about this -- perhaps enabling for chromium only might be worthwhile.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2, 3, 4, 5  Next
Page 4 of 5

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum