Chromium compile doesn't seem to be using ccache

[mi_unixbird]

Code: Select all

—— — ccache -s
Cacheable calls:   2085603 / 3286807 (63.45%)
  Hits:             482121 / 2085603 (23.12%)
    Direct:         240773 /  482121 (49.94%)
    Preprocessed:   241348 /  482121 (50.06%)
  Misses:          1603482 / 2085603 (76.88%)
Uncacheable calls:  499694 / 3286807 (15.20%)
Errors:             701510 / 3286807 (21.34%)
Local storage:
  Cache size (GB):    30.0 /    30.0 (99.98%)
  Cleanups:            384
  Hits:             482121 / 2085603 (23.12%)
  Misses:          1603482 / 2085603 (76.88%)
 —— — sudo emerge -1 chromium
[ebuild     U ] www-client/chromium-128.0.6613.137 [128.0.6613.119]

Would you like to merge these packages? [Yes/No]
>>> Verifying ebuild manifests
>>> Running pre-merge checks for www-client/chromium-128.0.6613.137
 * Checking for at least 4 GiB RAM ...                                                     [ ok ]
 * Checking for at least 25 GiB disk space at "/var/tmp/build/portage/www-client/chromium-128.0.6613.137/temp" ...                                                                           [ ok ]
>>> Emerging (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Installing (1 of 1) www-client/chromium-128.0.6613.137::gentoo
>>> Completed (1 of 1) www-client/chromium-128.0.6613.137::gentoo

 * Messages for package www-client/chromium-128.0.6613.137:

 * PYTHONPATH='/home/user/.local/lib/Python'
 * VA-API is disabled by default at runtime. You have to enable it
 * by adding --enable-features=VaapiVideoDecoder to CHROMIUM_FLAGS
 * in /etc/chromium/default.
 —— — ccache -s
Cacheable calls:   2085603 / 3326157 (62.70%)
  Hits:             482121 / 2085603 (23.12%)
    Direct:         240773 /  482121 (49.94%)
    Preprocessed:   241348 /  482121 (50.06%)
  Misses:          1603482 / 2085603 (76.88%)
Uncacheable calls:  499760 / 3326157 (15.03%)
Errors:             740794 / 3326157 (22.27%)
Local storage:
  Cache size (GB):    30.0 /    30.0 (99.98%)
  Cleanups:            384
  Hits:             482121 / 2085603 (23.12%)
  Misses:          1603482 / 2085603 (76.88%)

The way I interpret this with the “cachable calls” number remaining identical at 2085603 befoe and after is that not a single call was “cachable” which seems weird. I won't profess to exactly know what al this means but I assume it means that it somehow can't use ccache? Does this have something to do with some kind of custom build system it uses?

Also, for what it's worth:

Code: Select all

 —— — ls -l /usr/lib/ccache/bin/
total 104
lrwxrwxrwx 1 root root 15 Sep  7 04:48 c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 c99 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 i686-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang++-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-clang-18 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-g++-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 x86_64-pc-linux-gnu-gcc-14 -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Sep  7 04:48 zvbi-ntsc-cc -> /usr/bin/ccache

[mi_unixbird]

In case anyone later stumbles upon this thread. I solved the issue, more or less, “ccache -sv” revealed that the error was “compiler check failed” so I removed the “compiler_check” configuration option in distcc.conf. This now does apparently mean that a re-installation of the same compiler version will invalidate the cache but I never had that.

[logrusx]

I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.

Best Regards,
Georgi

[mi_unixbird]

I spent a lot of time on that back when I had a dual core CPU with maximum limit of 8BG RAM which I didn't even have. Chromium doesn't benefit from ccache either way unless you recompile one and the same version.

Best Regards,
Georgi

I remember Chromium benefiting immensely from ccache wen it still worked for me and it cutting down the compilation times from 8 hours to 2.5 hours:

Code: Select all

     Mon Sep  4 05:40:19 2023 >>> www-client/chromium-116.0.5845.140
       merge time: 7 hours, 54 minutes and 13 seconds.

     Wed Sep 13 09:39:21 2023 >>> www-client/chromium-116.0.5845.187
       merge time: 2 hours, 19 minutes and 58 seconds.

     Wed Oct  4 07:36:08 2023 >>> www-client/chromium-117.0.5938.132
       merge time: 7 hours, 53 minutes and 23 seconds.

     Fri Oct 13 05:29:07 2023 >>> www-client/chromium-117.0.5938.149
       merge time: 2 hours, 47 minutes and 43 seconds.

I read a lot of things on the internet about ccache supposedly not doing much and only slowing down, all of those posts not giving any numbers but theoretical reasons like that cache misses slow things down, which should be true, but it should also be completely trivial compared to cache hits because hashes are incredibly cheap compared to actual compilation. The numbers on my system at least speak for themselves I might do an actual test later with a smaller package than chromium which is of course immense but these numbers above with chromium while a compiler update happened twice speak for tthemselves. In fact, ccache makes it so much faster oe could argue it's worth it to not update compilers for a considerable time merely for the sheer compilation speed benefits it brings.

Anyway, I found the ultimate source now. The compiler_check was set to “%compiler -v%” but on clang that does't return the version at all like on GCC but is the verbose output flag and it errors out when not given anything to copile of course; I wonder when that change happened. I honestly have been thinking for a while now that chromium actually got this ridiculously big with ccache that it took 9 hours now with it, not realizing that it wasn't uing it at all but still had to do a copiler check for every file and was only slowing it down.

[logrusx]

I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.

I then switched to ungoogled-chromium-bin, but ultimately I switched to Firefox.

Best Regards,
Georgi

[mi_unixbird]

I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.

How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?

We'll see what difference it brings with the next update though.

And yes, I still remember when Chromium took a mere two hours without cccache. I really wondner what happened. I've been considering switching to Firefox purely for the compilation speed advantage but from what I read it's less snappy and I would need to redo all my settings. The Chomium distfiles are also 7 GiB by now.

[logrusx]

I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.

How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?

Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk.

And yes, I still remember when Chromium took a mere two hours without cccache.

Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later.

I would test it now to see how it works but it wants to recompile nodejs as well and I don't want to go down that route and compile at least 6-7 hours just to test.

Best Regards,
Georgi

[mi_unixbird]

I haven't compiled chromium for a long time, it still used GCC when I did. Back then it was written in a way ccache wasn't doing much. It used to take around two and a half hours to compile, but with ccache it was taking maybe 15 minutes more when the same version is recompiled and it had very high hit rates.

How does this work? Surely the number of cache hits is really the only thing that matters for time reduction?

Disk IO. Waiting for SSD is still an endless wait for the CPU. Ironically enough, it was faster to compile the files than looking them up from the cache on the disk.

This doesn't make any sense to me. New compilations have to write the files to the drive somehow which is far slower while ccache only compares hashes of identical compiler and compilation options stored for each cached file. I would assume that if it copy files, it's smart enough to use reflink=auto which is practically free on a copy-on-write filesystem.

Don't get me wrong, it took 2 and a half hours on a Ryzen 7 5800H with 32GB RAM in a cold room 3 and a half years ago. On my old dual core computer it took more like 7 hours with jumbo build back when they supported it and maybe 40 when they stopped supporting it more than 4 years ago. I used to use ccache as a means of resuming the build process because it was fragile back then and was breaking when resuming with keepwork feature enabled. This way I could compile for several consecutive nights while still being able to use my computer through the day. Later I discovered a not guaranteed to work fix was to delete .setuped in the temp directory, so that portage was forced to recreate the environment and successfully resume. I think they fix that too later.

Two hours for me was when I first installed it, which was about a decade ago I'd say. On my machine now, it's about 8-9 hours now without the cache at least, but suffering the pœnalty of that it tries it at least. With the next update I will know how much time ccache cuts again but the numbers of even early this year look very promising with 2.5 hour compile times being the norm opposed to the 8 they take now.

[Chiitoo]

I believe the last version of Chromium to have USE="jumbo-build" was 78.0.3904.108, which was removed back in 2019 [1].

That flag, when enabled, would usually cut down the build time to half, while using more RAM.

Aside from that, I think Chromium just got bigger.

(Qt still maintains 'jumbo-build' for their fork of Chromium, so one can someawhats test the difference using that still.)

Code: Select all

$ genlop -t chromium
 * www-client/chromium

     Thu May  8 01:39:04 2014 >>> www-client/chromium-35.0.1916.86
       merge time: 42 minutes and 38 seconds.

     Mon Mar 16 09:00:10 2015 >>> www-client/chromium-41.0.2272.76
       merge time: 1 hour, 27 minutes and 29 seconds.

     Sat Jan 28 02:31:53 2023 >>> www-client/chromium-110.0.5481.38
       merge time: 6 hours, 44 minutes and 37 seconds.

     Sat May  6 16:19:52 2023 >>> www-client/chromium-113.0.5672.63
       merge time: 1 hour, 22 minutes and 12 seconds.

     Mon May 15 10:20:34 2023 >>> www-client/chromium-113.0.5672.63
       merge time: 11 seconds.

     Wed May 17 09:34:44 2023 >>> www-client/chromium-113.0.5672.92
       merge time: 1 hour, 33 minutes and 59 seconds.

     Fri May 26 10:58:24 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 1 hour, 39 minutes and 19 seconds.

     Sat May 27 14:49:57 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 8 minutes and 24 seconds.

     Sat May 27 21:07:17 2023 >>> www-client/chromium-113.0.5672.126
       merge time: 1 hour, 19 minutes and 29 seconds.

     Sun Jun 11 22:36:50 2023 >>> www-client/chromium-114.0.5735.110
       merge time: 1 hour, 41 minutes and 40 seconds.

The builds before 2023 were using an AMD Phenom II X6 1090T, while the later builds used AMD Ryzen 9 7950X (don't mind the 6-hour one; something special definitely was going on there).

I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.

1. https://gitweb.gentoo.org/repo/gentoo.g ... 517e346f15

[logrusx]

I never really tested it with Chromium, but perhaps 'ccache' is still useful when moving between versions such as 116.0.5845.140 and 116.0.5845.187, but yeah, definitely seen that it has never been much, if any use between major versions.

It hasn't been much even with minor versions. They used to make changes that would result in untouched files giving completely different result after being preprocessed so ccache was just a waste. I hesitate to argue it still is because although I don't think something changed in the development of chromium, I have seen lucky runs which decreased time significantly in the past but were one-offs. Who knows, they may really have improved their development process.

Best Regards,
Georgi

[mi_unixbird]

Well, we'll see what changes with the next compile, but those three hour build times of Chromium I still had earlier this year before ccache failed do certainly look attractive. We'll see if it go back to that but it's hard to argue with theory against the statistics that in my genlop -t logs of chromium there are 3 hour build times [ccache succeeded], 8 hour build times [ccache didn't succeed due to say a compiler update or the recent problem], and pretty much nothing in between.

[mi_unixbird]

First cached update:

Code: Select all

     Wed Sep 18 08:50:15 2024 >>> www-client/chromium-129.0.6668.42
       merge time: 10 hours, 29 minutes and 55 seconds.

     Sat Sep 21 13:32:19 2024 >>> www-client/chromium-129.0.6668.58
       merge time: 25 minutes and 58 seconds.

I would assume this was a very good case though since the version differences are very minor. I assume a major version update won't be quite this spectacular since as said, I remember 3 hour builds and I assume changing major dependencies in between also ruins quite a bit but at the very least the “best case” is absolutely spectacular.

In the meanwhile I looked at some statistics for ccache usage on big projects it seems to diverge from that 100% cache misses, as in first population increases compile time to 101% to 115% depending on the mode used while 100% cache hits, as in a direct rebuild, reduces it to 0.5%-5% for most big projects. In general the more preprocessor intensive a project is the lower the benefits of ccache are but it seems to me that even with 10% hits on average it should speed up.

[logrusx]

I still can't believe chromium build makes good use of ccache. There's a new 129 version available. Also 130. I'm curious what will happen with one of those.

Best Regards,
Georgi

[mi_unixbird]

I still can't believe chromium build makes good use of ccache. There's a new 129 version available. Also 130. I'm curious what will happen with one of those.

Best Regards,
Georgi

I just had it updated. This one was disastrous:

Code: Select all

     Sun Sep 29 19:32:00 2024 >>> www-client/chromium-129.0.6668.70
       merge time: 9 hours, 42 minutes and 35 seconds.

Code: Select all

Cacheable calls:   39428 / 39474 (99.88%)
  Hits:             1962 / 39428 ( 4.98%)
    Direct:         1962 /  1962 (100.0%)
    Preprocessed:      0 /  1962 ( 0.00%)
  Misses:          37466 / 39428 (95.02%)
Uncacheable calls:    46 / 39474 ( 0.12%)

I actually zeroed the statistics before compiling this time and cache hit ratio was abysmal. I don't think a compiler update happened but clearly something happened that made the ratio so low. Maybe it has something to do with that I switched to depend mode in the meanwhile

It's still slightly faster than how it was when all calls where uncachable which was around 10~10:30 hours but it's not a pretty picture here. We'll see what the next update brings. I have been muckking around a bit and tuning some things and switched to a per-package cache originally just forked of the main cache by using a reflink copy of it so maybe that did something. Come to think of I didn't use “cp -a” to do this. Perhaps ccache really doesn't like it if file creation dates are updated. I definitely should've done that.

[mi_unixbird]

After getting a similarly abysmal result I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless. I read somewhere that depend mode was generally faster but it seemed that only applied to rebuilding the same identical thing twice.

[logrusx]

I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless.

Good job. However the common hash information includes the path for all modes, so they are all useless.

The following information is always included in the hash:

the extension used by the compiler for a file with preprocessor output (normally .i for C code and .ii for C++ code)

the compiler’s size and modification time (or other compiler-specific information specified by compiler_check)

the name of the compiler

the current directory (if hash_dir is enabled)

So you may try with hash_dir disabled. If that works you will have given hundreds, if not thousands of Gentoo users new hope for compiling Chromium and its derivatives with ccache.

Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Best Regards,
Georgi

[mi_unixbird]

I did some research and it turns out that depend mode is entirely useless for upgrades because it by necessity needs to use the file pah in the hash then, not simply the content of the file, and since portage includes the version in the file path in the build dir it's of course entirely useless.

Good job. However the common hash information includes the path for all modes, so they are all useless.

The following information is always included in the hash:

the extension used by the compiler for a file with preprocessor output (normally .i for C code and .ii for C++ code)

the compiler’s size and modification time (or other compiler-specific information specified by compiler_check)

the name of the compiler

the current directory (if hash_dir is enabled)

So you may try with hash_dir disabled. If that works you will have given hundreds, if not thousands of Gentoo users new hope for compiling Chromium and its derivatives with ccache.

Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Best Regards,
Georgi

I already had hash_dir disabled of course.

It occurs to me that the people who had bad results, though they said it was despite getting high hit ratios might actually have it enabled. It's common knowledge to disable it I thought and I honestly don't understand why it's enabled by default. I would obviously not be getting those 3 hours and even 30 minute compile times if I had it enabled.

In fact, reading on about how it works. It might actually be worthwhile to completely disable direct mode so it doesn't even try for updates since it will almost always fail anyway, and have it always run in preprocessor mode. I'm not sure what the reason for including the current working directory is but using the current file names is because a file may contain a “__FILE__” macro and would thus get a different result with a different filename, perhaps this is also the reason the current working directory is included for relative filenames, but I don't see how that would still be relevant for files after preprocessing.

Or if you find a way to cut that part of the path that contains the version information, but that I think is not trivial and may need support in the ebuild itself.

Honestly, this made me realize that the preprocessing for some files will still be different as they will contain debug information that contains the full path they are located in.I wonder if one can set up something with namespaces to begin with to make everything that compiles believe it's workdir is actually just under the same default directory while it isn't. This seems like a useful feature for more than one reason.

But even then many things will still contain versioning numbers I suppose that will be different in debug output. Also, I have distcc set up to rewrite all paths to relative but even the relative paths often still contain versioning numbers which I suppose is needed to provide accurate debugging.

[mi_unixbird]

After some further digging and hacking I found that ccache is obviously heavily influenced by line marker metadata emitted by the preprocessor depending on the build system which may include a version number in the path. I now used a compiler wrapper which passes “-P” to the preprocessor to strip this. This led to mesa, a package I didn't use ccache on because hits were abysmal as in the 5% range to actually get over 90% hit ratio now on an upgrade cutting built time down from 9 to 1 minute.

As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio.

[logrusx]

Any news about that?

Best Regards,
Georgi

[haukurpj]

Indeed, how are your experiments going?

[mi_unixbird]

Any news about that?

Best Regards,
Georgi

Indeed, how are your experiments going?

I actually saw the first post a while back already but chromium had switched to llvm 19 form 18 since the last compile back then so the last compile was a long one again without any meaningful results, I just had a new one which did use the cache and I specifically zeroed the stats for it.

We have:

Code: Select all

 —— — portage-ccache www-client/chromium -sv
Cache directory:        /var/cache/portage/ccache/www-client/chromium
Config file:            /var/cache/portage/ccache/www-client/chromium/ccache.conf
System config file:     /etc/ccache.conf
Stats updated:          Fri Jan 10 20:45:34 2025
Cacheable calls:         40746 / 40790 (99.89%)
  Hits:                  40042 / 40746 (98.27%)
    Direct:                  0 / 40042 ( 0.00%)
    Preprocessed:        40042 / 40042 (100.0%)
  Misses:                  704 / 40746 ( 1.73%)
Uncacheable calls:          44 / 40790 ( 0.11%)
  Called for linking:       42 /    44 (95.45%)
  Preprocessing failed:      2 /    44 ( 4.55%)
Successful lookups:
  Direct:                    0
  Preprocessed:          40042 / 40746 (98.27%)
Local storage:
  Cache size (GB):         5.0 /   5.0 (100.0%)
  Files:                178454
  Cleanups:                 26
  Hits:                  40042 / 40746 (98.27%)
  Misses:                  704 / 40746 ( 1.73%)
  Reads:                 40746
  Writes:                  704

Weird name of the command is because I made a script now that manages per-package caches in portage. The 5 GB cache size for chromium is probably too large. It fills about 1.4 GiB when it's empied from a single compile.

The build times this insane 98.27% cache hits translates to is:

Code: Select all

     Mon Dec 16 03:00:41 2024 >>> www-client/chromium-131.0.6778.139
       merge time: 8 hours, 20 minutes and 24 seconds.

     Fri Jan 10 20:47:27 2025 >>> www-client/chromium-131.0.6778.264
       merge time: 1 hour, 49 minutes and 49 seconds.

Obviously a difference that's very obvious but I should add that in my experience 98.27% hits is rare. You're usually looking at 80% hits. I guess not a lot was changed in terms of chromium itself and headers this time

Unusually because I started it before going to bed, but then realized I still had something to do, I was actually there when it happened, usually I'm asleep when chromium compiles, I noticed that the bulk of time was spent on python, not a C-compiler so I guess it uses some fairly intensive python processing that ccache obviously can't do anything with. In any case, I somehow still haven't been able to test it to my knowledge with an actual major update since every time a major update did happen, it coincided with an LLVM update before it or something else that dropped cache hits to something really low to nonexistent but the results thusfar with minor updates obviously cut the build times quite measurably.

[mi_unixbird]

New update which required the use of -system-icu due to chromium apparently not being willing to compile with icu-76.1. :

Code: Select all

 —— — portage-ccache www-client/chromium -s
Cacheable calls:   41186 / 41230 (99.89%)
  Hits:              976 / 41186 ( 2.37%)
    Direct:            0 /   976 ( 0.00%)
    Preprocessed:    976 /   976 (100.0%)
  Misses:          40210 / 41186 (97.63%)
Uncacheable calls:    44 / 41230 ( 0.11%)
Local storage:
  Cache size (GB):   5.0 /   5.0 (99.98%)
  Cleanups:         1351
  Hits:              976 / 41186 ( 2.37%)
  Misses:          40210 / 41186 (97.63%)

Absolutely distrastrous, only 2.37% hits and

Code: Select all

     Tue Jan 21 15:56:18 2025 >>> www-client/chromium-132.0.6834.83
       merge time: 9 hours, 27 minutes and 21 seconds.

I wonder if this is really caused by icu being absolutely everywhere and thus requiring almost every file to be recompiled or whether somthing else was going on but it's remarkable to see how deeply the cache hit ratio can fluctuate. We've seen both the best and worst case. I'm still hoping to see what a normal 70-80% cache hit compile does.

[mi_unixbird]

Well, I got my double wish I guess, just after I updated Gentoo realized that icu should be kept at 75.1 if chromium doesn't play nice with it and wanted to downgrade, but, the interesting thing is that the downgrade and associated rebuild got me my reasonable 73% hit ratio:

Code: Select all

 —— — portage-ccache www-client/chromium -s
Cacheable calls:   40741 / 40785 (99.89%)
  Hits:            29821 / 40741 (73.20%)
    Direct:            0 / 29821 ( 0.00%)
    Preprocessed:  29821 / 29821 (100.0%)
  Misses:          10920 / 40741 (26.80%)
Uncacheable calls:    44 / 40785 ( 0.11%)
Local storage:
  Cache size (GB):   5.0 /   5.0 (99.95%)
  Cleanups:          479
  Hits:            29857 / 40777 (73.22%)
  Misses:          10920 / 40777 (26.78%)

Which leads to these three merges, one with a spectacular hit ratio, one with an abysmal one, and one with an average one:

Code: Select all

     Fri Jan 10 20:47:27 2025 >>> www-client/chromium-131.0.6778.264
       merge time: 1 hour, 49 minutes and 49 seconds.

     Tue Jan 21 15:56:18 2025 >>> www-client/chromium-132.0.6834.83
       merge time: 9 hours, 27 minutes and 21 seconds.

     Thu Jan 23 07:49:08 2025 >>> www-client/chromium-132.0.6834.83
       merge time: 5 hours, 32 minutes and 55 seconds.

I should add that the first two had other compiles going on at the same time. The last one is the only one where chromium was really the only thing compiling the entire time. There was also another one in this thread where with that spectacular hit rate it was only 40 minutess which I guess was really chromium purely running alone.
guess this experiment is kind of done now and I won't be zealously tracking hit ratios and so forth any more in the future since they require zeroing the cache before each compile and I wish it could just print statistics between specific dates to avoid all that but しょうがない.

Conclusion: I think I still have to disagree with the people who say that ccache doesn't work on updates. The spectacular cases with amazing hit ratios which two were included here, but many more where had obviously speak for itself. The 73% hit ratio cases aren't as amazing but still roughly cut it in half. The abysmall ones in fact still seem to be faster than without the cache but not by much and these numbers might not be reliable.

For reference with nodejs, which I also recently included in the cache, no idea why I didn't sooner:

Code: Select all

 —— — portage-ccache net-libs/nodejs -s
Cacheable calls:   3632 / 3664 (99.13%)
  Hits:            2643 / 3632 (72.77%)
    Direct:           0 / 2643 ( 0.00%)
    Preprocessed:  2643 / 2643 (100.0%)
  Misses:           989 / 3632 (27.23%)
Uncacheable calls:   32 / 3664 ( 0.87%)
Local storage:
  Cache size (GB):  0.3 /  0.5 (60.61%)
  Hits:            2643 / 3632 (72.77%)
  Misses:           989 / 3632 (27.23%)
 —— — 

Code: Select all

     Wed Jan 15 17:42:44 2025 >>> net-libs/nodejs-22.12.0-r1
       merge time: 54 minutes and 39 seconds.

     Tue Jan 21 04:37:29 2025 >>> net-libs/nodejs-22.12.0-r1
       merge time: 1 hour, 3 minutes and 30 seconds.

     Wed Jan 22 13:33:04 2025 >>> net-libs/nodejs-22.12.0-r1
       merge time: 7 minutes and 5 seconds.

This is nodejs being rebuilt thrice due to this little issue with icu. This one seems to benefit immensely from a mere 72 hit right so I assume the files that stayed the same were the big ones here but I haven't tested this package on an actual update yet but I'm left to assume that people who say that it doesn't work just have it misconfigured, as in they actually cache the working directory and obviously don't employ my preprocessor hack which also technically creates wrong results on a debugging level but who cares about that. I wish ccache had an option to just pass this flag itself without having to rely on such an utter hack.

[mi_unixbird]

I'm actually still running this experiment because I'm actually quite curious:

Code: Select all

     Sun Feb  9 07:33:35 2025 >>> www-client/chromium-132.0.6834.159
       merge time: 10 hours, 58 minutes and 23 seconds.

     Fri Feb 14 11:19:15 2025 >>> www-client/chromium-133.0.6943.53
       merge time: 9 hours, 56 minutes and 54 seconds.

     Sun Feb 16 07:45:52 2025 >>> www-client/chromium-133.0.6943.98
       merge time: 4 hours, 9 minutes and 39 seconds.

The first long 11 hour compile I get. LLVM and Clang were updated there, it stands to reason that would invalidate the entire cache. The second one that's 10 however, which had less than 1% hits. Looking at what was compiled between it, the only thing I can think of is boost. It's possible that an update of boost also completely invalidates the cache for chromium. Regardless, the four hour compile was 89% cache hits.

So regardless, we're still at the situation that there is a lot that can completely invalidate the cache for some projects at least, but when it not happen, then we're still dealing with a 90% cache hit rate even on updates, cutting far over half of the build time off.

[user]

Which package provide "portage-ccache" ?