Chromium compile doesn't seem to be using ccache

[logrusx]

https://wiki.gentoo.org/wiki/Ccache

[user]

https://wiki.gentoo.org/wiki/Ccache

Code: Select all

 * Contents of dev-util/ccache-4.10.2-r1:
/usr
/usr/bin
/usr/bin/ccache
/usr/bin/ccache-config
/usr/share
/usr/share/doc
/usr/share/doc/ccache-4.10.2-r1
/usr/share/doc/ccache-4.10.2-r1/AUTHORS.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/CONTRIBUTING.md.lz
/usr/share/doc/ccache-4.10.2-r1/MANUAL.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/NEWS.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/README.md.lz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/ccache.1.lz
/usr/share/shadowman
/usr/share/shadowman/tools
/usr/share/shadowman/tools/ccache

no portage-ccache

[mi_unixbird]

Which package provide "portage-ccache" ?

Nothing, it's just a script I made that does nothing more than automatically setting the CCACHE_DIR environment variable.

The important part is this part of /etc/portage/bashrc:

Code: Select all

if [[ ${FEATURES} == *ccache* && ${EBUILD_PHASE_FUNC} == src_* ]]; then
	export CCACHE_BASEDIR="${PORTAGE_TMPDIR}/portage"
	if [[ ${CCACHE_DIR} == /var/cache/portage/ccache ]]; then
		if [[ ${CATEGORY} == kde-frameworks ]]; then
			export CCACHE_DIR=${CCACHE_DIR}/${CATEGORY}
		else
			export CCACHE_DIR=${CCACHE_DIR}/${CATEGORY}/${PN}
		fi
		mkdir -p "${CCACHE_DIR}" || die
	fi
fi

Which assigns each package it's own private cache in /var/cache/portage/ccache for better statistics and monitoring.

[logrusx]

As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio.

Where do you put that?

Best Regards,
Georgi

[mi_unixbird]

As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio.

Where do you put that?

Best Regards,
Georgi

In the /etc/ccache.conf or local per-cache configuration.

https://ccache.dev/manual

The “prefix_command_cpp” option and the configuration style syntax is explained here.

[logrusx]

Thanks. I almost filled the cache, only a few thousand units remain until the compilation completes. We'll see what happens on the next release. I've never had good results with ccache, but I remain optimistic. No big hoes though.

EIDT: it finished. Cache size is 2.1 Gb. It even had 2 cache hits during the compilation. However I'm disappointed. Three years after they enabled Wayland support and Chromium still does not allow me to select more than 1080p YouTube video quality, while Firefox has been supporting it for quite a while now. I've abandoned Chromium for more than a year and I'm doing this out of curiosity only.

Best Regards,
Georgi

[logrusx]

I don't know what I'm doing wrong but I've never got good cache hit ratios with chromium. Some people claimed they sometimes had good rates even without the change you suggested in the past, but I never did. I used ungoogled chromium for this experiment and I terminated it third of the way with abysmal 10% hit rate. Yes, 10% is more than I've ever had, but I don't think it's worth.

This time mesa was pulled in and I enabled cache for it too and I'll see how it does next update.

Best Regards,
Georgi

[mi_unixbird]

I don't know what I'm doing wrong but I've never got good cache hit ratios with chromium. Some people claimed they sometimes had good rates even without the change you suggested in the past, but I never did. I used ungoogled chromium for this experiment and I terminated it third of the way with abysmal 10% hit rate. Yes, 10% is more than I've ever had, but I don't think it's worth.

This time mesa was pulled in and I enabled cache for it too and I'll see how it does next update.

Best Regards,
Georgi

My ccache.conf for chromium looks like this, not sure whether you forgot a setting:

Code: Select all

# Maximum cache size to maintain
max_size = 5.0G

# Allow others to run 'ebuild' and share the cache.
umask = 002

# Preserve cache across GCC rebuilds and
# introspect GCC changes through GCC wrapper.
compiler_check = %compiler% -dumpversion

# I expect 1.5M files. 300 files per directory.
cache_dir_levels = 3

cache_dir = /var/cache/portage/ccache/www-client/chromium

file_clone=true
depend_mode=false
direct_mode=false
hash_dir=false
run_second_cpp=false

I don't use the strip preprocessor hack for chromium because it doesn't do anything for it, I only use it for other packages.

The only time i get sub 10% hits is if some kind of fundamental dependency like the compiler itself updated and the entire cache is invalidated. With normal updates it rarely drops below 80%

[keekkenen]

Three years after they enabled Wayland support and Chromium still does not allow me to select more than 1080p YouTube video quality, while Firefox has been supporting it for quite a while now.

It's very strange, I've no problem with 1440p and 2160p in Chromium under plasma with wayland.

[mi_unixbird]

By the way, I got two absolutely terrible cache hit ratios in a row now, quite quickly after each other too.

Code: Select all

 —— — portage-ccache www-client/chromium -s
Cacheable calls:   41192 / 41237 (99.89%)
  Hits:                8 / 41192 ( 0.02%)
    Direct:            0 /     8 ( 0.00%)
    Preprocessed:      8 /     8 (100.0%)
  Misses:          41184 / 41192 (99.98%)
Uncacheable calls:    45 / 41237 ( 0.11%)
Local storage:
  Cache size (GB):   5.0 /   5.0 (99.97%)
  Cleanups:         1099
  Hits:                8 / 41192 ( 0.02%)
  Misses:          41184 / 41192 (99.98%)

I have no idea what could've updated in the meanwhile that caused this. The last one was like 5%, not 0.02% but I hope this doesn't persist due to some build system change that makes chromium non-viable with ccache from now on. Abseil seems like a prbably candidate, maybe it was that that caused it.

[keekkenen]

Yes, I saw it. Existing ccache had no effect to cromium compilation. I don't know why.

[keekkenen]

By the way, I got two absolutely terrible cache hit ratios in a row now, quite quickly after each other too.

You have better static than me.
I cleaned ccache statistic before test and you can see below statistic after first and next compilation.

Code: Select all

Cacheable calls:       0 / 41880 ( 0.00%)
  Hits:                0
    Direct:            0
    Preprocessed:      0
  Misses:              0
Uncacheable calls:    46 / 41880 ( 0.11%)
Errors:            41834 / 41880 (99.89%)
Local storage:
  Cache size (GB):   0.0 /  10.0 ( 0.00%)

Code: Select all

Cacheable calls:       0 / 83779 ( 0.00%)
  Hits:                0
    Direct:            0
    Preprocessed:      0
  Misses:              0
Uncacheable calls:   111 / 83779 ( 0.13%)
Errors:            83668 / 83779 (99.87%)
Local storage:
  Cache size (GB):   0.0 /  10.0 ( 0.00%)

[mi_unixbird]

New update again:

Code: Select all

Cacheable calls:   41192 / 41237 (99.89%)
  Hits:            39697 / 41192 (96.37%)
    Direct:            0 / 39697 ( 0.00%)
    Preprocessed:  39697 / 39697 (100.0%)
  Misses:           1495 / 41192 ( 3.63%)
Uncacheable calls:    45 / 41237 ( 0.11%)
Local storage:
  Cache size (GB):   5.0 /   5.0 (99.99%)
  Cleanups:           40
  Hits:            39728 / 41223 (96.37%)
  Misses:           1495 / 41223 ( 3.63%)

That's what I'm talking about. It seems to still work fine if specific packages aren't getting updated.

[keekkenen]

I changed options from MAKEOPTS="-j28 -l28" to MAKEOPTS="-j31 -l31" and got worse last compile time.

Code: Select all

2025-04-06T17:06:04 >>> www-client/chromium: 1:47:38
2025-04-10T00:11:49 >>> www-client/chromium: 1:55:32

It's just funny.

[mi_unixbird]

I changed options from MAKEOPTS="-j28 -l28" to MAKEOPTS="-j31 -l31" and got worse last compile time.

Code: Select all

2025-04-06T17:06:04 >>> www-client/chromium: 1:47:38
2025-04-10T00:11:49 >>> www-client/chromium: 1:55:32

It's just funny.

Just how many cores do you have that this is a good idea?

Also:

Code: Select all

Cacheable calls:   82384 / 82474 (99.89%)
  Hits:            76679 / 82384 (93.08%)
    Direct:            0 / 76679 ( 0.00%)
    Preprocessed:  76679 / 76679 (100.0%)
  Misses:           5705 / 82384 ( 6.92%)
Uncacheable calls:    90 / 82474 ( 0.11%)
Local storage:
  Cache size (GB):   5.0 /   5.0 (99.97%)
  Cleanups:          224
  Hits:            76713 / 82418 (93.08%)
  Misses:           5705 / 82418 ( 6.92%)

I didn't empty the statistics this time so some basic maths is required but it seems to have been around 90% hit rate this time opposed to the 96% last time which is still good of course. I wish ccache stats allowed one to just see the stats between certain dates without having to zero them.

In any case, though obviously some updates alter something big that essentially cause the entire cache to come invalidated. I think I have firmly shown at this point that ccache is absolutely effective with Chromium between updates, being able to cut down 10 hour compiles to 2.5 with the right setup which involves not hashing the current working directory. This will of course make debug information print the wrong version number at times in terms of what file causes some error but I don't think anyone really caress about that. ccache's default setup prioritizes correctness above everything else.