| View previous topic :: View next topic |
| Author |
Message |
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 3031
|
|
| Back to top |
|
 |
user
Apprentice
Joined: 08 Feb 2004
Posts: 236
|
 Posted: Sun Feb 16, 2025 11:22 am Post subject: |
 |
|
| logrusx wrote: | | https://wiki.gentoo.org/wiki/Ccache |
| Code: | * Contents of dev-util/ccache-4.10.2-r1:
/usr
/usr/bin
/usr/bin/ccache
/usr/bin/ccache-config
/usr/share
/usr/share/doc
/usr/share/doc/ccache-4.10.2-r1
/usr/share/doc/ccache-4.10.2-r1/AUTHORS.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/CONTRIBUTING.md.lz
/usr/share/doc/ccache-4.10.2-r1/MANUAL.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/NEWS.adoc.lz
/usr/share/doc/ccache-4.10.2-r1/README.md.lz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/ccache.1.lz
/usr/share/shadowman
/usr/share/shadowman/tools
/usr/share/shadowman/tools/ccache
|
no portage-ccache |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Sun Feb 16, 2025 12:12 pm Post subject: |
|
|
| user wrote: | | Which package provide "portage-ccache" ? |
Nothing, it's just a script I made that does nothing more than automatically setting the CCACHE_DIR environment variable.
The important part is this part of /etc/portage/bashrc:
| Code: | if [[ ${FEATURES} == *ccache* && ${EBUILD_PHASE_FUNC} == src_* ]]; then
export CCACHE_BASEDIR="${PORTAGE_TMPDIR}/portage"
if [[ ${CCACHE_DIR} == /var/cache/portage/ccache ]]; then
if [[ ${CATEGORY} == kde-frameworks ]]; then
export CCACHE_DIR=${CCACHE_DIR}/${CATEGORY}
else
export CCACHE_DIR=${CCACHE_DIR}/${CATEGORY}/${PN}
fi
mkdir -p "${CCACHE_DIR}" || die
fi
fi |
Which assigns each package it's own private cache in /var/cache/portage/ccache for better statistics and monitoring.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 3031
|
| Posted: Sun Feb 16, 2025 4:41 pm Post subject: |
|
|
| mi_unixbird wrote: |
As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio. |
Where do you put that?
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Sun Feb 16, 2025 6:48 pm Post subject: |
|
|
| logrusx wrote: | | mi_unixbird wrote: |
As in: “prefix_command_cpp=/usr/local/libexec/ccache_pp_wrapper” with this simply executing “exec "$@" -P” and nothing more. It seems like this change makes a lot of packages which didn't work well with upgrades compile get a high hit ratio. |
Where do you put that?
Best Regards,
Georgi |
In the /etc/ccache.conf or local per-cache configuration.
https://ccache.dev/manual
The “prefix_command_cpp” option and the configuration style syntax is explained here.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 3031
|
| Posted: Mon Feb 17, 2025 5:33 pm Post subject: |
|
|
Thanks. I almost filled the cache, only a few thousand units remain until the compilation completes. We'll see what happens on the next release. I've never had good results with ccache, but I remain optimistic. No big hoes though.
EIDT: it finished. Cache size is 2.1 Gb. It even had 2 cache hits during the compilation. However I'm disappointed. Three years after they enabled Wayland support and Chromium still does not allow me to select more than 1080p YouTube video quality, while Firefox has been supporting it for quite a while now. I've abandoned Chromium for more than a year and I'm doing this out of curiosity only.
Best Regards,
Georgi |
|
| Back to top |
|
|
logrusx
Advocate
Joined: 22 Feb 2018
Posts: 3031
|
| Posted: Thu Feb 20, 2025 10:49 am Post subject: |
|
|
I don't know what I'm doing wrong but I've never got good cache hit ratios with chromium. Some people claimed they sometimes had good rates even without the change you suggested in the past, but I never did. I used ungoogled chromium for this experiment and I terminated it third of the way with abysmal 10% hit rate. Yes, 10% is more than I've ever had, but I don't think it's worth.
This time mesa was pulled in and I enabled cache for it too and I'll see how it does next update.
Best Regards,
Georgi |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Fri Feb 21, 2025 7:25 pm Post subject: |
|
|
| logrusx wrote: | I don't know what I'm doing wrong but I've never got good cache hit ratios with chromium. Some people claimed they sometimes had good rates even without the change you suggested in the past, but I never did. I used ungoogled chromium for this experiment and I terminated it third of the way with abysmal 10% hit rate. Yes, 10% is more than I've ever had, but I don't think it's worth.
This time mesa was pulled in and I enabled cache for it too and I'll see how it does next update.
Best Regards,
Georgi |
My ccache.conf for chromium looks like this, not sure whether you forgot a setting:
| Code: | # Maximum cache size to maintain
max_size = 5.0G
# Allow others to run 'ebuild' and share the cache.
umask = 002
# Preserve cache across GCC rebuilds and
# introspect GCC changes through GCC wrapper.
compiler_check = %compiler% -dumpversion
# I expect 1.5M files. 300 files per directory.
cache_dir_levels = 3
cache_dir = /var/cache/portage/ccache/www-client/chromium
file_clone=true
depend_mode=false
direct_mode=false
hash_dir=false
run_second_cpp=false |
I don't use the strip preprocessor hack for chromium because it doesn't do anything for it, I only use it for other packages.
The only time i get sub 10% hits is if some kind of fundamental dependency like the compiler itself updated and the entire cache is invalidated. With normal updates it rarely drops below 80%
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
keekkenen
n00b
Joined: 05 Oct 2024
Posts: 70
|
| Posted: Fri Feb 21, 2025 10:01 pm Post subject: |
|
|
| logrusx wrote: | | Three years after they enabled Wayland support and Chromium still does not allow me to select more than 1080p YouTube video quality, while Firefox has been supporting it for quite a while now. |
It's very strange, I've no problem with 1440p and 2160p in Chromium under plasma with wayland.
_________________
7950x3d / x670e MSI Tomagawk / Sapphire RX 7800XT 16Gb / G.Skill 64Gb 5600 / A-Data Legend 960 2Tb (x2), A-Data SX8200PNP 256Gb |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Sun Apr 06, 2025 12:12 am Post subject: |
|
|
By the way, I got two absolutely terrible cache hit ratios in a row now, quite quickly after each other too.
| Code: | —— — portage-ccache www-client/chromium -s
Cacheable calls: 41192 / 41237 (99.89%)
Hits: 8 / 41192 ( 0.02%)
Direct: 0 / 8 ( 0.00%)
Preprocessed: 8 / 8 (100.0%)
Misses: 41184 / 41192 (99.98%)
Uncacheable calls: 45 / 41237 ( 0.11%)
Local storage:
Cache size (GB): 5.0 / 5.0 (99.97%)
Cleanups: 1099
Hits: 8 / 41192 ( 0.02%)
Misses: 41184 / 41192 (99.98%)
|
I have no idea what could've updated in the meanwhile that caused this. The last one was like 5%, not 0.02% but I hope this doesn't persist due to some build system change that makes chromium non-viable with ccache from now on. Abseil seems like a prbably candidate, maybe it was that that caused it.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
keekkenen
n00b
Joined: 05 Oct 2024
Posts: 70
|
| Posted: Sun Apr 06, 2025 1:26 pm Post subject: |
|
|
Yes, I saw it. Existing ccache had no effect to cromium compilation. I don't know why.
_________________
7950x3d / x670e MSI Tomagawk / Sapphire RX 7800XT 16Gb / G.Skill 64Gb 5600 / A-Data Legend 960 2Tb (x2), A-Data SX8200PNP 256Gb |
|
| Back to top |
|
|
keekkenen
n00b
Joined: 05 Oct 2024
Posts: 70
|
| Posted: Sun Apr 06, 2025 10:58 pm Post subject: |
|
|
| mi_unixbird wrote: | By the way, I got two absolutely terrible cache hit ratios in a row now, quite quickly after each other too.
|
You have better static than me.
I cleaned ccache statistic before test and you can see below statistic after first and next compilation.
| Code: | Cacheable calls: 0 / 41880 ( 0.00%)
Hits: 0
Direct: 0
Preprocessed: 0
Misses: 0
Uncacheable calls: 46 / 41880 ( 0.11%)
Errors: 41834 / 41880 (99.89%)
Local storage:
Cache size (GB): 0.0 / 10.0 ( 0.00%) |
| Code: | Cacheable calls: 0 / 83779 ( 0.00%)
Hits: 0
Direct: 0
Preprocessed: 0
Misses: 0
Uncacheable calls: 111 / 83779 ( 0.13%)
Errors: 83668 / 83779 (99.87%)
Local storage:
Cache size (GB): 0.0 / 10.0 ( 0.00%) |
_________________
7950x3d / x670e MSI Tomagawk / Sapphire RX 7800XT 16Gb / G.Skill 64Gb 5600 / A-Data Legend 960 2Tb (x2), A-Data SX8200PNP 256Gb |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Thu Apr 10, 2025 2:28 am Post subject: |
|
|
New update again:
| Code: | Cacheable calls: 41192 / 41237 (99.89%)
Hits: 39697 / 41192 (96.37%)
Direct: 0 / 39697 ( 0.00%)
Preprocessed: 39697 / 39697 (100.0%)
Misses: 1495 / 41192 ( 3.63%)
Uncacheable calls: 45 / 41237 ( 0.11%)
Local storage:
Cache size (GB): 5.0 / 5.0 (99.99%)
Cleanups: 40
Hits: 39728 / 41223 (96.37%)
Misses: 1495 / 41223 ( 3.63%) |
That's what I'm talking about. It seems to still work fine if specific packages aren't getting updated.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
keekkenen
n00b
Joined: 05 Oct 2024
Posts: 70
|
| Posted: Thu Apr 10, 2025 11:01 pm Post subject: |
|
|
I changed options from MAKEOPTS="-j28 -l28" to MAKEOPTS="-j31 -l31" and got worse last compile time.
| Code: | 2025-04-06T17:06:04 >>> www-client/chromium: 1:47:38
2025-04-10T00:11:49 >>> www-client/chromium: 1:55:32 |
It's just funny.
_________________
7950x3d / x670e MSI Tomagawk / Sapphire RX 7800XT 16Gb / G.Skill 64Gb 5600 / A-Data Legend 960 2Tb (x2), A-Data SX8200PNP 256Gb |
|
| Back to top |
|
|
mi_unixbird
Tux's lil' helper
Joined: 24 Jul 2015
Posts: 147
|
| Posted: Thu Apr 17, 2025 3:40 am Post subject: |
|
|
| keekkenen wrote: | I changed options from MAKEOPTS="-j28 -l28" to MAKEOPTS="-j31 -l31" and got worse last compile time.
| Code: | 2025-04-06T17:06:04 >>> www-client/chromium: 1:47:38
2025-04-10T00:11:49 >>> www-client/chromium: 1:55:32 |
It's just funny. |
Just how many cores do you have that this is a good idea?
Also:
| Code: | Cacheable calls: 82384 / 82474 (99.89%)
Hits: 76679 / 82384 (93.08%)
Direct: 0 / 76679 ( 0.00%)
Preprocessed: 76679 / 76679 (100.0%)
Misses: 5705 / 82384 ( 6.92%)
Uncacheable calls: 90 / 82474 ( 0.11%)
Local storage:
Cache size (GB): 5.0 / 5.0 (99.97%)
Cleanups: 224
Hits: 76713 / 82418 (93.08%)
Misses: 5705 / 82418 ( 6.92%)
|
I didn't empty the statistics this time so some basic maths is required but it seems to have been around 90% hit rate this time opposed to the 96% last time which is still good of course. I wish ccache stats allowed one to just see the stats between certain dates without having to zero them.
In any case, though obviously some updates alter something big that essentially cause the entire cache to come invalidated. I think I have firmly shown at this point that ccache is absolutely effective with Chromium between updates, being able to cut down 10 hour compiles to 2.5 with the right setup which involves not hashing the current working directory. This will of course make debug information print the wrong version number at times in terms of what file causes some error but I don't think anyone really caress about that. ccache's default setup prioritizes correctness above everything else.
_________________
execctl --path exec filectl --current-directory list |
|
| Back to top |
|
|
|
Portage & Programming
