View previous topic :: View next topic |
Author |
Message |
katabami n00b
Joined: 11 Jul 2009 Posts: 40
|
Posted: Tue Feb 15, 2011 3:35 am Post subject: Per-package ccache-dir support in portage? (proposal) |
|
|
Hi. I think per-package ccache-dir support in portage will be nice. Suppose, when a new option CCACHE_SPLIT is set to "yes", ccache is stored in <ccache_root>/<category>/<package>. It's good because:
* When the ccache repo is corrupt, you only have to delete the cache for that package.
* If portage prints the 'ccache -s' output before and after building, users can tune each cache size. Smaller cache is better, isn't it? (I don't know hash well, though.) From my experience, (initial use of cache) * 1.5 is a good size.
Personally, I don't use ccache much, because I don't do periodic "emerge -e world". On the other hand, I develop some packages, and I have a wrapper of emerge for these packages, like
Code: | BEFORE=`ccache -s | grep "\(hit\|miss\)"`
emerge -v mypacks 2>&1 | tee $LOG
AFTER=`ccache -s | grep "\(hit\|miss\|size\)"`
echo "${BEFORE}
----------
${AFTER}" | tee -a $LOG |
(For pipes, "set -o pipefail" is nice, but I omitted it here.)
If many approves, I'll send a feature request to the bugzilla. Or if you can send them a patch, it's great.
Thanks. |
|
Back to top |
|
|
katabami n00b
Joined: 11 Jul 2009 Posts: 40
|
Posted: Tue Feb 22, 2011 5:37 am Post subject: |
|
|
I've got 2 new ideas, both of which depend on per-package ccache.
First, an option to use ccache only for rebuilding, or -r[0-9] upgrades. Otherwise, the cache is cleaned before emerging. Of course there'll be less cache hits. But ccache doesn't mean much between bumped versions.
For example, I turned on ccache for minimal upgrading of openssl, 1.0.0c -> 1.0.0d, and only 177/703 hits were there. And build time took longer with ccache. Other processes were running, so not accurate, though. From my experience, low hit rate, say 30%, doesn't improve the build time.
Upgrading of openssh 5.6_p1-r2 -> 5.8_p1 was a minimal bump in amd64 stable, and only 16/307 hits were seen.
My wild guesses (don't take too seriously) are:- Smaller files tend to hit the cache more often.
- Compilation time is roughly proportional to the file size. OTOH cache seeking and reading overhead doesn't vary much, which is slower by orders. Remember that reading is blocking. So smaller files are far less efficient even with cache hit.
Cache access is slow. Deletion of ccache repository of 10M took me seconds on xfs. (I know ext4 is far better than xfs, especially in file deletion.)
So the second idea: Save the cache in tar file without compression. One tar for each package. When emerge begins, extract ccache files to tmpfs. When emerge is done, make a new tar, and replace the old one. So only one read and one write of HDD per package.
This trick may become soon obsolete in SSD era. I don't think that complicated procedure is good, but a good food for thought. :) |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9522 Location: beyond the rim
|
Posted: Tue Feb 22, 2011 9:00 am Post subject: |
|
|
I think this is stuff for hooks, not for integration into the package manager (never been a fan of the distcc and ccache stuff in portage). |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|