Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Per-package ccache-dir support in portage? (proposal)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
katabami
n00b
n00b


Joined: 11 Jul 2009
Posts: 40

PostPosted: Tue Feb 15, 2011 3:35 am    Post subject: Per-package ccache-dir support in portage? (proposal) Reply with quote

Hi. I think per-package ccache-dir support in portage will be nice. Suppose, when a new option CCACHE_SPLIT is set to "yes", ccache is stored in <ccache_root>/<category>/<package>. It's good because:

* When the ccache repo is corrupt, you only have to delete the cache for that package.
* If portage prints the 'ccache -s' output before and after building, users can tune each cache size. Smaller cache is better, isn't it? (I don't know hash well, though.) From my experience, (initial use of cache) * 1.5 is a good size.

Personally, I don't use ccache much, because I don't do periodic "emerge -e world". On the other hand, I develop some packages, and I have a wrapper of emerge for these packages, like

Code:
BEFORE=`ccache -s | grep "\(hit\|miss\)"`

emerge -v mypacks 2>&1 | tee $LOG

AFTER=`ccache -s | grep "\(hit\|miss\|size\)"`
echo "${BEFORE}
----------
${AFTER}" | tee -a $LOG


(For pipes, "set -o pipefail" is nice, but I omitted it here.)

If many approves, I'll send a feature request to the bugzilla. Or if you can send them a patch, it's great.

Thanks.
Back to top
View user's profile Send private message
katabami
n00b
n00b


Joined: 11 Jul 2009
Posts: 40

PostPosted: Tue Feb 22, 2011 5:37 am    Post subject: Reply with quote

I've got 2 new ideas, both of which depend on per-package ccache.

First, an option to use ccache only for rebuilding, or -r[0-9] upgrades. Otherwise, the cache is cleaned before emerging. Of course there'll be less cache hits. But ccache doesn't mean much between bumped versions.

For example, I turned on ccache for minimal upgrading of openssl, 1.0.0c -> 1.0.0d, and only 177/703 hits were there. And build time took longer with ccache. Other processes were running, so not accurate, though. From my experience, low hit rate, say 30%, doesn't improve the build time.

Upgrading of openssh 5.6_p1-r2 -> 5.8_p1 was a minimal bump in amd64 stable, and only 16/307 hits were seen.

My wild guesses (don't take too seriously) are:
  • Smaller files tend to hit the cache more often.
  • Compilation time is roughly proportional to the file size. OTOH cache seeking and reading overhead doesn't vary much, which is slower by orders. Remember that reading is blocking. So smaller files are far less efficient even with cache hit.
Cache access is slow. Deletion of ccache repository of 10M took me seconds on xfs. (I know ext4 is far better than xfs, especially in file deletion.)

So the second idea: Save the cache in tar file without compression. One tar for each package. When emerge begins, extract ccache files to tmpfs. When emerge is done, make a new tar, and replace the old one. So only one read and one write of HDD per package.

This trick may become soon obsolete in SSD era. I don't think that complicated procedure is good, but a good food for thought. :)
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9013
Location: beyond the rim

PostPosted: Tue Feb 22, 2011 9:00 am    Post subject: Reply with quote

I think this is stuff for hooks, not for integration into the package manager (never been a fan of the distcc and ccache stuff in portage).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum