Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Speed up recompilation with ccache
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Sat Jun 22, 2002 4:26 am    Post subject: Speed up recompilation with ccache Reply with quote

Update: from portage-2.0.7 up, CCACHE_DIR must be set as /var/tmp/ccache instead of /var/cache/ccache.

From the ccache ebuild:
Quote:
Description: ccache is a fast compiler cache. It is used as a front end to your compiler to safely cache compilation output. When the same code is compiled again the cached output is used giving a significant speedup.

From the ccache manual page:
Quote:
INSTALLATION

There are two ways to use ccache. You can either prefix your compile commands with "ccache" or you can create a symbolic link between ccache and the names of your compilers. The first method is most convenient if you just want to try out ccache or wish to use it for some specific projects. The second method is most useful for when you wish to use ccache for all your compiles.

To install for usage by the first method just copy ccache to somewhere in your path.

To install for the second method do something like this:
Code:
cp ccache /usr/local/bin/
ln -s /usr/local/bin/ccache /usr/local/bin/gcc
ln -s /usr/local/bin/ccache /usr/local/bin/cc

This will work as long as /usr/local/bin comes before the path to gcc (which is usually in /usr/bin).
(...)

HOW IT WORKS

The basic idea is to detect when you are compiling exactly the same code a 2nd time and use the previously compiled output.
(...)


These are the steps to use ccache in Gentoo:

1. Update Portage, if needed. You'll need Portage version 2.0.2 or later (this is a sandbox issue).

2. Install ccache. (It's worth noticing that ccache has no dependencies, so it can be installed quite early, even before the "emerge system" install step! I wouldn't recommend installing it before bootstraping, though.)

3. Read the ccache manual.

4. Create the symbolic links:
Code:
# ln -s /usr/bin/ccache /bin/gcc
# ln -s /usr/bin/ccache /bin/cc
# ln -s /usr/bin/ccache /bin/g++
# ln -s /usr/bin/ccache /bin/c++

5. Create the ccache cache dir:
Code:
# mkdir /var/tmp/ccache

6. Add the following line to /etc/profile:
Code:
export CCACHE_DIR="/var/tmp/ccache"

7. Log out and back in, or manually run
Code:
# export CCACHE_DIR="/var/tmp/ccache"

8. Be happy!


I made some tests to see if it is really worthy. I ran the following commands twice with ccache disabled and three times with ccache enabled:
Code:
# ebuild /usr/portage/net-www/mozilla/mozilla-1.0-r1.ebuild clean
# ebuild /usr/portage/net-www/mozilla/mozilla-1.0-r1.ebuild unpack
# time ebuild /usr/portage/net-www/mozilla/mozilla-1.0-r1.ebuild compile

The results were, in order:
Code:
real    36m47.153s
user    32m57.496s
sys     2m18.362s

real    36m45.588s
user    32m59.537s
sys     2m17.597s

real    38m5.326s
user    33m10.850s
sys     2m22.291s

real    10m54.726s
user    6m1.358s
sys     1m37.028s

real    10m30.090s
user    5m42.772s
sys     1m37.005s


You can draw your own conclusions... :)
_________________
| www.gentoo.org | www.tldp.org | www.google.com |


Last edited by fghellar on Sun Jul 21, 2002 4:05 pm; edited 1 time in total
Back to top
View user's profile Send private message
lamer
n00b
n00b


Joined: 16 May 2002
Posts: 16
Location: St. Paul, Mn

PostPosted: Sat Jun 22, 2002 3:59 pm    Post subject: dude Reply with quote

what are you compiling this on that mozilla only takes 38 minutes?
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Sat Jun 22, 2002 5:30 pm    Post subject: Reply with quote

Athlon XP1700, 384 Mb PC2100.
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
asimon
l33t
l33t


Joined: 27 Jun 2002
Posts: 979
Location: Germany, Old Europe

PostPosted: Thu Jun 27, 2002 3:08 pm    Post subject: Reply with quote

Do those ccache timings with newer KDE sources and gcc 3.1 and you'll get completly different numbers ;-)

I tested ccache for compiling cvs snapshots of kde 3.0.5 with gcc 3.1.
99% of all items in ccache's statistics were: "Compiler option not supported".
The compiler option in question was some obscure pre-processor flag which KDE uses when it detects gcc 3. So ccache was useless for me.

Cheers,
Andreas
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Thu Jun 27, 2002 7:23 pm    Post subject: Reply with quote

Can you provide more information about this? For example, what do you mean with "obscure pre-processor flag"? Did you link the gcc 3 binaries to ccache (they get different names if gcc 2 is installed too)? etc...
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
rukhoven
n00b
n00b


Joined: 18 Apr 2002
Posts: 54
Location: The flat lands of nether elevation

PostPosted: Thu Jun 27, 2002 7:31 pm    Post subject: Reply with quote

If you want to compile something with other cflags, then it will use your previously compiled code however wont it? It seems it will not be usefull in such a case...
_________________
*****************************
AMDXp 2000 per sis745 con 60 GB HD
......Is what I'd like to have....... :)
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Thu Jun 27, 2002 7:44 pm    Post subject: Reply with quote

That would be a really stupid program, don't you think? :)

Anyways, step 3 above should make it clear to you, esp. the section "How it works". (You can find it by going to Google and typing "ccache manual", if you don't have it installed.)
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Thu Jun 27, 2002 8:03 pm    Post subject: Reply with quote

Important update:

I just read in the gentoo-dev mailing list that there's been problems with ccache and newer versions of Portage. I'll see how it goes and come back with any related info.

Link: http://lists.gentoo.org/pipermail/gentoo-dev/2002-June/013248.html
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
asimon
l33t
l33t


Joined: 27 Jun 2002
Posts: 979
Location: Germany, Old Europe

PostPosted: Thu Jun 27, 2002 8:30 pm    Post subject: Reply with quote

fghellar wrote:
Can you provide more information about this? For example, what do you mean with "obscure pre-processor flag"? Did you link the gcc 3 binaries to ccache (they get different names if gcc 2 is installed too)? etc...


I always got "argument -MT is unsupported" from ccache. This options gets added by KDEs building system when you use gcc 3. When ccache encountered this option in a compile command (and that was the case 99%) the compilation output was not put into the cache. After compiling most of KDEs modules (build dir > 2.5 GB) the cache of ccache was only 20 MB big, so it was quite useless. After two more compilations of the whole kde stuff I only had around 100 cache hits. But it was great when I used gcc 2.96 to compile KDE.

After some compilations you can check with ccache -s how efficent the cache was.

Cheers,
Andreas
Back to top
View user's profile Send private message
rukhoven
n00b
n00b


Joined: 18 Apr 2002
Posts: 54
Location: The flat lands of nether elevation

PostPosted: Thu Jun 27, 2002 8:42 pm    Post subject: Reply with quote

fghellar wrote:
That would be a really stupid program, don't you think? :)

Anyways, step 3 above should make it clear to you, esp. the section "How it works". (You can find it by going to Google and typing "ccache manual", if you don't have it installed.)


So when I change the CFLAGS, it will not use the cached stuff if I recompile a program? Ok you seem sure of that, but I couldn't interpret the manual in such way to rule that out/in ... :) Well I guess I'll just try that. Would some code be compiled more than once when compiling say, kdelibs ?
_________________
*****************************
AMDXp 2000 per sis745 con 60 GB HD
......Is what I'd like to have....... :)
Back to top
View user's profile Send private message
asimon
l33t
l33t


Joined: 27 Jun 2002
Posts: 979
Location: Germany, Old Europe

PostPosted: Thu Jun 27, 2002 8:57 pm    Post subject: Reply with quote

Quote:

So when I change the CFLAGS, it will not use the cached stuff if I recompile a program? If ccache supports the flags which get used it works. Usually the options you put in CFLAGS are no problem for ccache. Would some code be compiled more than once when compiling say, kdelibs ?


If ccache can determine that a given compilation happend before (i.e. would result in the very same output as is already in the cache) the output in the cache is used and the compilation is not done a second time. But if you change code in the file, or change compile options, etc. the output of the new compilation would not be the same. Thus ccache cannot use the one in it s cache.

Cheers,
Andreas
Back to top
View user's profile Send private message
Damasz
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2002
Posts: 91
Location: Groningen, the Netherlands

PostPosted: Sat Jun 29, 2002 2:56 pm    Post subject: Reply with quote

Wouldn't it be cool if the cache could be shared on a p2p network between gentoo users? (and other linux users....)
You could download the files instead of compiling them if another user with the same settings as you allready compiled the program.

Then the biggest problem a lot of people have with gentoo (long compile time) would be gone! Sweet.....

Gert-Jan.
Back to top
View user's profile Send private message
ph317
n00b
n00b


Joined: 02 Jun 2002
Posts: 43

PostPosted: Wed Jul 03, 2002 6:02 pm    Post subject: Reply with quote

*edit* there was a long post here about ideas for a global network-wide compiler cache - after thinking over it for a weekend I decided it was a dumb idea for a whole host of reasons, and deleted the post so that nobody else wasted brain cycles reading it and thinking about it *edit* :)

Last edited by ph317 on Mon Jul 08, 2002 5:27 pm; edited 1 time in total
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Thu Jul 04, 2002 4:46 am    Post subject: Reply with quote

Damasz wrote:
You could download the files instead of compiling them if another user with the same settings as you allready compiled the program.

Not always a good idea: https://forums.gentoo.org/viewtopic.php?p=33015.
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
tfoss
n00b
n00b


Joined: 04 Jul 2002
Posts: 8

PostPosted: Fri Jul 05, 2002 8:29 pm    Post subject: Reply with quote

Just a warning:

I spent a couple hours trying to figure out why emerging kdelibs kept giving some odd libSTL... not found error. Turns out unmerging ccache fixed it.

-Ted
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Sat Jul 06, 2002 5:05 am    Post subject: Reply with quote

Can you check if it has anything to do with this?
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
micron
Guru
Guru


Joined: 23 Jul 2003
Posts: 411
Location: Bergamo, Italy

PostPosted: Sat Sep 11, 2004 11:48 am    Post subject: Reply with quote

I've followed all the steps but
Code:
/var/tmp/ccache

is empty... 8O
what can I do?
_________________
~ "Progress is merely a realisation of utopias" ~
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Sun Sep 12, 2004 8:58 pm    Post subject: Reply with quote

micron wrote:
I've followed all the steps but
Code:
/var/tmp/ccache

is empty... 8O
what can I do?


This is a VERY old post.

the ccache feature has now been buil into portage.

here:
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3#doc_chap2
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
micron
Guru
Guru


Joined: 23 Jul 2003
Posts: 411
Location: Bergamo, Italy

PostPosted: Sun Sep 12, 2004 11:39 pm    Post subject: Reply with quote

MooktaKiNG wrote:

This is a VERY old post.

the ccache feature has now been buil into portage.

here:
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=3#doc_chap2

Thanks a lot!
_________________
~ "Progress is merely a realisation of utopias" ~
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum