Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Massive speedup using ccache in disk backed RAMFS
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sat Jul 31, 2010 4:13 pm    Post subject: Massive speedup using ccache in disk backed RAMFS Reply with quote

It was driven mostly by needs (think speed) and what means I already had (think 12GB RAM). Here is what I needed:

1. ccache gains are good on a fast SSD but it trashes the FS and the disk (SSDs are sensitive to write amplification, resulting in lower life and performance).
2. It should not slow down the first emerge, where there is no hit in ccache. If you use normal HDD, you will see that without ccache, the first compile goes faster. If ccache is not going to give me a lot of hits, this means that I am doing additional work of storing that data and losing performance overall in long term.

Only way to get these two requirements to satisfy was to let go of 1.5GB (size of my ccache) of RAM for good. Put the ccache on the tmpfs and find ways to sync it to disk only on shutdown and load into RAM on startup. Here is what I did:

1. /etc/make.conf
Code:
# ccache in RAMFS
CCACHE_DIR=/var/tmp/ccache


2. /etc/fstab
Code:

# for faster builds.
none                                    /var/tmp                tmpfs   size=6856M,nr_inodes=1M         0 0


3. Restore script
Code:
$ cat ccache-restore
#!/bin/sh
if [[ ! -d /var/tmp/ccache ]]
then
        mkdir /var/tmp/ccache
fi
rsync -av -H -x /usr/portage/local/ccache/x64/ /var/tmp/ccache/ > /dev/null 2>/var/log/ccache-restore.log
touch /var/tmp/ccache/.restored


4. Store script
Code:
$ cat ccache-store
#!/bin/sh
if [[ -f /var/tmp/ccache/.restored ]]
then
        rsync -av -H -x /var/tmp/ccache/ /usr/portage/local/ccache/x64/ > /dev/null 2>/var/log/ccache-store.log
fi


5. Put restore in /etc/conf.d/local.start
Code:
# restore ccache in a lazy fashion
ccache-restore &
disown %1


6. Put store in /etc/conf.d/local.stop
Code:
# store ccache on disk
ccache-store


Results:

Code:

     Sat Jul 31 00:34:39 2010 >>> net-libs/xulrunner-1.9.2.8
       merge time: 4 minutes and 24 seconds.                                                                                                       

     Sat Jul 31 00:37:24 2010 >>> net-libs/xulrunner-1.9.2.8
       merge time: 1 minute and 17 seconds.


Code:
     Sat Jul 31 00:08:31 2010 >>> mail-client/thunderbird-3.1.1
       merge time: 4 minutes and 21 seconds.                                                                                                       

     Sat Jul 31 00:10:19 2010 >>> mail-client/thunderbird-3.1.1
       merge time: 1 minute and 13 seconds.

Code:
     Sat Jul 31 08:47:32 2010 >>> media-video/ffmpeg-0.6
       merge time: 41 seconds.                                                                                                                                             

     Sat Jul 31 08:47:55 2010 >>> media-video/ffmpeg-0.6
       merge time: 15 seconds.


Achieved:

1. No small random writes to SSD until shutdown, which is many days or even weeks for me because I suspend-to-ram.
2. No loss during first run.
3. Massive gain on hits.

Caveats:

Do not try this if you don't have hardware to support it.
You need a modern enough multi-core CPU to make sure that you can calculate hashes orders of magnitude faster than you can compile.
You need at least 6GB of RAM (and that may be small and you may want to reduce the ccache size).
And you need a fast SSD for "/" if you don't want your shutdown to take forever and your startup to slow down because of IO.

I will post the same numbers for openoffice build in a short while.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sat Jul 31, 2010 4:57 pm    Post subject: Reply with quote

Code:
     Thu Jul 29 18:36:12 2010 >>> app-office/openoffice-3.2.1
       merge time: 27 minutes and 19 seconds.                                                                                   

     Sat Jul 31 09:50:34 2010 >>> app-office/openoffice-3.2.1
       merge time: 6 minutes and 58 seconds.
What remains to be seen is when openoffice-3.2.2 comes out, whether my system will compile it in 27 mins or 7 mins. Likely somewhere in the middle.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Sat Sep 11, 2010 3:17 am    Post subject: Reply with quote

This came in handy while moving from the overlay based kde to portage based.
Code:
     Tue Aug 31 18:15:36 2010 >>> kde-base/kdelibs-4.5.1
       merge time: 5 minutes and 25 seconds.

     Fri Sep 10 20:13:16 2010 >>> kde-base/kdelibs-4.5.1
       merge time: 1 minute and 41 seconds.
The whole kde 4.5.1 (at least whatever I am interested in) emerged in like 12 minutes.
Back to top
View user's profile Send private message
neuron
Advocate
Advocate


Joined: 28 May 2002
Posts: 2371

PostPosted: Sat Sep 11, 2010 7:14 pm    Post subject: Reply with quote

That's... crazy. With speedups that high I wouldn't be surprised if compcache + less ram would also be well worth it.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2995
Location: Bay Area, CA

PostPosted: Wed Sep 22, 2010 7:31 am    Post subject: Reply with quote

Ok, this just got more interesting with latest feature in compcache or zram as its called now in latest kernel.

Basically, instead of creating /var/tmp as tmpfs, create a compressed block device using zram, create ext4 on it and use that to mount on top of /var/tmp. The advantage is that for 1.3Gig of ccache, it compresses it down to 500MB i.e. only 500MB of real RAM is used. You get a little bit of overhead on CPU but I have a pretty powerful CPU in i7 920. So, that's fine!

Here is what I did:

Code:
#!/bin/sh

# compcache is not there, get out!
ramz=`find /lib/modules/$(uname -r)/kernel/drivers/ -name zram.ko 2>/dev/null`
[ -z "$ramz" ] && exit 0

# turn it on if nothing passed in as args
if [ -z "$1" ]
then
        # load the zram devices
        modprobe zram num_devices=2
        sleep 1
        # 7GB of /var/tmp
        echo $((7*1024*1024*1024)) > /sys/block/zram0/disksize
        # 1GB of compressed swap
        echo $((1*1024*1024*1024)) > /sys/block/zram1/disksize
        mkswap /dev/zram1
        swapon -p 100 /dev/zram1
        mkfs.ext4 -m0 -N 1000000 -L "zram-vol" /dev/zram0
        mount -o noatime /dev/zram0 /var/tmp
else
        lsmod | grep -q "^zram"
        if [ $? -ne 0 ]
        then
                exit 0
        fi
        # unload the swap devices
        if [ -b /dev/zram1 ]
        then
                swapoff /dev/zram1
                sleep 0.2
                echo 1 > /sys/block/zram1/reset
        fi
        if [ -b /dev/zram0 ]
        then
                umount /dev/zram0
                sleep 0.2
                echo 1 > /sys/block/zram0/reset
        fi
        modprobe -r zram
fi
I put this in /usr/bin/compcache, and I invoke compcache from /etc/conf.d/local.start and 'compcache off' from /etc/conf.d/local.stop. Make sure to save the ccache before calling 'compcache off' in local.stop and restore it after calling 'compcache' in local.start.

This is the stats I got after restoring ccache to zram-vol mounted at /var/tmp:

Code:
# ~/bin/zram_stats
/sys/block/zram0
disksize:         7516192768
num_reads:               188
num_writes:           423258
invalid_io:                0
notify_free:               0
discard:                   0
zero_pages:            88848
orig_data_size:   1324761088
compr_data_size:   471440303
mem_used_total:    481484800
avg_compr_ratio:          35 %
mem_overhead:              2 %


Check it out:
Code:
hg clone https://compcache.googlecode.com/hg/ compcache
cd compcache
vi README
Back to top
View user's profile Send private message
vh4x0r
n00b
n00b


Joined: 11 May 2009
Posts: 37

PostPosted: Wed Nov 17, 2010 10:20 am    Post subject: Reply with quote

Just stopped by to say that local.start and local.stop are deprecated since openrc-0.6.4. So steps 5 and 6 should be modified to add the corresponding scripts in the local_start and local_stop functions of /etc/conf.d/local.
Code:
local_start() {
   # This is a good place to load any misc programs
   # on startup (use &>/dev/null to hide output)

   # Restore ccache in a lazy fashion
   /usr/local/bin/ccache-restore &
   disown %1

   # We should always return 0
   return 0
}

local_stop() {
   # This is a good place to unload any misc.
   # programs you started above.

   # Save ccache to disk
   /usr/local/bin/ccache-store
   
   # We should always return 0
   return 0
}

_________________
main () { while (true) fork (); }
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum