..or why/how to install and use CCache 3.0pre0 in Scratchbox
A few days ago, I was building and rebuilding a Debian package in Scratchbox to see if it builds, installs and works properly after my modifications. To make sure it works, I had to do a full dpkg-buildpackage -rfakeroot rebuild, which of course does a full clean every time. Each rebuild took 10 minutes, and I started wondering how to make it go faster.
Use ccache 3.0 and make sure your regular build does not use compiler options incompatible with ccache! Read on for details.
Enter ccache 3.0pre0
Ccache is a compiler cache system developed by the Samba team: http://ccache.samba.org/
Of course I knew that by default, Scratchbox uses ccache to speed up building. I had disabled it earlier since I suspected it caused some problems for me. But I wanted speed, so I enabled it again. After some more rebuilding I started thinking that it’s not doing a very good job – it was still very slow. I checked the version of ccache supplied with Scratchbox – it was 2.3, from 2002. I checked and yes, there is a newer pre-release version from this year – 3.0pre0 and I set out to see if it would be faster.
Because the software I was rebuilding is proprietary, for the purposes of this article I fished out for something publicly available and did the same benchmarking and tweaking using that. I selected the game OpenTTD, and the Harmattan UI Framework because of their size – something that really benefits from rebuild speedups. I chose to build them using the Maemo 5 SDK in Scratchbox.
My home system is a 2.4GHz dual-core Core 2 Duo with 3 gigabytes of RAM running Ubuntu 10.04. To minimize effects of filesystem cache and any random load variations from background processes etc., I ran each package build three times. One set of three with ccache disabled, another with the default ccache 2.3 enabled, and a third set with ccache 3.0pre0 enabled. When switching between ccache versions, I executed "rm -rf /scratchbox/ccache/*" outside Scratchbox to make sure there is no version mismatch between the cache files of different ccache versions. Packages were built with the command "dpkg-buildpackage -rfakeroot". The parameter -j2 was added for the make options of Harmattan UI Framework to take advantage of my dual-core processor for parallelized compilation.
See the diagrams above. Upgrading to ccache 3.0pre0 yielded some performance boost, but in my case the most important finding was our own project’s usage of incompatible compiler options. Getting rid of those yielded the greatest benefits for our project.
Incompatible compiler options !
In the proprietary software I initially started experimenting ccache with, I found out that while some parts of the rebuild became faster than lightning, for some other files there was no change. I checked the ccache status with "ccache -s" and found out there were hundreds of occurrences of "Unsupported Compiler Option". I realized that the unsupported compiler option cases were all in our unit test suite and soon understood that it was the profiling options (-ftest-coverage) that were the incompatible ones. I removed those from our unittest compiler options, and now ccache was able to cache every object file produced by our build! The speed difference for second and n’th builds was incredible, since every single object file came from the cache. The whole source code set was rebuilt in 20 seconds, down from 10 minutes.
For profiling use I will now make a separate build with the profiler options enabled. It’s better to optimize for the normal case than the exception.
TODO: Upgrading cache the right way
I used the lazy way to install ccache 3.0 into Scratchbox – I built it as a static binary: "export CFLAGS=-static -Os" before running configure-dev. I then copied the resulting ccache binary into /scratchbox/tools/bin. This has the problem that it doesn’t work for ARM builds! In the coming days I will download the Scratchbox sources and check what is the proper way to build ccache for the Scratchbox environment so that it works regardless of the target architecture.