Chapter 22. Caching Built Files

On multi-developer software projects, you can sometimes speed up every developer's builds a lot by allowing them to share the derived files that they build. After all, it is relatively rare that any in-progress change affects more than a few derived files, most will be unchanged. SCons makes this easy and reliable.

22.1. Specifying the Shared Cache Directory

To enable sharing of derived files, use the CacheDir function in any SConscript file:


The cache directory you specify must be readable and writable by all developers who will be sharing derived files. It should also be in some central location that all builds will be able to access. In environments where developers are using separate systems (like individual workstations) for builds, this directory would typically be on a shared or NFS-mounted file system. While SCons will create the specified cache directory as needed, in this multi user scenario it is usually best to create it ahead of time so the access rights can be set up correctly.

Here's what happens: When a build has a CacheDir specified, every time a file is built, it is stored in that cache directory along with its build signature. On subsequent builds, before an action is invoked to build a file, SCons will check the shared cache directory to see if a file with the exact same build signature already exists. [4] If so, the derived file will not be built locally, but will be copied into the local build directory from the shared cache directory, like this:

% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q
Retrieved `hello.o' from cache
Retrieved `hello' from cache

Note that the CacheDir feature requires that build signatures be calculated, even if you configure SCons to use timestamps to decide if files are up to date (see the Chapter 6, Dependencies chapter for information about the Decider function), since the build signature is used to determine if a target file exists in the cache. Consequently, using CacheDir may reduce or eliminate any potential performance improvements from using timestamps for up-to-date decisions.

[4] A few inner details: SCons tracks two main kinds of cryptographic hashes: content signatures, which are a hash of the contents of a file; and build signatures, which are a hash of the elements needed to build a target, such as the exact command line, the contents of the sources, and possibly information about tools used in the build. The hash function produces a unique signature from its inputs, no other set of inputs can produce that same signature. The build signature from building a target is used as the filename of the target file in the shared cache - that way lookups are efficient, just compute a build signature and see if a file exists with that as the name.