Chapter 22. Caching Built Files

On multi-developer software projects, you can sometimes speed up every developer's builds a lot by allowing them to share a cache of the derived files that they build. After all, it is relatively rare that any in-progress change affects more than a few derived files, most will be unchanged. Using a cache can also help an individual developer: for example if you wish to start work on a new feature in a clean tree, those build artifacts which could be reused can be retrieved from the cache to populate the tree and save a lot of initial build time. SCons makes this easy and reliable.

22.1. Specifying the Derived-File Cache Directory

To enable caching of derived files, use the CacheDir function in any SConscript file:


The cache directory you specify must have read and write access for all developers who will be accessing the cached files (if --cache-readonly is used, only read access is required). It should also be in some central location that all builds will be able to access. In environments where developers are using separate systems (like individual workstations) for builds, this directory would typically be on a shared or NFS-mounted file system. While SCons will create the specified cache directory as needed, in this multi user scenario it is usually best to create it ahead of time so the access rights can be set up correctly.

Here's what happens: When a build has a CacheDir specified, every time a file is built, it is stored in that cache directory indexed by its build signature. On subsequent builds, before an action is invoked to build a file, the build signature is computed and SCons checks the derived-file cache directory to see if a file with the exact same build signature already exists. [4] If so, the derived file will not be built locally, but will be copied into the local build directory from the derived-file cache directory, like this:

% scons -Q
cc -o hello.o -c hello.c
cc -o hello hello.o
% scons -Q -c
Removed hello.o
Removed hello
% scons -Q
Retrieved `hello.o' from cache
Retrieved `hello' from cache

Note that the CacheDir feature requires that the build signature be calculated, even if you configure SCons to use timestamps to decide if files are up to date (see the Chapter 6, Dependencies chapter for information about the Decider function), since the build signature is used to determine if a target file exists in the cache. Consequently, using CacheDir may reduce or negate any performance improvements from using timestamps for up-to-date decisions.

[4] A few inside details: SCons tracks two main kinds of cryptographic hashes: a content signature, which is a hash of the contents of a file participating in the build (dependencies as well as targets); and a build signature, which is a hash of the elements needed to build a target, such as the command line, the contents of the sources, and possibly information about tools used in the build. The hash function produces a unique signature from its inputs, no other set of inputs can produce that same signature. The build signature from building a target is used as the filename of the target file in the derived-file cache - that way lookups are efficient, just compute a build signature and see if a file exists with that as the name.

The use of the build signature provides protection from concflicts: if two developers have different setups, so they would produce built objects that are not identical, then because the difference in tools will show up in the build signature, which is used as the name of the cache entry, they will end up being stored as separate entries.