SCons 4.4.0 is available

A new SCons release, 4.4.0, is now available on the SCons download page:

https://scons.org/pages/download.html

Here is a summary of the changes since 4.3.0:

NOTE: If you build with Python 3.10.0 and then rebuild with 3.10.1 (or higher), you may
see unexpected rebuilds. This is due to Python internals changing which changed the signature of a Python Action Function.

NEW FUNCTIONALITY

  • Added MSVC_USE_SCRIPT_ARGS Environment variable which specifies command line arguments to pass to the script specified by MSVC_USE_SCRIPT.
  • Added Configure.CheckMember() checker to check if struct/class has the specified member.
  • Added SHELL_ENV_GENERATORS construction variable. This variable should be set to a list (or an iterable) which contains functions to be called in order when constructing the execution environment (Generally this is the shell environment variables). This allows the user to customize how (for example) PATH is constructed. Note that these are called for every build command run by SCons. It could have considerable performance impact if not used carefully.
  • Added MSVC_USE_SETTINGS construction variable to pass a dictionary to configure the msvc compiler system environment as an alternative to bypassing Visual Studio autodetection entirely.
  • Added MSVC_SDK_VERSION construction variable which allows building with a specific Microsoft SDK version. This variable is used with the msvc batch file determined via autodetection. Refer to the documentation for additional requirements and validation details.
  • Added MSVC_TOOLSET_VERSION construction variable which allows building with a specific toolset version. This variable is used with the msvc batch file determined via autodetection. This variable does not affect the autodetection and selection of msvc instances. The toolset version is applied after an msvc instance is selected. This could be the default version of msvc. Refer to the documentation for additional requirements and validation details. Addresses issue #3265, issue #3664, and pull request #4149.
  • Added MSVC_SPECTRE_LIBS construction variable which allows building with spectre-mitigated Visual C++ libraries. This variable is used with the msvc batch file determined via autodetection. Refer to the documentation for additional requirements and validation details.
  • Added MSVC_SCRIPT_ARGS construction variable which specifies command line arguments that are passed to the msvc batch file determined via autodetection. Refer to the documentation for additional requirements and validation details. Addresses enhancement issue #4106.
  • Ninja: Added new alias "shutdown-ninja-scons-daemon" to allow ninja to shutdown the daemon. Also added cleanup to test framework to kill ninja scons daemons and clean ip daemon logs. NOTE: Test for this requires python psutil module. It will be skipped if not present.
  • Ninja: Added command line variable NINJA_CMD_ARGS that allows to pass through ninja command line args. This can also be set in your Environment().
  • Added a global policy setting and an environment construction variable for specifying the action to be taken when an msvc request cannot be satisfied. The available options are "error", "exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be set and retrieved via the functions msvc_set_notfound_policy and msvc_get_notfound_policy, respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment construction variable is MSVC_NOTFOUND_POLICY. When defined, the environment construction variable overrides the global policy setting for a given environment. When the active policy is "error" or "exception", an MSVCVersionNotFound exception is raised. When the active policy is "warning" or "warn", a VisualCMissingWarning warning is issued and the constructed environment is likely incomplete. When the active policy is "ignore" or "suppress", no action is taken and the constructed environment is likely incomplete. As implemented, the default global policy is "warning". The ability to set the global policy via an SCons command-line option may be added in a future enhancement.
  • Added a global policy setting and an environment construction variable for specifying the action to be taken when msvc script errors are detected. The available options are "error", "exception", "warning", "warn", "ignore", and "suppress". The global policy variable may be set and retrieved via the functions msvc_set_scripterror_policy and msvc_get_scripterror_policy, respectively. These two methods may be imported from SCons.Tool.MSCommon. The environment construction variable is MSVC_SCRIPTERROR_POLICY. When defined, the environment construction variable overrides the global policy setting for a given environment. When the active policy is "error" or "exception", an MSVCScriptExecutionError exception is raised when msvc batch file errors are detected. When the active policy is "warning" or "warn", an MSVCScriptExecutionWarning warning is issued when msvc batch file errors are detected. When the active policy is "ignore" or "suppress", msvc batch error messages are suppressed. As implemented, the default global policy is "ignore". The ability to set the global policy via an SCons command-line option may be added in a future enhancement.
  • Added experimental function msvc_query_version_toolset to SCons.Tool.MSCommon. Given a version specification, this function will return an msvc version and an msvc toolset version. The msvc toolset version may be None. The msvc version and msvc toolset version can be used in the environment construction variables MSVC_VERSION and MSVC_TOOLSET_VERSION, respectively. The version specification may be an msvc version or an msvc toolset version. This is a proxy for using an msvc toolset version to select an msvc instance. This function may be removed when an msvc toolset version is used during msvc instance selection.
  • Fortran: a new construction variable FORTRANCOMMONFLAGS is added which is applied to all Fortran dialects, to enable global (all-dialect) settings.
  • lex: two new construction variables are introduced (LEX_HEADER_FILE and LEX_TABLES_FILE) as the preferred way of specifying extra files that the tool can generate.
  • yacc: two new construction variables are introduced (YACC_HEADER_FILE and YACC_GRAPH_FILE) as the preferred way of specifying extra files that the tool can generate (applies only when using GNU flex and GNU bison).

CHANGED/ENHANCED EXISTING FUNCTIONALITY

  • On Windows, %AllUsersProfile%\scons\site_scons is now the default "system" location for a site_scons directory. %AllUsersProfile%\Application Data\scons\site_scons will continue to work. There does not seem to be any existing convention to use an "Application Data" subdirectory here.
  • Action._subproc() can now be used as a python context manager to ensure that the POpen object is properly closed.
  • SCons help (-H) no longer prints the "ignored for compatibility" options, which are still listed in the manpage.
  • Help is now sensitive to the size of the terminal window: the width of the help text will scale to wider (or narrower) terminals than 80 characters.
  • Ninja: Changed generated build.ninja file to run SCons only build Actions via a SCons Deamon. Added logic for starting and connecting to SCons daemon (currently only used for ninja)
  • The change to "content" and "content-timestamp" Decider names is reflected in the User Guide as well, since the hash function may be other than md5 (tidying up from earlier change)
  • If the (experimental) SCONS_CACHE_MSVC_CONFIG feature is used, it will now attempt a sanity check for the cached compiler information, and regenerate it if needed. Previously, this cache would fail if a compiler upgrade caused a change to internal paths (e.g. upgrading from 17.1 to 17.2 causes a necessary path component in some of the cached vars to need to 14.32.31326 instead of 14.31.31103), and the cache file needed to be manually removed. The default cachefile name is now "scons_msvc_cache.json" rather than ".scons_msvc_cache" so there should be no transition problem if using the default; if using a custom cache file name, the cache should still be manually removed if there are problems to transition to the new style.
  • Ninja: Update ninja file generation to only create response files for build commands which exceed MAXLINELENGTH
  • Update the debug output written to stdout for MSVC initialization which is enabled by setting SCONS_MSCOMMON_DEBUG=- to use the logging module. Also changed the debug output format written to stdout to include more information about the source for each message of MSVC initialization debugging output. A single space was added before the message for all debugging output records written to stdout and to files.
  • Ninja: Made ninja tool force the ninja file as the only target. Also improved the default targets setup and made sure there is always a default target for the ninja file, which excludes targets that start and stop the daemon.
  • Ninja: Update ninja tool so targets passed to SCons are propgated to ninja when scons automatically executes ninja.
  • Add JavaScanner to include JAVACLASSPATH as a dependency when using the Java tool.
  • The build argument (i.e., x86) is no longer passed to the MSVC 6.0 to 7.1 batch files. This may improve the effectiveness of the internal msvc cache when using MSVC detection and when bypassing MSVC detection as the MSVC 6.0 to 7.1 batch files do not expect any arguments.
  • Propagate the OS and windir environment variables from the system environment to the msvc environment. The OS and windir environment variables are used in the MSVC 6.0 batch file and the SDK 6.0-7.1 SetEnv.cmd batch files. Inclusion of the OS and windir environment variables eliminates some partial paths and warnings generated by the MSVC 6.0 and SDK 6.0-7.1 batch files when the variables are not defined.
    • Note: Attempting to run the SDK 6.0-7.1 batch files directly via MSVC_USE_SCRIPT can lead to build failures and/or incomplete build environments. The SDK 6.0-7.1 batch files require delayed expansion to be enabled which is currently not supported and is typically not enabled by default on the host system. The batch files may also require environment variables that are not included by default in the msvc environment.
  • An exception is raised when MSVC_UWP_APP is enabled for Visual Studio 2013 and earlier. Previous behavior was to silently ignore MSVC_UWP_APP when enabled for Visual Studio 2013 and earlier. Refer to the documentation for additional requirements and validation details. MSVC_UWP_APP was extended to accept True, False, and None in addition to '1' and '0'.
  • Ninja: added option "--skip-ninja-regen" to enable skipping regeneration of the ninja file if scons can determine the ninja file doesnot need to be regenerated, which will also skip restarting the scons daemon. Note this option is could result in incorrect rebuilds if scons Glob or scons generated files are used in ninja build target's command lines.
  • Tool loading used to have a special case for Jython, it no longer does. This effectively means SCons doesn't work with Jython, which has in reality been the case ever since SCons dropped Python 2 support - there is still no timeline for Jython switching to Python 3 compatibility.

FIXES

  • Fix a number of Python ResourceWarnings which are issued when running SCons and/or it's tests with python 3.9 (or higher)
  • Ninja: Fix issue where Configure files weren't being properly processed when build run via ninja.
  • Fixed crash in C scanner's dictify_CPPDEFINES() function which happens if AppendUnique is called on CPPPATH. (Issue #4108).
  • Added default values for source and target arguments to _defines() function. This is used to expand CPPDEFINES (and others). Previous change added those arguments with no defaults, so old usage where _defines() was called without source and target arguments would yield an exception. This issue was found via qt4 and qt5 tools in scons-contrib https://github.com/SCons/scons-contrib/issues/45
  • Fix issue where if you only had mingw installed on a Windows system and no MSVC compiler, and did not explicitly request the mingw tool, mingw tool initialization would fail and set the default compiler to MSVC which wasn't installed, yielding broken build. Updated mingw tool so that the generate and exists methods use the same mingw search paths (issue #4134).
  • Ninja: Added NINJA_GENERATED_SOURCE_ALIAS_NAME which allows user to specify an Alias() which the ninja tool can use to determine which files are generated sources. If this is not set by the user then the ninja tool will still dynamically determine which files are generated sources based on NINJA_GENERATED_SOURCE_SUFFIXES, and create a phony target _ninja_generated_sources. Generated sources will be built first by ninja. This is needed because ninja cannot determine which generated sources are required by other build targets. Code contributed by MongoDB.
  • Added special case for ninja scons daemon to work in win32 python3.6 environments. This particular environment does a bad job managing popen standard file handles, so some special workarounds are needed.
  • Added user configurable setting of ninja depfile format via NINJA_DEPFILE_PARSE_FORMAT. Now setting NINJA_DEPFILE_PARSE_FORMAT to [msvc,gcc,clang] can force the ninja expected format. Compiler tools will also configure the variable automatically.
  • Fix issue where Express versions of the MSVC compiler were not detected due to differences in initial msvc detection and msvc batch file determination when configuring the build environment. This could lead to build failures when only an MSVC Express instance is installed and the MSVC version is not explicitly specified (issue #2668 and issue #2697).
  • Restore the ability of the content-timestamp decider to see that a a source which is a symlink has changed if the file-system target of that link has been modified (issue #3880)
  • Fix typo in ninja scons daemon startup which causes ConnectionRefusedError to not retry to connect to the server during start up.
  • Fix incorrect Java classpath generation when a NodeList is used as part of any JAVA*PATH variables.
  • The system environment variable names imported for MSVC 7.0 and 6.0 were updated to be consistent with the variables names defined by their respective installers. This fixes an error caused when bypassing MSVC detection by specifying the MSVC 7.0 batch file directly.
  • lex: Fixed an issue with the lex tool where file arguments specified to either "--header-file=" or "--tables-file=" which included a space in the path to the file would be processed incorrectly
  • Modify the MSCommon logger configuration to be independent of the root logger. This fixes an issue when multiple loggers are created and the MSCommon logger added computed fields to the root logger that are not present in other logging instances.
  • Modify the MSVC_USE_SCRIPT_ARGS test fixture to disable the msvc cache. This fixes an issue where the MSVC_USE_SCRIPT_ARGS test for success relied on a debug log message that was not produced when the msvc cache file exists and the test keys are already in the cache as the msvc script invocation was bypassed.
  • Suppress issuing a warning when there are no installed Visual Studio instances for the default tools configuration (issue #2813). When msvc is the default compiler because there are no compilers installed, a build may fail due to the cl.exe command not being recognized. At present, there is no easy way to detect during msvc initialization if the default environment will be used later to build a program and/or library. There is no error/warning issued for the default tools as there are legitimate SCons uses that do not require a c compiler.

IMPROVEMENTS

  • Verify that a user specified msvc script (via MSVC_USE_SCRIPT) exists and raise an exception immediately when the user specified msvc script does not exist.
  • Add cache-debug messages for push failures.
  • Ninja: Added ninja mingw support and improved ninja CommandGeneratorAction support.
  • Command-line help is now sensitive to the size of the terminal window: the width of the help text will scale for terminals other than 80 chars wide.
  • Refactor the msvc code so that the same data structures are used during initial msvc detection and msvc batch file determination when configuring the build environment. Simplify the msvc code by eliminating special case handling primarily due to the differences between the full versions and express versions of visual studio.
  • Small refactor of scons daemons using a shared StateInfo class for communication between the scons interactive thread and the http server thread. Added error handling for scons interactive failing to startup.
  • Ninja: Updated ninja scons daemon scripts to output errors to stderr as well as the daemon log.
  • Ninja: Ensure that all targets set as default via Default() in SConstruct/SConscripts are default targets in the generated ninja.build file.

PACKAGING

  • Added project_url for mailing lists and Discord
  • Updated setup.cfg to remove Python 3.5 and add Python 3.10

Thanks to the following contributors listed below for their contributions to this release.

git shortlog --no-merges -ns 4.3.0..HEAD
174  Joseph Brill
126  Mats Wichmann
  93  William Deegan
  64  Daniel Moody
  4  SergBobrovsky
  2  dependabot[bot]
  1  djh
  1  Ivan Kravets
  1  Vishwajith-K
  1  Zhichang Yu