Very long command lines on Windows

Short version

We found some problems with MingW on windows during the link phase. The command line length could exceed 10000 characters, and this was causing problems.

Just go to the bottom of the page for the code we used to fix it.

Long version (for those interested in the reasons)

When we changed the build process for [http://aztec.sourceforge.net/ Aztec] to use SCons, we encountered a few problems along the way. Our project contained a number of SharedLibraries and a number of Program's as well, and wanted to build them on Windows, using both Microsoft Visual C and MingW.

MingW had some problems during the link phase of the build process. It turned out our link lines were in excess of 10000 characters, and this was causing some major grief with python calling spawn().

After some investigation, it turns out this is not a limitation of Windows, but a limitation of the C Runtime that MSVC uses (which Python was compiled with). This meant that it was at least fixable. Thanks to the flexible design of SCons, it was not only fixable, but it was very easy to do.

All we need to do was to change the env['SPAWN'] to use our own custom spawn function. We should replace PIPED_SPAWN as well, but that never got called in our build process, so we never got around to it. The fix uses the Win32 API directly, so we avoid the command line length problem entirely.

Here is the code that we placed in our Scons files, and it solves the linking problem with MingW quite well. The interesting part is that it does not handle the standard Windows commands, like del or mkdir. Special consideration has to be made for these.

Here is the final section of code to acheive what we wanted.

if env['PLATFORM'] == 'win32':
      import win32file
      import win32event 
      import win32api
      import win32con
      import win32event
      import win32file
      import win32pipe
      import win32process
      import win32security
      import pywintypes
      import msvcrt
      import win32gui 
      import string

      def my_spawn(sh, escape, cmd, args, spawnenv):
          for var in spawnenv : 
              spawnenv[var] = spawnenv[var].encode('ascii', 'replace')

          sAttrs = win32security.SECURITY_ATTRIBUTES()
          StartupInfo = win32process.STARTUPINFO()
          newargs = string.join(map(escape, args[1:]), ' ')
          cmdline = cmd + " " + newargs

          # check for any special operating system commands
          if cmd == 'del' :
              win32file.DeleteFile(string.join(args[1:], ' '))
              exit_code = 0
          else :
              # otherwise execute the command.
              hProcess, hThread, dwPid, dwTid = win32process.CreateProcess(None, cmdline, None, None, 1, 0, spawnenv, None, StartupInfo)
              win32event.WaitForSingleObject(hProcess, win32event.INFINITE)
              exit_code = win32process.GetExitCodeProcess(hProcess)
              win32file.CloseHandle(hProcess);
              win32file.CloseHandle(hThread);
          return exit_code 

      env['SPAWN'] = my_spawn


Acknowledgements Many thanks to Tobias Sargeant for working out most of this, especially the Python side of things. Ack