Building Pyrex extensions for Python using SCons

This recipe is based on the example for building boost.python extensions from page PythonExtensions. Pyrex builder was peeked from http://pyinsci.blogspot.com/2007/01/building-pyrex-with-scons.html

I've combined info from both pages and now I have SConstruct to build my Pyrex extensions under OS Windows with MSVC 2003 compiler. This works fine for me on Python 2.5/2.4 with SCons v.1.1.

I've tried to make my Sconstruct.pyrex more or less smart and decided to reuse typical setup.py for creating scons targets. To achieve this one need to slightly change setup.py:

1) SConstruct.pyrex tries to get extension list to build by invoking function get_extensions() from setup.py. This function should return list of extensions to build.

2) Because setup.py is imported by SConstruct.pyrex one need to guard main setup() function from execution, to achieve this one need to make it conditional, e.g.:

if __name__ == '__main__':
        setup(...)

3) To reduce probability of user errors it's better to use get_extensions() in your setup.py, i.e.

if __name__ == '__main__':
     setup(
       name = 'Demos',
       ext_modules=get_extensions(),
       cmdclass = {'build_ext': build_ext}
     )

Changing setup.py in this way one ends up with fully workable python setup.py build_ext command and in the same time one can use SConstruct.pyrex for building extensions.

I've attached my SConstruct.pyrex and modified primes.pyx example from standard Pyrex sources tarball (the only thing I've changed is setup.py). You can build it either with

python setup.py build_ext -i

scons -f SConstruct.pyrex

I've tried to write SConstruct.pyrex in cross-platform way, but I suspect it should be adjusted to use with mingw or gcc (@Linux).

SConstruct.pyrex:

   1 import distutils.sysconfig
   2 import os
   3 import re
   4 import sys
   5 
   6 
   7 def TOOL_DISTUTILS(env):
   8     """Add stuff needed to build Python/Pyrex extensions [with MSVC]."""
   9     (cc, opt, so_ext) = distutils.sysconfig.get_config_vars('CC', 'OPT', 'SO')
  10     if cc:
  11         env['CC'] = cc
  12     env.AppendUnique(CPPPATH=[distutils.sysconfig.get_python_inc()])
  13     if os.name == 'nt':     # OS Windows
  14         if sys.version_info[:2] in ((2,4), (2,5)):
  15             # this flags suitable for Python 2.4/2.5 + MSVC 2003 compiler
  16             cppflags = Split("/Ox /MD /W3 /GX /DNDEBUG")
  17         else:
  18             raise Exception("Unsupported Python version.")
  19     env.AppendUnique(CPPFLAGS=cppflags)
  20     if opt:
  21         env.AppendUnique(CPPFLAGS=opt)
  22     env.AppendUnique(LIBPATH=[distutils.sysconfig.PREFIX+"/libs"])
  23     env['SHLIBPREFIX'] = ""   # gets rid of lib prefix
  24     env['SHLIBSUFFIX'] = so_ext
  25 
  26 
  27 env = Environment(tools=['default', TOOL_DISTUTILS])
  28 
  29 # adding Pyrex builder
  30 if os.name == 'nt':
  31     pyrex_executable = '"%s" "%s"' % (sys.executable,
  32         os.path.join(sys.prefix, 'Scripts', 'pyrexc.py'))
  33 else:
  34     pyrex_executable = 'pyrexc'
  35 pyxbld = Builder(action='%s -o $TARGET $SOURCE' % pyrex_executable)
  36 env.Append(BUILDERS={'Pyrex': pyxbld})
  37 
  38 
  39 # build extension(s)
  40 import setup
  41 for e in setup.get_extensions():
  42     envx = env.Clone()
  43     # adjust compiler flags/options
  44     if e.define_macros:
  45         envx.AppendUnique(CPPDEFINES=dict(e.define_macros))
  46     if e.libraries:
  47         envx.AppendUnique(LIBS=e.libraries)
  48     if e.include_dirs:
  49         envx.AppendUnique(CPPPATH=e.include_dirs)
  50     # looking for common src dir prefix
  51     sources = e.sources[:]
  52     build_dir = None
  53     src_dir = None
  54     for s in sources:
  55         parts = re.split(r'[\\/]', s, 1)
  56         if len(parts) != 2 or parts[0] == '':
  57             break
  58         else:
  59             prefix = parts[0]
  60             if src_dir is None:
  61                 src_dir = prefix
  62             elif prefix != src_dir:
  63                 break
  64     else:
  65         # src_dir found
  66         if src_dir:
  67             build_dir = os.path.join('build',
  68                 'temp.%s-%d.%d' % (sys.platform, sys.version_info[0], sys.version_info[1]))
  69             envx.VariantDir(os.path.join(build_dir, src_dir),
  70                 src_dir,
  71                 duplicate=0)
  72             sources = [os.path.join(build_dir, s) for s in sources]
  73     # check if we build pyrex extension
  74     for ix, s in enumerate(sources):
  75         if s.endswith('.pyx'):
  76             cfile = s[:-4]+'.c'
  77             sources[ix] = cfile
  78             envx.Pyrex(cfile, s)
  79     # and schedule it for building
  80     envx.SharedLibrary(e.name.replace('.', os.sep), sources)

Example of setup.py:

   1 from distutils.core import setup
   2 #from distutils.extension import Extension
   3 from Pyrex.Distutils.extension import Extension
   4 from Pyrex.Distutils import build_ext
   5 
   6 
   7 def get_extensions():
   8     return [
   9         Extension("primes",       ["primes.pyx"]),
  10 #        Extension("spam",         ["spam.pyx"]),
  11 #        Extension("numeric_demo", ["numeric_demo.pyx"]),
  12         ]
  13 
  14 
  15 if __name__ == '__main__':
  16     setup(
  17       name = 'Demos',
  18       ext_modules=get_extensions(),
  19       cmdclass = {'build_ext': build_ext}
  20     )

primes.pyx source file:

   1 def primes(int kmax):
   2     cdef int n, k, i
   3     cdef int p[1000]
   4     result = []
   5     if kmax > 1000:
   6         kmax = 1000
   7     k = 0
   8     n = 2
   9     while k < kmax:
  10         i = 0
  11         while i < k and n % p[i] <> 0:
  12             i = i + 1
  13         if i == k:
  14             p[k] = n
  15             k = k + 1
  16             result.append(n)
  17         n = n + 1
  18     return result

PyrexPythonExtensions

Building Pyrex extensions for Python using SCons