Please note:The SCons wiki is in read-only mode due to ongoing spam/DoS issues. Also, new account creation is currently disabled. We are looking into alternative wiki hosts.
Differences between revisions 13 and 14
Revision 13 as of 2006-05-09 10:36:38
Size: 4640
Comment: Python extension module builder
Revision 14 as of 2008-03-12 02:47:09
Size: 4662
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
This section will outline some ways in which I see a system for building Python packages being used. I'll use [http://www.scipy.org/ NumPy] for some of the examples, since this is a good example of a project with a large body of C and Fortran code. The !NumPy developers had to write [http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils lots of extra code] to get their package to build across many platforms with many configurations. This section will outline some ways in which I see a system for building Python packages being used. I'll use [[http://www.scipy.org/|NumPy]] for some of the examples, since this is a good example of a project with a large body of C and Fortran code. The !NumPy developers had to write [[http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/distutils|lots of extra code]] to get their package to build across many platforms with many configurations.
Line 10: Line 10:
 1. The user goes to the [http://www.scipy.org/Download NumPy download page].  1. The user goes to the [[http://www.scipy.org/Download|NumPy download page]].
Line 26: Line 26:
 1. The user has purchased the [http://www.intel.com/cd/software/products/asmo-na/eng/perflib/mkl/266858.htm Intel Matrix Kernel Library] for Windows or downloaded the free version for Linux.  1. The user has purchased the [[http://www.intel.com/cd/software/products/asmo-na/eng/perflib/mkl/266858.htm|Intel Matrix Kernel Library]] for Windows or downloaded the free version for Linux.
Line 36: Line 36:
 * Automatically find/download/install/upgrade dependencies at build time using the [http://peak.telecommunity.com/DevCenter/EasyInstall EasyInstall] tool
 * Create [http://peak.telecommunity.com/DevCenter/PythonEggs Python Eggs]
 * Automatically find/download/install/upgrade dependencies at build time using the [[http://peak.telecommunity.com/DevCenter/EasyInstall|EasyInstall]] tool
 * Create [[http://peak.telecommunity.com/DevCenter/PythonEggs|Python Eggs]]
Line 39: Line 39:
 * [http://mail.python.org/pipermail/distutils-sig/2006-April/006240.html setuptools now in Python 2.5 trunk]  * [[http://mail.python.org/pipermail/distutils-sig/2006-April/006240.html|setuptools now in Python 2.5 trunk]]
Line 42: Line 42:
 * [http://peak.telecommunity.com/DevCenter/setuptools Building and Distributing Packages with setuptools]
 * [http://ianbicking.org/docs/setuptools-presentation/ Python Packaging with Setuptools]
 * [[http://peak.telecommunity.com/DevCenter/setuptools|Building and Distributing Packages with setuptools]]
 * [[http://ianbicking.org/docs/setuptools-presentation/|Python Packaging with Setuptools]]
Line 47: Line 47:
Which Python versions will be supported? SCons supports many old versions, but setuptools only supports 2.3 and 2.4. I think that support for 2.3 and 2.4 might be enough. Other projects such as !NumPy [http://thread.gmane.org/gmane.comp.python.numeric.general/5042/focus=5042 only support 2.3 and 2.4]. Which Python versions will be supported? SCons supports many old versions, but setuptools only supports 2.3 and 2.4. I think that support for 2.3 and 2.4 might be enough. Other projects such as !NumPy [[http://thread.gmane.org/gmane.comp.python.numeric.general/5042/focus=5042|only support 2.3 and 2.4]].
Line 58: Line 58:
  [http://www.gnu.org/prep/standards/html_node/Managing-Releases.html#Managing-Releases GNU Coding Standard].   [[http://www.gnu.org/prep/standards/html_node/Managing-Releases.html#Managing-Releases|GNU Coding Standard]].

Introduction

This page is currently under construction.

Stories

This section will outline some ways in which I see a system for building Python packages being used. I'll use NumPy for some of the examples, since this is a good example of a project with a large body of C and Fortran code. The NumPy developers had to write lots of extra code to get their package to build across many platforms with many configurations.

New User

  1. The user goes to the NumPy download page.

  2. The user chooses the binary package (.rpm, .dmg, .egg, .exe) of the latest release for his platform and CPU type.
  3. The user installs the binary package.
  4. import numpy works.

Bug Reporter

  1. The user finds a bug in NumPy and reports it to the developers.

  2. The bug is fixed in SVN.
  3. The user is asked to verify the bugfix.
  4. The user checks out the latest revision from SVN.
  5. python setup.py installl

  6. import numpy works and the user verifies the bugfix.

In this case, the user isn't interested in building the package with optimized libraries or a specific compiler. He simply wants to build it using whatever tools are available on his system. For some packages, he might have to install some third party libraries in default locations prior to building.

Experienced User

  1. The user has purchased the Intel Matrix Kernel Library for Windows or downloaded the free version for Linux.

  2. The user checks out the latest revision of NumPy from SVN.

  3. python setup.py build --with-mkl install

  4. The user uses his optimized version of NumPy.

In this case, recompiling NumPy with MKL was anticipated by the developers and they were able to add useful default library paths and other details to the setup script.

Existing Tools

setuptools

Links

Things to think about

Which Python versions will be supported? SCons supports many old versions, but setuptools only supports 2.3 and 2.4. I think that support for 2.3 and 2.4 might be enough. Other projects such as NumPy only support 2.3 and 2.4.

Comments

Feel free to add comments in this section.

  • While distutils is a good example to look for inspiration, I think this task should not exclusively focus on packaging Python modules, but be useful for packaging software in general. Thus it is important to capture requirements not only from python conventions but other packaging conventions such as the relevant sections from the

    GNU Coding Standard. On a design-related note, I believe it would be best to work on a layered approach, where the lower layer provides 'builders' that hook up common packaging tools (rpm, deb, wininst, tar), and a layer that is largely orthogonal to SCons, and thus can be provided on top of it, which encapsulates the actual workflow. Users will thus be able to wire the low-level builders up manually, without necessarily following any standard (install prefixes, etc.), while typical users will use the high level layer, which works out-of-the-box on all platforms, but doesn't provide flexibility advanced users may desire.

    --StefanSeefeld

Random Code

   1 import distutils.sysconfig
   2 self['PYEXTPREFIX'] = ''
   3 self['PYEXTSUFFIX'] = distutils.sysconfig.get_config_vars('SO')
   4 def PythonExtensionBuilder(*args, **kw):
   5     kw.setdefault('SHLIBPREFIX', '$PYEXTPREFIX')
   6     kw.setdefault('SHLIBSUFFIX', '$PYEXTSUFFIX')
   7     # ...[0] returns only the dynamic library
   8     return self['BUILDERS']['SharedLibrary'](*args, **kw)[0]
   9 self['BUILDERS']['PythonExtension'] = PythonExtensionBuilder 

SummerOfCodeIdeas/DistutilsReplacement (last edited 2008-03-12 02:47:09 by localhost)