Regular python packaging (in our case in combination with buildout) is something we use a lot. So a talk on this at the djangocon.eu is something I look upon with a favourable eye. (I’ll even take the opportunity to point you at my software releases series.)
Some terms are ambiguous:
A python package is a directory with python files and an
__init__.py. This is a module.
A release is something like “django 1.3”.
A distribution is a source/binary form of a release, though this is often called a package.
It started out with distutils: a standard way for building, distributing and installing python modules. Especially also C extensions. It installs into a library directory.
It wasn’t completely up to the task, so several extensions showed up. Some of those came with a PEP, for instance PEP 241 for package metadata such as “version”, “description”, “author”. And also now there are some PEPs that are currently being implemented. There’s continuous work in this area.
The basis for packaging is a setup.py file. A problem with this is that
such a setup file is used for a couple of dissimilar tasks:
Generating documentation.
Creating a release (source/binary).
Actually installing the software.
Combining these tasks in one file is a bit of a hazard and leads to problems now and then.
Tip regarding documentation: add a so-called “long description” by
concatenating the readme and changelog and other documentation files into the
description field in your setup.py. This is rendered as restructured text
on the pypi page if you upload it.
List all additional non-python files that you need in package_data,
otherwise they won’t get installed. Likewise, add a MANIFEST.in that lists
all your files.
Use sane versions. 1.2, 1.3.2, not .000001 or
unreleased.unofficialdev. And use release names in any prose instead of
version numbers: release “batman” is nicer than release “1.2”.
Regarding regular package management versus python-only installations? Just use your regular package management (“apt-get install xyz”) in case it are binary packages. They have dependencies and header file requirements that are best handled globally by the OS (PIL, lxml, database adapters, etc.)
Install often-updated python-only packages locally, on the other hand. Don’t install them globally. Isolate with, for instance, virtualenv. Tip when you’re using virtualenv: http://www.doughellmann.com/docs/virtualenvwrapper/ .
Combine virtualenv with pip for sane installation and uninstalling. It can also “freeze” and “unfreeze” your requirements and your versions. You can just copy a text file to another machine and re-create your environment there.
Pip also has automatic support for pypi mirrors (the http://d.pypi.python.org/ ones). You can configure pip to globally cache your downloaded packages, handy in case you’re without internet.
You can also make your own simple package index with apache’s
DirectoryIndex. There’s also a pypi clone made with django: Chishop.
There’s a lot of work happening, revolving around Tarek Ziade: distutils2. It is much cleaner and set up more as a toolbox than as a stand-alone do-everything packaging tool. There is an associated install tool called “pysetup” that is pip-like. Distutils2 has just landed in python 3 trunk as “packaging”. It is (being) backported to python2, too.
Some good new features: better requirement specifications. “On python 2.5, I need package x, 2.6+ doesn’t need it”. “I make package xyz obsolete”. “I only run with python 2.4+”.
Also possible: you can specify that you need some external non-python package, like libxml2 which is needed by python’s lxml. The hope is that external packagers will start to work with that metadata.
Moving from distutils to distutils2: there’s “pysetup3 create” for automatic conversion from setup.py to the new setup.cfg format.
Some closing comments:
Keep your setup.py files simple. It is a python file, so you can do anything with it. But restrict yourself.
Follow the standards.
Use virtualenv/pip.
Look out for distutils2.
When in doubt, read the hitchhikers’ guide to packaging
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):
