How I learned to stop worrying and love python packaging - Jannis Leidel (djangocon.eu)

Tags: djangocon, django

Regular python packaging (in our case in combination with buildout) is something we use a lot. So a talk on this at the djangocon.eu is something I look upon with a favourable eye. (I’ll even take the opportunity to point you at my software releases series.)

The past of python packaging

Some terms are ambiguous:

  • A python package is a directory with python files and an __init__.py. This is a module.
  • A release is something like “django 1.3”.
  • A distribution is a source/binary form of a release, though this is often called a package.

It started out with distutils: a standard way for building, distributing and installing python modules. Especially also C extensions. It installs into a library directory.

It wasn’t completely up to the task, so several extensions showed up. Some of those came with a PEP, for instance PEP 241 for package metadata such as “version”, “description”, “author”. And also now there are some PEPs that are currently being implemented. There’s continuous work in this area.

Common pitfalls and gotchas and tips

The basis for packaging is a setup.py file. A problem with this is that such a setup file is used for a couple of dissimilar tasks:

  • Generating documentation.
  • Creating a release (source/binary).
  • Actually installing the software.

Combining these tasks in one file is a bit of a hazard and leads to problems now and then.

Tip regarding documentation: add a so-called “long description” by concatenating the readme and changelog and other documentation files into the description field in your setup.py. This is rendered as restructured text on the pypi page if you upload it.

List all additional non-python files that you need in package_data, otherwise they won’t get installed. Likewise, add a MANIFEST.in that lists all your files.

Everyday software development

Use sane versions. 1.2, 1.3.2, not .000001 or unreleased.unofficialdev. And use release names in any prose instead of version numbers: release “batman” is nicer than release “1.2”.

Regarding regular package management versus python-only installations? Just use your regular package management (“apt-get install xyz”) in case it are binary packages. They have dependencies and header file requirements that are best handled globally by the OS (PIL, lxml, database adapters, etc.)

Install often-updated python-only packages locally, on the other hand. Don’t install them globally. Isolate with, for instance, virtualenv. Tip when you’re using virtualenv: http://www.doughellmann.com/docs/virtualenvwrapper/ .

Combine virtualenv with pip for sane installation and uninstalling. It can also “freeze” and “unfreeze” your requirements and your versions. You can just copy a text file to another machine and re-create your environment there.

Pip also has automatic support for pypi mirrors (the http://d.pypi.python.org/ ones). You can configure pip to globally cache your downloaded packages, handy in case you’re without internet.

You can also make your own simple package index with apache’s DirectoryIndex. There’s also a pypi clone made with django: Chishop.

The future of python packaging

There’s a lot of work happening, revolving around Tarek Ziade: distutils2. It is much cleaner and set up more as a toolbox than as a stand-alone do-everything packaging tool. There is an associated install tool called “pysetup” that is pip-like. Distutils2 has just landed in python 3 trunk as “packaging”. It is (being) backported to python2, too.

Some good new features: better requirement specifications. “On python 2.5, I need package x, 2.6+ doesn’t need it”. “I make package xyz obsolete”. “I only run with python 2.4+”.

Also possible: you can specify that you need some external non-python package, like libxml2 which is needed by python’s lxml. The hope is that external packagers will start to work with that metadata.

Moving from distutils to distutils2: there’s “pysetup3 create” for automatic conversion from setup.py to the new setup.cfg format.

Some closing comments:

  • Keep your setup.py files simple. It is a python file, so you can do anything with it. But restrict yourself.
  • Follow the standards.
  • Use virtualenv/pip.
  • Look out for distutils2.
  • When in doubt, read the hitchhikers’ guide to packaging
0-scale cats and a delipidated station building
blog comments powered by Disqus
 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):