Software releases 2: packaging with setuptools¶

Why packaging? Why versions?¶

If you use a couple of python files in several projects, you probably do not want to copy them around time and again. Fix a bug in one and the fix should end up in the others, too. It is handier to just use an installed package in each of your projects instead of coping around by hand.

So if you have an existing monolithic project, see if you can extract some useful functionality out of it so you can re-use it in another project. Once you get the hang of it it can give you loads of productivity! The cheaper and easier it is to re-use existing software, the quicker you can get things done. “Cheaper” does mean an initial price in setting up things and understanding the mechanisms. I’d personally say packaging often pays for itself real quick.

Packaging is a no-brainer if you want others to use your software. Or if you have to distribute it to a customer. Non-packaged software is a maintenance nightmare waiting to happen. It needs manual steps to “install”. You cannot reliably override an older installation if all you do is putting *.py files in a directory. Old files will stay put. People will forget to copy all files, leaving an old one in place (good luck debugging that one when the customer phones you). People will copy files into the wrong directory. Just package up your software properly and you can sleep more soundly at night.

Grouping python files¶

The end result you want is:

import sample

sample.utils.init_something('hurray')
bla_bla = sample.database.BlaBla()
bla_bla.calculate()

So: a couple of python files (utils.py and database.py in this case) grouped together under one name so you can import them from somewhere else (import sample or from sample import utils).

Take the python files you want to group together and put them in a directory with a good name:

• The name, by convention and pep8 should be lowercase.
• You’ll need an __init__.py in the directory to tell python that the directory contents are importable (see explanation). The __init__.py can be empty, but I normally put a # package comment in it as empty files can give problems (like not showing up in “svn diff” sometimes).

Enabling setuptools¶

You’ve probably seen instructions like:

• “Run python setup.py install on the extracted mypackage-1.0.tgz to install it”.
• “Just run easy_install mypackage to install it”.
• “Just run easy_install http://example.org/dist/mypackage-1.0.tgz to install it”.

These commands install the “mypackage” package neatly into your python’s so-called “site-packages” directory. Dig around in your python installation, you’ll find a site-packages directory in there somewhere. Start up the python prompt and you can do an import mypackage.

The setup.py file is what makes all of this work. A setup.py uses setuptools (or its replacement distribute) to actually install the software. This means placing a setup.py in a directory and putting your directory of grouped code in a subdirectory. An example:

The setup.py tells a few things like name, version, dependencies:

from setuptools import setup

setup(name='sample',
      version='1.0',
      description="My sample package",
      long_description="",
      author='Reinout',
      author_email='reinout@vanrees.org',
      license='GPL',
      packages=['sample'],
      zip_safe=False,
      install_requires=['dependency1',
                        'dependency2'],
      )

I’ve left out quite a number of possible fields here. That’s one of the problems with setup.py rightfully mentioned by Jacob Kaplan-Moss: you basically have to cargo-cult your config files from other projects.

What this example config file tells setuptools, however, is:

• The package’s name is “sample”. If you install this package, you’ll get a site-packages/sample-1.0/.... And if you release it on pypi, the python package index, it will become http://pypi.python.org/pypi/sample.
• The version is “1.0”. Read pep 386 if you want to know everything there is to know about version numbers. For now, just use the “0.1, 0.2, ... 1.0, 1.1,...” style of numbering. And a “1.0dev” if you’re still developing that version, for instance.
• Some author and license and description information, mostly for the benefit of automatic overviews like the python package index provides.
• The packages=['sample'] line tells that, inside the current directory, sample/ is an actual package (so: grouping of python files) that belongs in this “sample” package.
• zip_safe=False is just there to make sure python unzips the package while installing it. So instead of one file (a zipped .egg) in your python site-package directory, you will get a directory with the unzipped contents of your package: waaaaay handier. (Corrected 2010-02-23 after tip from Maurits).
• “install_requires” is the jackpot. Automatic dependency installation! If you mention packages here, they’ll get installed automatically when you install this “sample” package. A word of warning is in place here. Unless you’re working in an isolated environment (virtualenv or buildout: I’ll explain it later in this “software releases” series”), everything will be installed in your system python. So if you depend on Plone, you’ll pull in virtually the whole world.

Installing¶

You of course want to install your software. Some ways to do this:

• python setup.py install installs it in your system python.
• When developing, you can do python setup.py develop, which also installs it in your system python, but keeps track of changes. It is essentially a link to the directory you’re developing in. So changes in your code show up right away in the installed version. Only exception: if you’ve changed the setup.py, you have to re-run python setup.py develop.
• Run python setup.py sdist, which creates a “source distribution”. Place the resulting dist/sample-1.0.tar.gz (for example) somewhere on-line and tell others to do easy_install http://what.you.told.them.org/sample-1.0.tar.gz.
• If the software is of good quality and open source licensed, a python setup.py sdist register upload will upload the .tar.gz to the python package index, ready for all to install with a simple easy_install sample.

Where I’ll go from here¶

• Make sure you store your code in a version control system. See Software releases 1: svn structure for hints on that. Use svn tags to tag your releases. I’ll show you how zest.releaser can make that quick and easy.
• Setting up an initial setup.py and readme.txt and so is a chore. Let so-called “project skeletons” do the work for you. I’ll write about that later in this series. For now, look for an example at ZopeSkel.
• You’ve got a company-internal svn? Let tha.sdistmaker create a custom pypi for you. Automatically.
• Isolation for your projects: don’t fill up your site-packages with incompatible versions but isolate yourself with buildout.