Software releases 4: your own pypi

Tags: python, softwarereleasesseries

There will always be releases that you do not want to release to the public python package index. Internal company software; just-for-your-own-use software; trying-stuff-out software. What to do with that? Solution: a personal pypi is relatively easy to set up with sdistmaker, provided you use subversion.

What sdistmaker does:

  • It searches through your svn repository, looking for tags. It has support for filtering out sections of your repository structure, so it won’t look where you don’t want it to.
  • For every tag found, it creates a source distribution (“python setup.py sdist”).
  • The source distributions are stored in per-project sub-directories, just like on http://pypi.python.org/simple/ .

Installation and basic usage

A simple easy_install sdistmaker is enough. This gives you two scripts:

  • make_sdist, mainly for test purposes. Pass it an svn tag URL and a destination dir and it will make a release.
  • sdists_from_tags is the main script. It searches an svn structure for suitable tags and makes releases of them.

For starters, just run sdists_from_tags. It will create a var/private directory and fill it with (as an example!) all zest.releaser releases.

Both scripts have a --help option that show all available options and a usage instruction.

Configuration

Configuration is by means of a python file. Easiest way to get started is by printing sdistmaker’s own base defaults.py by doing:

$> sdists_from_tags --print-example-defaults

Save the output as a python file (suggestion: defaults.py). You can then adapt it to your liking and use it with sdists_from_tags --defaults-file=defaults.py. The defaults file is documented in-line, so it should be easy to adapt.

Of course, in the end you want to run sdists_from_tags automatically (Unix cron job, svn post-commit hook, etc.)

Usage in a buildout

You can use sdistmaker in a buildout like this:

[buildout]
parts = sdists

[sdists]
recipe = zc.recipe.egg
eggs = sdistmaker
scripts = sdists_from_tags
# arguments =
#     defaults_file='${buildout:directory}/defaults.py',

The defaults.py is created in the same way as above.

Using sdistmaker in combination with the real pypi

A structure like generated with sdistmaker is a perfect index for easy_install and buildout if you let Apache host it. Only problem: you can only have one index (note: pip apparently supports multiple indexes). You can solve this problem by having Apache redirect you to pypi when something is not found.

Here’s an example Apache config snippet:

# Allow indexing
Options +Indexes
IndexOptions FancyIndexing VersionSort

# Start of rewriterules to use our own var/private/* packages
# when available and to redirect to pypi if not.
RewriteEngine On
# Use our robots.txt:
RewriteRule ^/robots.txt - [L]
# Use our apache's icons:
RewriteRule ^/icons/.* - [L]
# We want OUR index.  Specified in a weird way as apache
# searches in a weird way for index.htm index.html index.php etc.
RewriteRule ^/index\..* - [L]

# Use our var/private/PROJECTNAME if available,
# redirect to pypi otherwise:
RewriteCond /path/on/server/var/private/$1 !-f
RewriteCond /path/on/server/var/private/$1 !-d
RewriteRule ^/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [L]

# Use our var/private/PROJECTNAME/project-0.1.tar.gz if available,
# redirect to pypi otherwise:
RewriteCond /path/on/server/var/private/$1 !-d
RewriteRule ^/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [L]

(Correction 2010-03-05 after tip from Marc Rijken: the last two rewriterules had [P,L] instead of [L]. I copied an older version instead of the actually-works config).

Using the index

You can use such a custom apache-served index in two ways. Easy_install has a -i option for passing along an index:

$> easy_install -i http://packages.my.server/ zest.releaser

In buildout, you can set it like this:

[buildout]
index = http://packages.my.server/
parts =
    ...

Software releases series

This post is number four of my releasing python software series.

  • First set up a proper svn structure with trunk/tags/branches.
  • Package your python software with setuptools (or its replacement, distribute) so that you can make proper releases.
  • Take the menial and manual and boring steps out of tagging your releases with zest.releaser.
  • This post showed you how to actually access your proper releases in a handy way (when they’re not released on pypi).

The next post I’ll describe buildout: controlling which versions of those released packages you use. Repeatable and isolated environment. Handy extra automation tasks.