Software releases 4: your own pypi

Tags: python, softwarereleasesseries

There will always be releases that you do not want to release to the public python package index. Internal company software; just-for-your-own-use software; trying-stuff-out software. What to do with that? Solution: a personal pypi is relatively easy to set up with sdistmaker, provided you use subversion.

What sdistmaker does:

  • It searches through your svn repository, looking for tags. It has support for filtering out sections of your repository structure, so it won’t look where you don’t want it to.
  • For every tag found, it creates a source distribution (“python setup.py sdist”).
  • The source distributions are stored in per-project sub-directories, just like on http://pypi.python.org/simple/ .

Installation and basic usage

A simple easy_install sdistmaker is enough. This gives you two scripts:

  • make_sdist, mainly for test purposes. Pass it an svn tag URL and a destination dir and it will make a release.
  • sdists_from_tags is the main script. It searches an svn structure for suitable tags and makes releases of them.

For starters, just run sdists_from_tags. It will create a var/private directory and fill it with (as an example!) all zest.releaser releases.

Both scripts have a --help option that show all available options and a usage instruction.

Configuration

Configuration is by means of a python file. Easiest way to get started is by printing sdistmaker’s own base defaults.py by doing:

$> sdists_from_tags --print-example-defaults

Save the output as a python file (suggestion: defaults.py). You can then adapt it to your liking and use it with sdists_from_tags --defaults-file=defaults.py. The defaults file is documented in-line, so it should be easy to adapt.

Of course, in the end you want to run sdists_from_tags automatically (Unix cron job, svn post-commit hook, etc.)

Usage in a buildout

You can use sdistmaker in a buildout like this:

[buildout]
parts = sdists

[sdists]
recipe = zc.recipe.egg
eggs = sdistmaker
scripts = sdists_from_tags
# arguments =
#     defaults_file='${buildout:directory}/defaults.py',

The defaults.py is created in the same way as above.

Using sdistmaker in combination with the real pypi

A structure like generated with sdistmaker is a perfect index for easy_install and buildout if you let Apache host it. Only problem: you can only have one index (note: pip apparently supports multiple indexes). You can solve this problem by having Apache redirect you to pypi when something is not found.

Here’s an example Apache config snippet:

# Allow indexing
Options +Indexes
IndexOptions FancyIndexing VersionSort

# Start of rewriterules to use our own var/private/* packages
# when available and to redirect to pypi if not.
RewriteEngine On
# Use our robots.txt:
RewriteRule ^/robots.txt - [L]
# Use our apache's icons:
RewriteRule ^/icons/.* - [L]
# We want OUR index.  Specified in a weird way as apache
# searches in a weird way for index.htm index.html index.php etc.
RewriteRule ^/index\..* - [L]

# Use our var/private/PROJECTNAME if available,
# redirect to pypi otherwise:
RewriteCond /path/on/server/var/private/$1 !-f
RewriteCond /path/on/server/var/private/$1 !-d
RewriteRule ^/([^/]+)/?$ http://pypi.python.org/pypi/$1/ [L]

# Use our var/private/PROJECTNAME/project-0.1.tar.gz if available,
# redirect to pypi otherwise:
RewriteCond /path/on/server/var/private/$1 !-d
RewriteRule ^/([^/]+)/([^/]+)$ http://pypi.python.org/pypi/$1/$2 [L]

(Correction 2010-03-05 after tip from Marc Rijken: the last two rewriterules had [P,L] instead of [L]. I copied an older version instead of the actually-works config).

Using the index

You can use such a custom apache-served index in two ways. Easy_install has a -i option for passing along an index:

$> easy_install -i http://packages.my.server/ zest.releaser

In buildout, you can set it like this:

[buildout]
index = http://packages.my.server/
parts =
    ...

Software releases series

This post is number four of my releasing python software series.

  • First set up a proper svn structure with trunk/tags/branches.
  • Package your python software with setuptools (or its replacement, distribute) so that you can make proper releases.
  • Take the menial and manual and boring steps out of tagging your releases with zest.releaser.
  • This post showed you how to actually access your proper releases in a handy way (when they’re not released on pypi).

The next post I’ll describe buildout: controlling which versions of those released packages you use. Repeatable and isolated environment. Handy extra automation tasks.

blog comments powered by Disqus
 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):