Pypi eggproxy plus private packages¶

Tags: python, buildout, thehealthagency

To have some local control over pypi’s availability, we’re using an internal proxy in front of pypi using collective.eggproxy. (I’ve written about that before in collective.eggproxy improvements).

Our (= The Health Agency) internal packages that aren’t on pypi: how to release those? The quickest way was to use an html page with svn url links, one per release. You can put the url to that page in your buildout’s find-links setting. Originally we maintained that list by hand. I quickly wrote a script to handle it once the amount of packages got out of hand: tha.taglist.

We hit some problems with that approach due to subversion 1.6. Not everywhere was there an updated setuptools/distribute that could handle those newer subversion checkouts. A real sdist tarball would be way better. The solution:

• I already had a script that finds all the tags in an svn structure (including filtering, blacklists, etc.). The actual finding is extracted in a small library: tha.tagfinder. I used the same tag to create the html page with svn urls.

• I added a script that goes through all those svn tag urls and that makes a checkout + sdist tarball out of them. And copies it to a directory structure, just like the simple pypi index: libraryname/library-versionnumber.tar.gz.

• I changed the eggproxy’s apache config to only pass certain requests through to the proxy, using the information on http://wiki.python.org/moin/PyPiImplementations .

The relevant portions from the apache config:

DocumentRoot /server/dir/var/private

# Allow indexing, but make it fancy and sort
# on version numbers
Options +Indexes
IndexOptions FancyIndexing VersionSort

RewriteEngine On
RewriteRule ^/icons/.* - [L]
# Catch all index.html, index.cgi: we handle it
RewriteRule ^/index\..* - [L]

# Use our var/private/project if available, proxy to
# the eggproxy at port 8080 otherwise.
# "!-f" means "if there's no file, "!-d" is about
# directories.
RewriteCond /server/dir/var/private/$1 !-f
RewriteCond /server/dir/var/private/$1 !-d
RewriteRule ^/([^/]+)/?$ http://localhost:8080/$1/ [P,L]

# Use our var/private/project/project-0.1.tar.gz if available,
# otherwise proxy to the eggproxy at 8080
RewriteCond /server/dir/var/private/$1 !-d
RewriteRule ^/([^/]+)/([^/]+)$ http://localhost:8080/$1/$2 [P,L]

Pretty luxurious. Just mention this one proxy (index = my.proxy.org) in buildout and you’re set for both your own packages and for pypi packages. Just make sure the package names don’t overlap :-)