collective.eggproxy improvements¶

(update 2009-03-17: this entry now has working links...)

At the health agency we've had our share of pypi outages. Actually, those pypi outages aren't the problem. I've got three alternative index servers (like http://download.zope.org/ppix) in my buildout default config, ready for a quick comment-out. They'll take over just fine.

The harder problem is that some packages, like dateutil, aren't hosted on pypi. There's just a link on pypi to the place where you can find them. And those sites don't have mirrors. So using mirroring (like the pypi mirroring project) doesn't solve our core problem. Update: Andreas Jung twittered me that pypi-mirror does support externally hosted packages (something I didn't see in the docs).

The solution: an egg proxy. You request all eggs from your own (company-wide) proxy and the proxy will locally store all eggs that you requested. We use ingeniweb's collective.eggproxy for that.

We had some additional wishes, so I made a couple of changes that are now released in 0.3:

• Better buildout integration. It can now use a local config file inside the buildout. See the pypi page for a handy example buildout config.
• An always_refresh option that always queries pypi for potential newer versions (but with a timeout so that you're not hosed when pypi is down). You're never out of date with this option. Very handy, as this was a severe pain point for me in my previous proxy/mirror experience.

So far we're using the proxy in this setting for a week and it works fine. Give it a try!