collective.eggproxy improvementsΒΆ

Tags: python, plone, thehealthagency

(update 2009-03-17: this entry now has working links...)

At the health agency we've had our share of pypi outages. Actually, those pypi outages aren't the problem. I've got three alternative index servers (like in my buildout default config, ready for a quick comment-out. They'll take over just fine.

The harder problem is that some packages, like dateutil, aren't hosted on pypi. There's just a link on pypi to the place where you can find them. And those sites don't have mirrors. So using mirroring (like the pypi mirroring project) doesn't solve our core problem. Update: Andreas Jung twittered me that pypi-mirror does support externally hosted packages (something I didn't see in the docs).

The solution: an egg proxy. You request all eggs from your own (company-wide) proxy and the proxy will locally store all eggs that you requested. We use ingeniweb's collective.eggproxy for that.

We had some additional wishes, so I made a couple of changes that are now released in 0.3:

  • Better buildout integration. It can now use a local config file inside the buildout. See the pypi page for a handy example buildout config.
  • An always_refresh option that always queries pypi for potential newer versions (but with a timeout so that you're not hosed when pypi is down). You're never out of date with this option. Very handy, as this was a severe pain point for me in my previous proxy/mirror experience.

So far we're using the proxy in this setting for a week and it works fine. Give it a try!

(Old imported comments)
"Proxies useful for more closed sites too" by on 2009-03-23 20:29:58
Thanks for the pointer, this should solve a problem our government customer wants solved: get code from "trusted" servers rather than those evil rogue h4x0r servers like Pypi. Yeah, yeah, it only proxies those evil repos, shhhhh. :-)

But it also allows our QA and Production servers to grab content from internal servers, rather than getting blocked by our current firewall policy that prevents production boxes from contacting unknown external sites. logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):