Ubuntu PPA madness¶

Tags: python, django, plone

I’m going flipping insane. In ye olde days, when I was programming with the python CMS Plone, my dependencies were limited to python and PIL. Perhaps lxml. LXML was a pain to install sometimes, but there were ways around it.

Working on OSX was no problem. Server setup? Ubuntu. The only thing you really had to watch in those days was your python version. Does this old site still depends on python 2.4 or is it fine to use 2.6? Plone had its own Zope database, so you didn’t even need database bindings.

Now I’m working on Django sites. No problem with Django, btw! But… the sites we build with it are pretty elaborate geographical websites with lots of dependencies. Mapnik, matplotlib, numpy, scipy, gdal, spatialite, postgis. And that’s not the full list. So developing on OSX is no fun anymore, using a virtual machine (virtualbox or vmware) is a necessity. So: Ubuntu.

But… ubuntu 12.04, which we still use on most of the servers, has too-old versions of several of those packages. We need a newer gdal, for instance. And a newer spatialite. The common solution is to use a PPA for that, like ubuntugis-stable.

Now for some random things that can go wrong:

• We absolutely need a newer gdal, so we add the ubuntugis-stable PPA. This has nice new versions for lots of geo-related packages, for instance the “proj” projection library.

• It doesn’t include the “python-pyproj” package, though, which means that the ubuntu-installed python-pyproj package is compiled against a different proj library. Which means your django site segfaults. Digging deep with strace was needed to discover the problem.

• Of course, if you need that latest gdal for your site, you add the PPA to the server. Everything runs fine.

• A month later, the server has to be rebooted. Now the three other sites on that same server fail to start due to the pyproj-segfault. Nobody bothered to check the other sites on the server, of course. (This happened three times on different servers. This is the sort of stuff that makes you cast a doubtful eye on our quite liberal “sudo” policy…)

• Pinning pyproj to 1.9.3 helped, as 1.9.3 worked around the issue by bundling the proj library instead of relying on the OS-packaged one.

• Ubuntugis-stable sounds stable, but they’re of course focused on getting the latest geo packages into ubuntu. So they switched from gdal 1.9 to 1.10 somewhere around june. So /usr/lib/libgdal1.so became /usr/lib/libgdal1h.so and suddenly “apt-get update/upgrade” took down many sites.

  See this travis-ci issue for some background.

• The solution for this PPA problem was another PPA: the postgres one. That includes gdal 1.9 instead of the too-new 1.10.

• Possible problem: the postgres PPA also uses 1.9 on the new ubuntu 14.04. 14.04 contains gdal 1.10, so using the postgres PPA downgrades gdal. That cannot but break a lot of things for us.

• I just discovered a site that couldn’t possibly work. It needs the ubuntugis-stable PPA as it needs a recent spatialite. But it also needs the postgres PPA for the 1.9 gdal! And those two don’t match.

• It still works, though. I’m not totally sure why. On a compilation machine where we build a custom debian package for one of the components, the postgres PPA was installed manually outside of the automatic build scripts. And a jenkins server where we test it still has the ubuntugis PPA, but somehow it still has the old 1.9 gdal. Probably someone pinned it?

• Another reason is probably that one of the components was compiled before the 1.9/1.10 gdal change and didn’t need re-compilation yet. Once that must be done we’re probably in deep shit.

• If I look at some ansible scripts that are used to set up some of our servers, I see the ubuntugis PPA, the mapnik/v2.2.0 PPA and the redis PPA. Oh, how can that ever work? The software on those servers needs the 1.9 gdal, right?

• I asked a colleague. Apparently the servers were all created before june and they haven’t done an “apt-get upgrade” since. That’s why they still work.

Personally, I think the best way forward is to use ubuntu 14.04 LTS with its recent versions. And to stick to the base ubuntu as much as possible. And if one or two packages are needed in more recent versions, try to somehow make a custom package for it without breaking the rest. I did something like that for mapnik, where we somehow needed the ancient 0.7 version on some servers.

If a PPA equates to never being able to do “apt-get update”, I don’t really think it is the best way forward for servers that really have to stay up.

Does someone have other thoughts? Other solutions? And no, I don’t think docker containers are the solution as throwing around PPAs doesn’t get more stable once you isolate it in a container. You don’t break anything else, true, but the container itself can be broken by an update just fine.