I had some problems in some of my Django websites:
An external mysql db that is slow to respond or unreachable. Normally a request would take a few seconds tops, but today it takes 1.5 minutes. Long enough for nginx to kill off the request after 30 seconds.
A c-level library (so: non-pure-python) that spins 1 cpu for 100% in corner cases. (Mapnik rendering a shapefile with an error). A second request means two eternally-spinning 200% processes. And so on.
What I hoped was that gunicorn would kill off the offending process after a while. That was one of the reasons to switch to gunicorn. But the reliable-killing-off didn’t seem to be happening reliably. I couldn’t really put my finger on it.
Mysql problem: in some cases, the child got restarted and everything is OK. In other cases gunicorn is starting up child processes all the time but nginx (which I’ve got in front of it) keeps spitting out 502s. In those cases where it goes wrong, something like 60% of the requests goes OK, 40% hits a faulty process or whatever and 502-bad-gateway-s.
Shapefile problem: gunicorn simply never killed off the eternally-100%-cpu processes.
The one thing that stands out is that both problems are related to non-pure-python libraries. Libraries with some c-level code in them. Perhaps gunicorn cannot handle them right?
I tried out the
--preload option to gunicorn. This loads the application in memory
beforehand instead of when the individual worker starts. Perhaps this helps
with some global import-time state in one of those offending libraries? I
Hurray! If I use
--preload, the mysql problem goes away. “Away” in the
sense that newly started workers work reliably. At least in my current test.
… and hurray, with
--preload, gunicorn also reliably recovers from a
faulty shapefile (that would have it spinning 100% indefinitively otherwise).
I don’t fully understand the reason. My guess is that
better with not-pure-python libraries. If you have a similar problem, it might
be worth a try.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):