Gunicorn –preload helps with non-pure-python libraries¶

Tags: django

I had some problems in some of my Django websites:

• An external mysql db that is slow to respond or unreachable. Normally a request would take a few seconds tops, but today it takes 1.5 minutes. Long enough for nginx to kill off the request after 30 seconds.
• A c-level library (so: non-pure-python) that spins 1 cpu for 100% in corner cases. (Mapnik rendering a shapefile with an error). A second request means two eternally-spinning 200% processes. And so on.

What I hoped was that gunicorn would kill off the offending process after a while. That was one of the reasons to switch to gunicorn. But the reliable-killing-off didn’t seem to be happening reliably. I couldn’t really put my finger on it.

• Mysql problem: in some cases, the child got restarted and everything is OK. In other cases gunicorn is starting up child processes all the time but nginx (which I’ve got in front of it) keeps spitting out 502s. In those cases where it goes wrong, something like 60% of the requests goes OK, 40% hits a faulty process or whatever and 502-bad-gateway-s.
• Shapefile problem: gunicorn simply never killed off the eternally-100%-cpu processes.

The one thing that stands out is that both problems are related to non-pure-python libraries. Libraries with some c-level code in them. Perhaps gunicorn cannot handle them right?

I tried out the --preload option to gunicorn. This loads the application in memory beforehand instead of when the individual worker starts. Perhaps this helps with some global import-time state in one of those offending libraries? I don’t know.

Hurray! If I use --preload, the mysql problem goes away. “Away” in the sense that newly started workers work reliably. At least in my current test. ... and hurray, with --preload, gunicorn also reliably recovers from a faulty shapefile (that would have it spinning 100% indefinitively otherwise).

I don’t fully understand the reason. My guess is that --preload works better with not-pure-python libraries. If you have a similar problem, it might be worth a try.