I recently switched one of our django sites over from apache+mod_wsgi to apache(+mod_proxy)+gunicorn. The advantage of gunicorn is that every site runs in its own process instead of everything running within one apache. When a site barfs, you can at least see the culprit when you type “top”. Oh, and the speed seemed higher. (For an introduction on gunicorn, see my summary of the gunicorn talk at last year’s djangocon.eu).
But today a colleague using IE8 couldn’t load the PNG images. And in my log I’d get “connection reset by peer” errors out of gunicorn. I had no problems in firefox.
Assumption: perhaps django’s gzip middleware messes things up. I’ve seen gzip-related errors before. Never had a problem with it in django, but worth a try. So I disabled it. IE8 failed to work as before, but now also firefox didn’t load the images. They came in as zero-length responses!
The advantage: now I could debug it myself.
I did a
wget -S on the server itself to take a look at the headers as
close to the site as possible. First a request to gunicorn running on port
10003 (some lines omitted as they’re not relevant):
Connecting to localhost|127.0.0.1|:10003... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK Server: gunicorn/0.12.1 Connection: close Cache-Control: max-age=0 Content-Type: image/png Length: unspecified [image/png] Saving to: `image.png' [ <=> ] 17,961 --.-K/s in 0s 2011-05-13 11:28:54 (160 MB/s) - `image.png' saved 
And now the same request to apache on port 80:
Connecting to localhost|127.0.0.1|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Server: gunicorn/0.12.1 Cache-Control: max-age=0 Content-Type: image/png Content-Length: 0 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Length: 0 [image/png] Saving to: `image.png' [ <=> ] 0 --.-K/s in 0s 2011-05-13 11:29:32 (0.00 B/s) - `image.png' saved [0/0]
What? Yes, adding apache into the mix results in a zero-length response. Something is wrong. The error log showed this traceback:
Error processing request. Traceback (most recent call last): File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 70, in handle self.handle_request(req, client, addr) File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 98, in handle_request resp.write(item) File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/http/wsgi.py", line 227, in write util.write(self.sock, arg, self.chunked) File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 174, in write return write_chunk(sock, data) File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 170, in write_chunk sock.sendall(chunk) File "<string>", line 1, in sendall error: (104, 'Connection reset by peer')
Chunked? Oh, the response is being served in chunks instead of as one big response. In the gunicorn documentation on deployment, there’s a warning about needing to use one of the “async worker classes” instead of the default “sync” if you don’t use ngnix’s proxy buffering.
So I started googling for an apache config switch that would tell it to just grab gunicon’s response in one go. Turns out you can set a variable that does just that by enabling only http 1.0 for the proxy requests. See the apache documentation for mod_proxy environment settings. My apache config now looks like this:
<VirtualHost *:80> ServerName mysite.whatever # Force http 1.0 for proxying: needed for gunicorn! SetEnv force-proxy-request-1.0 1 ... lots of settings ... # Static files are hosted by apache itself. # User-uploaded media: MEDIA_URL = '/media/' Alias /media/ /srv/mysite/var/media/ # django-staticfiles: STATIC_URL = '/static_media/' Alias /static_media/ /srv/mysite/var/static/ RewriteEngine on ProxyPreserveHost On # Don't rewrite /media and /static_media RewriteRule ^/media/.* - [L] RewriteRule ^/static_media/.* - [L] # Django is run via gunicorn. So proxy the rest. RewriteRule ^(.*) http://localhost:10003$1 [P] </VirtualHost>
In the end, the two different headers I looked at with wget contained an extra
hint: one has
HTTP/1.1, the other
Summary: if you use gunicorn with a regular “sync” worker in combination
with apache, add
SetEnv force-proxy-request-1.0 1. Or use one of the async
workers (eventlet, gevent, etc).
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):