I recently switched one of our django sites over from apache+mod_wsgi to apache(+mod_proxy)+gunicorn. The advantage of gunicorn is that every site runs in its own process instead of everything running within one apache. When a site barfs, you can at least see the culprit when you type “top”. Oh, and the speed seemed higher. (For an introduction on gunicorn, see my summary of the gunicorn talk at last year’s djangocon.eu).
But today a colleague using IE8 couldn’t load the PNG images. And in my log I’d get “connection reset by peer” errors out of gunicorn. I had no problems in firefox.
Assumption: perhaps django’s gzip middleware messes things up. I’ve seen gzip-related errors before. Never had a problem with it in django, but worth a try. So I disabled it. IE8 failed to work as before, but now also firefox didn’t load the images. They came in as zero-length responses!
The advantage: now I could debug it myself.
I did a wget -S on the server itself to take a look at the headers as
close to the site as possible. First a request to gunicorn running on port
10003 (some lines omitted as they’re not relevant):
Connecting to localhost|127.0.0.1|:10003... connected.
HTTP request sent, awaiting response...
HTTP/1.0 200 OK
Server: gunicorn/0.12.1
Connection: close
Cache-Control: max-age=0
Content-Type: image/png
Length: unspecified [image/png]
Saving to: `image.png'
[ <=> ] 17,961 --.-K/s in 0s
2011-05-13 11:28:54 (160 MB/s) - `image.png' saved [17961]
And now the same request to apache on port 80:
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: gunicorn/0.12.1
Cache-Control: max-age=0
Content-Type: image/png
Content-Length: 0
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Length: 0 [image/png]
Saving to: `image.png'
[ <=> ] 0 --.-K/s in 0s
2011-05-13 11:29:32 (0.00 B/s) - `image.png' saved [0/0]
What? Yes, adding apache into the mix results in a zero-length response. Something is wrong. The error log showed this traceback:
Error processing request.
Traceback (most recent call last):
File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 70, in handle
self.handle_request(req, client, addr)
File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 98, in handle_request
resp.write(item)
File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/http/wsgi.py", line 227, in write
util.write(self.sock, arg, self.chunked)
File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 174, in write
return write_chunk(sock, data)
File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 170, in write_chunk
sock.sendall(chunk)
File "<string>", line 1, in sendall
error: (104, 'Connection reset by peer')
Chunked? Oh, the response is being served in chunks instead of as one big response. In the gunicorn documentation on deployment, there’s a warning about needing to use one of the “async worker classes” instead of the default “sync” if you don’t use ngnix’s proxy buffering.
So I started googling for an apache config switch that would tell it to just grab gunicon’s response in one go. Turns out you can set a variable that does just that by enabling only http 1.0 for the proxy requests. See the apache documentation for mod_proxy environment settings. My apache config now looks like this:
<VirtualHost *:80>
ServerName mysite.whatever
# Force http 1.0 for proxying: needed for gunicorn!
SetEnv force-proxy-request-1.0 1
... lots of settings ...
# Static files are hosted by apache itself.
# User-uploaded media: MEDIA_URL = '/media/'
Alias /media/ /srv/mysite/var/media/
# django-staticfiles: STATIC_URL = '/static_media/'
Alias /static_media/ /srv/mysite/var/static/
RewriteEngine on
ProxyPreserveHost On
# Don't rewrite /media and /static_media
RewriteRule ^/media/.* - [L]
RewriteRule ^/static_media/.* - [L]
# Django is run via gunicorn. So proxy the rest.
RewriteRule ^(.*) http://localhost:10003$1 [P]
</VirtualHost>
In the end, the two different headers I looked at with wget contained an extra
hint: one has HTTP/1.1, the other HTTP/1.0…
Summary: if you use gunicorn with a regular “sync” worker in combination
with apache, add SetEnv force-proxy-request-1.0 1. Or use one of the async
workers (eventlet, gevent, etc).
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):