Gunicorn on apache: zero length fix¶

Tags: django, djangocon

I recently switched one of our django sites over from apache+mod_wsgi to apache(+mod_proxy)+gunicorn. The advantage of gunicorn is that every site runs in its own process instead of everything running within one apache. When a site barfs, you can at least see the culprit when you type “top”. Oh, and the speed seemed higher. (For an introduction on gunicorn, see my summary of the gunicorn talk at last year’s djangocon.eu).

But today a colleague using IE8 couldn’t load the PNG images. And in my log I’d get “connection reset by peer” errors out of gunicorn. I had no problems in firefox.

Assumption: perhaps django’s gzip middleware messes things up. I’ve seen gzip-related errors before. Never had a problem with it in django, but worth a try. So I disabled it. IE8 failed to work as before, but now also firefox didn’t load the images. They came in as zero-length responses!

The advantage: now I could debug it myself.

I did a wget -S on the server itself to take a look at the headers as close to the site as possible. First a request to gunicorn running on port 10003 (some lines omitted as they’re not relevant):

Connecting to localhost|127.0.0.1|:10003... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK
  Server: gunicorn/0.12.1
  Connection: close
  Cache-Control: max-age=0
  Content-Type: image/png
Length: unspecified [image/png]
Saving to: `image.png'

    [ <=>                ] 17,961      --.-K/s   in 0s

2011-05-13 11:28:54 (160 MB/s) - `image.png' saved [17961]

And now the same request to apache on port 80:

Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: gunicorn/0.12.1
  Cache-Control: max-age=0
  Content-Type: image/png
  Content-Length: 0
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
Length: 0 [image/png]
Saving to: `image.png'

    [ <=>                    ] 0           --.-K/s   in 0s

2011-05-13 11:29:32 (0.00 B/s) - `image.png' saved [0/0]

What? Yes, adding apache into the mix results in a zero-length response. Something is wrong. The error log showed this traceback:

Error processing request.
Traceback (most recent call last):
  File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 70, in handle
    self.handle_request(req, client, addr)
  File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/workers/sync.py", line 98, in handle_request
    resp.write(item)
  File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/http/wsgi.py", line 227, in write
    util.write(self.sock, arg, self.chunked)
  File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 174, in write
    return write_chunk(sock, data)
  File "/srv/mysite/eggs/gunicorn-0.12.1-py2.5.egg/gunicorn/util.py", line 170, in write_chunk
    sock.sendall(chunk)
  File "<string>", line 1, in sendall
error: (104, 'Connection reset by peer')

Chunked? Oh, the response is being served in chunks instead of as one big response. In the gunicorn documentation on deployment, there’s a warning about needing to use one of the “async worker classes” instead of the default “sync” if you don’t use ngnix’s proxy buffering.

So I started googling for an apache config switch that would tell it to just grab gunicon’s response in one go. Turns out you can set a variable that does just that by enabling only http 1.0 for the proxy requests. See the apache documentation for mod_proxy environment settings. My apache config now looks like this:

<VirtualHost *:80>
  ServerName mysite.whatever

  # Force http 1.0 for proxying: needed for gunicorn!
  SetEnv force-proxy-request-1.0 1

  ... lots of settings ...

  # Static files are hosted by apache itself.
  # User-uploaded media: MEDIA_URL = '/media/'
  Alias /media/ /srv/mysite/var/media/
  # django-staticfiles: STATIC_URL = '/static_media/'
  Alias /static_media/ /srv/mysite/var/static/

  RewriteEngine on
  ProxyPreserveHost On
  # Don't rewrite /media and /static_media
  RewriteRule ^/media/.* - [L]
  RewriteRule ^/static_media/.* - [L]
  # Django is run via gunicorn. So proxy the rest.
  RewriteRule ^(.*) http://localhost:10003$1 [P]

</VirtualHost>

In the end, the two different headers I looked at with wget contained an extra hint: one has HTTP/1.1, the other HTTP/1.0…

Summary: if you use gunicorn with a regular “sync” worker in combination with apache, add SetEnv force-proxy-request-1.0 1. Or use one of the async workers (eventlet, gevent, etc).

Pink bikes in celebration of the Giro d'Italia that visited the Netherlands last year

Reinout van Rees

My name is Reinout van Rees and I program in Python, I live in the Netherlands, I cycle recumbent bikes and I have a model railway.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):