Django under the hood: twisted and django - Amber Brown¶

Tags: django, djangocon

(One of my summaries of a talk at the 2015 django under the hood conference).

Amber Brown is a Twisted core developer. This talk’s summary will be:

>>> Django == good
True
>>> Twisted == good
True

And everything will be better if we work together.

Synchronous and asynchronous. Synchronous code is code that returns inline. Asynchronous code is code that returns something possibly at a different time. The extra complication is that IO is often blocking.

Twisted is asynchronous. Regular python code like socket.write() is blocking. Twisted has its own socket that calls python’s behind the scenes. In user code, you should only use twisted’s version: then your code is async and it isn’t blocking on IO.

At the core there’s always something that tries to read/write data. But we normally work at a higher level. So there are protocols that we actually use that are build upon the lower-level read/write connection.

A lot of the async code works with callbacks. You call a function and pass in a second function that gets called with the result when the first function is ready. Twisted uses it often like this:

>>> deferred = Deferred()
>>> deferred.addCallback(lambda t: t+1)
<Deferred at ...>
>>> deferred.addCallback(lambda t: print t)
<Deferred at ...>
>>>deferred.call(12)
13

They’re trying to use more of the recent python 3 syntax goodness to make working with this easier. Generators, yield from, etcetera.

Now on to django.

Django does blocking IO. Making this asynchronous is hard/impossible. Everything has to cooperate or everything falls apart. It is hard/impossible to “bolt it on” afterwards.

People use to think “sync=easy, async=hard”. That’s not the case, though. Both have their own advantages and drawbacks:

• Sync code is easy to follow. One thing happens after the other. A drawback is that you can only do one thing at once. Persistent connections are hard.

• Async code is massively scalable. handling persistent/evented connections is super easy. Python 3 adds syntactic sugar that makes it easier to write. A drawback is that you can get into a “callback hell”. You have to be a good citizen: blocking in the reactor loop is disastrous for performance.

A way of running django is a threaded WSGI runner. Each thread can be blocking on IO, but you have lots of them. You could look at hendrix, a WSGI runner that can run django and which also includes websocket support.

There’s something new for django: django channels (note: corrected the link 2015-12-14, I pointed at django-channels.readthedocs.org instead of channels.readthedocs.org). Requests and websockets are now events that can be fed via channels to queues. Workers can grab work from the queue. When ready, the channel feeds it back to something on the other side. It supports websockets.

With django-channels you can use the @consumer('django.wsgi.request') decorator to subscribe to some queue.

It doesn’t really make django code asynchronous. It is “only” a way to use synchronous code in an async way. But that might just be enough! It is a big improvement for django and it is better than the current approach. There are talks of integrating django-channels in django core when it is polished a bit more.

But: adopting an asynchronous framework (=twisted) is a long-term way forward. Otherwise we keep bolting patches on a request/response mechanism that isn’t suited very much to the modern web.

Then we got shown an example of django running with an async ORM and handling requests in an async way. It was a quick hack with lots of bits missing, but it did work. It would probably work very fast on pypy.

Python 3.4 has the “yield from” statement that lets you use return in the function you’re calling. Python 3.5 has even more goodies like await.

Twisted is trying to modernize itself and trying to get more people onboard. A django-style deprecation policy. Removing 2.6 support. Using new python 3.4+ features.

Django should run on twisted!

And what about greenlets? Bad… Just read https://glyph.twistedmatrix.com/2014/02/unyielding.html

Image: television interview with not-quite-completely-painted scale figures on my in-progress ‘Eifelburgenbahn’ 1:87 railway layout.