Note: Eric is one of the founders of convore (see the djangocon eu 2011 group on convore). His talk is on converting a django site to the realtime age: does the django heart stay in place or is it replaced with something “more current” like node.js?
In the end, he told his story as a fairy tale. Entertaining and slick. Good job!
At a certain time he thought about writing a shiny new social photo website. All the cool sites were about social photo stuff. So he build one. And it became popular! And slow. So caching to the rescue. It helped. For a while. It became even more popular. Social photos mean frantic reloading of webpages. So it became slow again. Full-page reloads. Even expiry headers didn’t help.
Good artist copy, great artists steal. So he stole some ideas from twitter. Just a quick json endpoint that listed the most recent photos. And a bit of javascript that reloads that small bit of content every five seconds. If something has changed, a button would pop up so the user could reload. The users loved it. But every user that visited any page on the site would keep hitting that endpoint every five seconds. That sure put a serious load on the database. Luckily a caching mechanism was already in place. He hooked up memcached and that sure helped for quite a while.
But the site grew and grew. And grew and grew. And again the site was hitting the borders of technology. Something new was needed, again. He tried long polling. That looked to help a bit, but he ran into problems with the limited number of available django threads/processes. Would this mean the end of django and the start of node.js?
Well, that harsh step wasn’t necessary. Only that one single endpoint was strange compared to the rest. So that endpoint was separated out for implementation in a twister-style or co-routines-based part. Twister-style implementations are straightforward and explicit, but it becomes a hairy tangly mess easily. Co-routines (“eventlet”) look much more magical, but are simpler in the end. At least, he was used to threads, which look a lot like what co-routines end up as.
When trying to hook it all up, he discovered redis. That helped a lot in hooking up all the events and responses.
The advantage of the setup until this point: the hardest-hit part is heavily optimized and the rest of the code is relatively normal django code. It works quite well. A good example for how to set up a real-time website.
But he wasn’t satisfied yet. He contemplated. He figured out that this whole setup was actually just an implementation of sockets over http. So a bit of searching was in order. Hey, there’s “web sockets”. Sounds great! Till he read more about it and his enthousiasm waned. Web sockets are a mess. Security concerns at firefox that removed the support. Microsoft that won’t support it. Inconsistent implementation.
But… he figured it was OK anyway. Client-side detection for web socket support to use it when available. Extra functionality on the server side to support it. Only to run into ngnix problems. It doesn’t support web sockets and anyway it has problems with keeping connection alive to internal proxies. He solved it (note: but I didn’t get what the solution was, something with ‘http proxy’ in its name…).
Time for more features! Group support, commenting on friends’ photos. But every new feature meant a dual implementation: on the server and on the client. Two sources for bugs. This is a problem he’s not solved yet. Right now the site works pretty fine, but there’s room for improvement.
Three main points he wants us to remember:
The whole process is incremental. Incremental improvements. Incremental steps up in complexity. Build upon your existing knowledge.
You don’t need to jump straight to web sockets if you want to build a real-time web app. For many situation regular old polling works fine, for instance.
You don’t need to throw away the baby with the bath water. Don’t jump over onto node.js because it is the next best thing for real-time apps. You don’t need to dump all your django/rails/whatever knowledge and move over.
A quick word about socket.io. It tries to bridge the various ways in which you can run real-time web apps, abstracting it away a bit. It looks ok, but there are problems that need solving. But interesting to check out nonetheless.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):
