Circus: process and socket manager - Tarek Ziadé¶

Tags: django, djangocon

Tarek Ziadé can’t believe he’s giving a talk about his circus process manager in an actual circus tent :-)

A typical deployment is with nginx and gunicorn or uwsgi. But you add more and more items right next to your django process(es). Celery or haystack for instance. So you add a supervisor that starts ‘em all. An often-used one is supervisord. You could use a system-level tool like upstart, but you need root access for that. You don’t need it for supervisord.

Supervisord has some missing features like a powerful web console, clustering, realtime output, remote access and so. Supervisord has some of this, but not good enough. So they (mozilla) started with Circus.

They used several existing libraries, like psutil, zeroMQ, socket.io. psutil is the core of the system. Very handy for interacting with processes. It was a bit slow, but together with the psutils author they managed to make it fast.

ZeroMQ is an async library for message passing, so more or less a smart socket. They use message passing for making the various process data available to the circus tools, like ‘circus-top’ or ‘circusd-stats’.

And because everyting is nicely decoupled, it is possible to add your own plugins for custom interaction. There are already community-provided plugins available.

He showed the web interface: looks nice. Live graph per process with memory and CPU usage. A simple “+” to add an additional process.

One last thing: there are multiple levels of supervision. Supervisord or circus must be started in some way by the system. And gunicorn, launched by supervisord or circus, itself starts up django processes. They added chaussette as a wsgi runner that can also run new processes on already-opened sockets so that they can be managed by circus, too.