Alex Gaynor: what NoSQL support in the Django ORM looks like¶

Tags: django, djangocon

NoSQL is any database that doesn’t speak SQL, so non-relational databases. But that’s the end of similarity. Most NoSQL databases have wildly different characteristics.

Django has a problem with NoSQL databases as it is pretty much focused on its SQL ORM (object relational mapper). So Alex is going to focus on a) the current internals and b) the upcoming changes.

The current internals work on the models’ metadata. Django knows how to render a text field. And it knows what sql is needed. And it knows which sql validation it can do. And so on. It makes your brain hurt less.

Django has quite some internals regarding the ORM. Queries, compilers, backends, models. The top layer regarding queries is the QuerySet. It is database-backend independent. Queries are the next level. Semi-backend specific. It translates what you want into a a database query. The translation is a bit lossy, though. A problem with that is that a query can be given to a different backend, so if you mix in a NoSQL database into the mix, you get problems. A compiler takes a query and a connection and turns it into real SQL and executes it. A compiler is totally backend specific.

The technical problem for NoSQL is that a query thinks in terms of SQL: it already chooses join types, table aliases, chooses between SQL’s HAVING and WHERE and so on.

There are also non-technical problems: the design decisions you have to make regarding NoSQL support. Do you mimic joins? How do you handle the indices Django wants to create? In the end, you still have a problem with the various NoSQL databases: they’re different. You already have different backends for “standard” SQL :-)

A solution is to make Query do less. Push more work down to the compiler. So the compiler will do the joins, for instance (and it’ll be more efficient about it as it has global knowledge about the whole query). It also makes it probably easier to fix some other ORM bugs.

After changing the ORM in this way, a MongoDB prototype backend can be made. That’s the plan.