NoSQL panel¶

Tags: django, djangocon, python

Question: do you want to support special features of NoSQL DBs in the ORM even if relational databases don’t support those features?

• Yes.

• Yes.

• You choose a NoSQL database for its specific features, so those features need to be supported.

• Warning: there won’t be a common set of features that lets you move a DB transparently from one NoSQL db to another. The API will work the same way, but you will probably have to do some specific stuff.

Question: How many if any of the various NoSQL databases should Django support?

• All of them. Even filesystem or RCS…

• Try to find communality. Querying probably will have to be specific per DB.

• Will there be an SQL for NoSQL databases? Relational databases existed before SQL was there…

• Lots of talking on “what are common ways of querying?”. Javascript could end up as the default query language.

• For someone knowledgeable in SQL, mapreduce is a brainwreck.

Question: should the Django ORM try to emulate JOINs on a NoSQL DB?

• JOINs aren’t use all that often. So it makes no real sense to emulate things like that.

• Just tell everyone that it is limited support. Manage expectations, then they’re not going to complain that loudly. Just return a UseADifferentDatabaseBackend traceback.

• On some of these NoSQL databases, you could emulate JOINs. So people will try it.

• What are the write operations like? SQL loses some speed in reading, but is efficient in writing. NoSQL seems to be the other way around with, depending on the database, weird hacks and caveats regarding writing.

Question: what sort of API should Django have for a pure key/value store like Redis?

• Map to dictionaries?

• Nothing.

• Allow a bit of javascript. Allow that to be plugged in by the one that uses a “ORM”-like mapper. That provides lots of (necessary) customization.

• There are a lot of new ways of querying in NoSQL that someone new is not immediately familiar with. So ask around: for most problems there’s an existing simple solution.

Question: how are other high-level web frameworks approaching this issue?

• Nobody has done it yet. Rails isn’t doing it. Sqlalchemy said “which part of SQLalchemy didn’t you understand?” :-) (Update: see Mike Bayer’s comment below)

• We call it a “persistence layer for models” instead of an “ORM”. We have a shitty ORM but a good persistence layer for models. So by accident we’re pretty well positioned to at least attempt to support NoSQL as NoSQL is “just” another way of persisting models, technically.

Question: what about focusing on integrating with the forms, generic views and so on first and only worry about the ORM (and corner cases) later?

• That’s basically how we normally work.

• The first support for NoSQL in a release will be very basic. The next release will probably mostly have various fixes and cleanups all over the place for the benefit of NoSQL. So no big commit messages “NoSQL support”, but behind-the-scenes work.

• We need NoSQL support NOW. Model forms support for NoSQL can wait. First the basic stuff. Something like i18n pages are way simpler with schema-less NoSQL databases with their sub-structure support.

One of Django’s strong points is the amount of reusable applications. What about getting them to work transparently with NoSQL backends?

• Try it out from application to application and see what comes out.

• Piston helped to get not an object, but a dictionary out of django. That was the answer in that case, as a dictionary can be stored just fine in NoSQL.

• Switching won’t happen. It is a choice that you make in the beginning. Switching from mysql to postgresql can already be a pain. And I think something big like saatchmo won’t ever be able to support both sql and NoSQL.

• Class-based views (instead of view methods) would help a lot. It makes it much easier to selectively override/fix parts of a view (in order to support NoSQL in this case).

Question: realistically, in which version can we expect some sort of NoSQL support?

• Not in 1.3. We already announced that it would be a quick short release with mostly bugfixes and cleanups.

• We’ll need much testing with NoSQL support. At least 3-4 months of testing after a feature freeze. So if we’d want it in 1.3, 1.3 would only come out 1.5-2 years from now and that’s waaaaaay too late. So probably 1.4.

• There is code around that helps you to already get it working. So there’ll be code in 1.3 to support that better. But the real full NoSQL support will come later.

Question: what about keeping NoSQL backends out of the core and focusing instead on making adding backends easier?

• Rails already did this.

• We’re thinking about cutting the core up a bit and decoupling the release cycles of the parts.

• It is quite easy to make a backend! The backend API is very good and friendly. Those who made that API did a great job. In that sense, it is a solved problem.

Question: how to keep model forms and so working with the new databases?

• We’ll lock Alex up in a room and wait for him to solve it.

• We’re waiting for Simon to post more complete thoughts to the mailinglist.

• Mongokit already has some model forms support.

• Various people are already working with NoSQL in django. So it is plainly possible.