Pyramid experiment, (temporarily) no Django¶

Current situation: well-structured collection of Django apps¶

Well, our current system (called Lizard) is a big structured collection of Django apps.

• “Lizard-ui” is an app with base python views (class based views) and base templates for the UI and a whole bunch of css (bootstrap-based) and javascript (jquery, openlayers, lots of extras).

  This gives you a generic layout with a header, sidebar, the correct colors. Lots of stuff to make it easy to quickly build your project (once you buy into the basic structure and learn to work with it).

• “Lizard-map” builds upon lizard-ui and adds map views. Geographical information. Some “adapter mechanism” that basically provides an interface to connect any kind of data source to our map mechanism.

  There are quite a lot of data sources that we’ve mapped to maps this way. Rendering it, querying, returning a location’s CSV data, a location’s flot/matplotlib graph, combining various data sources, etc.

  Great to get going, but you do have to get to know the underdocumented interface. And it is clunky in places.

• “lizard-something”. Lots of apps (40 or so) that use lizard-ui for the UI and lizard-map for the map stuff.

The good thing: we’ve created 20 or 30 sites based on this structure. Mixing various Django Lizard apps. Works like a charm.

Why this structure? We made it collaboratively, but there’s a big piece of me at the core. I’m used to base templates (through Plone). I’m used to lots of python packages (so I introduced buildout/setuptools/pypi). I’m used to having a structure to work in. Etc.

Why? Well, if you know “belbin team roles” you can look at mine. One of my regular roles is “shaper”. Melodramatically you could translate that with “Reinout, bringer of structure”…

Good points¶

The good things basically comes from using Django as far as possible:

• A build-in admin. Handy for quickly getting an app to work, often enough for simple admin tasks. We hardly make custom forms (apart from admin.py customizations).

• Templates, template inheritance. Django template blocks. Everything inherits from a base lizardbase.html which sets up a bunch of blocks and generic layout and hooks in most of the css.

• Views. Class based views that fill in most of the template blocks’ data with good defaults. Override what you need to, leave the rest alone. Inherit from the base templates and you save yourself a lot of work.

• Static data: javascript/css. Django’s staticfiles functionality works great. UI/map provide most of the javascript/css libraries that you need in your project. And overriding a logo simply means placing a static/lizard_ui/logo.png in your site to override lizard-ui’s default one.

• Django apps. In our case they’re pretty much pluggable as they use the lizard-ui/lizard-map base apps.

The corresponding drawbacks¶

The bad and suboptimal also come from using Django as far as possible:

• It all works because it is such a high stack. That’s also the problem. You need to buy into the whole stack and you need to work with the whole stack. And you need to get to know the whole stack.

• The code and the templates belong together. Templates have an inheritance tree; the view code also. A template needs the view that belongs to it. A view prepares what the template needs.

  This is one of the main reasons you can develop something quickly, but it also ties you very much to the template implementation. Doing a nice fresh pure html+javascript interface that wants to talk to the site? You don’t know where to start. You have to buy into too much of the stack, currently.

• Everything you do pulls in the whole universe. To get your app’s view and template to work, your app depends on lizard-ui and lizard-map. Which pulls in matplotlib (for the graphs), mapnik (for custom map rendering), etc. And a whole bunch of geo libraries. Even if you don’t need them in your app.

• Apart from getting the full base template and view functionality, even if you don’t need all of it, you also get a bunch of javascript/css libraries with specific versions even if you need a slightly different one.

Theoretical interlude¶

Lizard does almost everything with inheritance. Inheritance of view code and inheritance (if you can call it that way) of templates. Django makes that easy with class based views and with blocks in templates. At least, it looks that way to me.

From reading Pyramid documentation I get the impression that composition is a way better structuring mechanism than inheritance. I have to think about this: would it improve Lizard’s structure? It sounds very plausible.

Why the non-Django Pyramid experiment?¶

Basically, the Django view/template integration we use in Lizard has been declared dirty internally. Too integrated. Too constricting. There’s no freedom on the UI side to experiment. Everything should communicate with a REST API. The UI should just be a static directory of html/css/javascript files.

So… the drawbacks of a tightly integrated view + template + static files collection of Django apps are now bigger than the gains (speed of development/integration) we get from it. At least, those drawbacks are internally strongly suspected to restrict us, that’s why we’re experimenting.

A quote from Jacob Kaplan-Moss’ keynote at last year’s djangocon.eu conference serves well as an illustration:

Html5. Lots of enthousiasm around ajax, APIs, compatibility, css, javascript, web sockets. Basically, the web has gone up a level. Who would have thought three years ago that all this fun stuff is reliably possible? The web is open again! We’re able to dream again.

Lots of buzzwords. Real-time. Ask beyond the hype and ask what lies behind them. People in this case want responsive apps. More desktop-like responsiveness. More interaction.

The critisism: this all is hard to do in Django. The state of the art is, sadly, parallel MVC stacks. Django on one side, Backbone on the other. Syncing them. “It is hard” as in “there are fundamental parts design decisions in Django that work against it”.

Look at meteor to see what’s possible. Javascript-only. A couple of lines of javascript, mongodb, etc. Lots of things you can complain about in meteor (“callback hell”), but… just think how much effort you’d need to expend to get it working in django! To get bi-directional sync like this working. (The same goes for flask and all the other python web frameworks).

Are we doomed to callback hell and javascript? Will everything be in javascript? Jacob likes javascript, but he likes python much more.

Brainstorming, there is already something in Django. You can already push the context a bit further down the line intead of baking it directly into a template. Cannot we push it even further? Into the browser with some automatic bindings? Just brainstorming.

This illustrates the problem. Are we dinosaurs when using a Django template?

So in the project I’m currently working on we’re now doing it all new. Static html/css/js project made by our UI expert. And he’s going to define what he needs out of the REST API.

Conclusion is that we no longer need templating and staticfiles on the server side. What’s left, then, of Django? ORM/models and views.

If we’re experimenting anyway, why not write the Python part in something else than Django? Due to my background Pyramid was an easy choice. Very small Python web framework with a solid structure and a good extensionability story.

Hey, I can even use the ZODB object database. Handy, because I don’t know yet what I’ll have to do for the REST API towards the javascript client. Plain Python objects stored in that object database are easier to change.

And I don’t need templates, just an external REST API. I’ll do have to figure out how to make a basic internal admin interface. Perhaps also some javascript Backbone-based form? Perhaps even Obviel :-)

It’s experiment time! (But in the end there’ll have to be a new structure again for all of our projects. Our business is not sustainable if we have to build everything from scratch the whole time. But perhaps that composition-instead-of-inheritance idea helps us there to set it up in a better way.)