Reinout van Rees’ weblog

Omnifocus GTD contexts


Tags: gtd

Following my blog post about grouping projects in omnifocus, I’ll also write down how I structure my contexts.

As a reminder, with the GTD method you group your actions according to two “axes”. An action has both a project and a context. So action “buy brake pads” has context “at the bicycle shop” and project “bike repairs”.

Having a project is handy for planning. Do I have everything I want to do in my system? Are there new actions I should add to bring my projects forward? Why do I want to have this project? What is the goal? Those kinds of questions.

Contexts are often handy for doing. First the non-context kind of doing, though: often I’ll just work on a project. I’m adding a software feature and I’ll just keep doing what needs doing there, whether it is talking to someone at the office or typing away on my keyboard or googling something.

But... contexts are very helpful if you use them well. What is a context? “DIY store” for intance. With all the actions in that context, I have my shopping list ready! Whether it is a bucket of paint I need for the “paint the staircase” project or some wood for the “build the model railway station” project or a bunch of screws I need for the kitchen... I have all my actions in one place.

Same with my “whiteboard” context. Sometimes I like to stand in front of the whiteboard in my study room at home: brainstorming! There are often a few projects that I need to brainstorm: do I want to do the project? What is the goal? What are tasks I can do? Having a list of I-need-to-brainstorm actions is handy.

Like for projects, I use a few folders to group some contexts. This is extra helpful for projects, as items in the sub-context show up in the parent. The “city” context has sub-contexts for some of the shops I visit regularly like “supermarket”, “DIY store”, “ikea”, “model train shop”. So when I’m in the city I just check the main “city” context to see if I need to get something.

Without ado, here’s an example list of my contexts. I’m fine-tuning it right now as I’m setting up my system anew. And I’ve left out some personal ones. Anyway, here it is:

  • City.

    • Supermarket.
    • Electronics shop (which can also mean “order online at”).
    • Bicycle shop (again, online ordering is fine).
    • DIY store.
    • Model railway exhibition (either an exhibition or a regular store or online or ebay).
    • Ikea.
    • Hema (a specific Dutch store).
  • Whiteboard (see above).

  • Home.

    • Study room.
    • With tool box (so: generic repairs, mostly).
  • Computer

    • Ipad.

    • Work (the bulk (almost everything) of my generic work tasks are on my computer, this is just to separate them from the tasks for personal projects).

    • Personal.

    • Google. This is for looking things up on the internet. “Search for general Balck as I read about him in a book”. “Look up 24 oktober, I just passed a tram stop called ‘24 oktober square’ and I wonder what the date means”.

      Stuff I don’t want to do right now, but I want to look it up later.

      24 oktober is the start date of the United Nations, btw. Which makes sense as the “24 oktoberplein” square in Utrecht is near the “United Nations street”. That’s the kind of knowledge you accumulate when you’ve got a context you can quickly drop in a few tasks without impending progress on whatever else you’re doing right now!

    • Omnifocus. GTD maintenance :-)

  • Telephone.

  • Church.

  • Office.

  • Sometime/maybe. Stuff I don’t want to do right now, but might want to do later on.

  • “To idea list”. Special context that I use for all sorts of ideas. Ideas pop up in my head all the time. And often they’re not about what I’m currently working on. So I just quickly add an action to omnifocus and continue working.

    Such ideas are often not immediately actionable, so I just dump them to a text file that I check once a month. This way I don’t clutter my omnifocus database with too many weird stuff. It is a bit of an alternative to the someday/maybe list.

Perhaps this list helps someone to make better use of his GTD contexts!

Omnifocus folders: grouping projects


Tags: gtd

I’m currently updating my GTD setup. I use omnifocus for that.

One of the handy features is that you can group your projects in folders (and subfolders, should you be so inclined, which I’m not). I use it to bring a bit of order in my huge project list, as I have 50-100 projects, which is quite a normal number if you don’t make your projects too big.

I use a couple of targeted folders:

  • Big long-term projects, like building my model railway. I don’t want to turn that into one mega-project. Instead I have smaller specific projects for digitalizing a certain locomotive, figuring out the track plan for one of the stations, building a station building.

    Another example folder from a few years ago was “preparing the new house” with projects like “new window in the roof”, “install the kitchen”, “paint the staircase”.

  • Longer-term roles. Half a year ago, my four year period as my church’s secretary ended. So... that was a folder, too. A project per meeting, as 50% of my work was meeting-centric. Preparing the calender beforehand, gathering info, mailing people. Handling the tasks resulting from the meeting afterwards.

    Omnifocus made it easy for me to put something on my todo list for a meeting half a year in the future. I don’t think I ever forgot anything. GTD realy shone for me during this time! I’m not that organized and neat by nature. The fact that everybody was happy with how I did it is a testimony to GTD :-)

    The other projects in there were for specific projects like “clean out the archive”.

    Next year a big part of my work will be devops/sysadmin/cleanup/quality work. That’s a nice big role that deserves its own projects to drive it forwards.

The rest of the projects is grouped into:

  • Generic work.
  • Personal.

Splitting projects between the two, for me, is practical. As with any ordering, there are projects that are bound to fall in two categories at the same time. You’ll just have to make a judgment call for them. In my case, some open source projects are both a personal hobby and they’re use a lot at work. So sometimes I work on them in the evenings, sometimes during office hours. Ah well.

An additional comment about the targeted groups: they are great for focusing attention and ensuring progress. So I use them sparingly. Every year I try to pick three goals/areas upon which I want to focus. One or two of them invariably ends up as a folder. This just seems to demand projects and progress :-)

GTD thoughts


Tags: gtd

I’ve been set to think about my GTD setup again because of three reasons:

  • I haven’t done any maintenance on my GTD system (omnifocus in my case) in months. So I’ve been more or less been working by accident and by deadline in the last months.

    One note I’d like to make is that I’ve effectively dropped most of my home/private projects and that I’ve worked almost fulltime on my model railway (see a full description in Dutch). So my work was cut out for me: just get it working. I did some brainstorming and planning of course, but the whole process was quite clear.

  • I’ve got a new project for which I really want to keep track of what I need to do. But I noticed I didn’t trust my own system for that... Time for long-overdue maintenance!


    A recent David Allen (GTD creator) video. 20 minutes. With a focus on projects. And that’s the number one area where I have problems getting the system to work right.

Some of my problems with projects:

  • I have no problems thinking up new ones. I have lots of them.

  • They tend to overlap if they’re not too clear. Or they’re too big.

  • Many of them run for a long time.

  • Many are never worked on. So I should probably scrap them as they’re not worth doing.

  • I keep them around anyway.

  • I want to structure them in folders, which sometimes gives too much structure. Or I’ve got a huge plain list that really needs structuring. Aargh.

  • Main problem: many projects have lots of actions. I tend to brainstorm and put all the tasks I can think of in the project. Followed by some ordering (“this needs to happen first”). In practice I need just one task to get started and then the rest follows along. Which means I next have to go through the list of actions again and re-order them.

    Should I only keep one or two (next) actions in there? Should the brainstorm results be stored somewhere else? I can re-brainstorm easily. I often do that anyway when re-starting work on a project.

  • Should I re-start with a clean list once a year (like, right now)?

Back to that recent video I mentioned. Per project (which is defined as a multi-step something you want to accomplish) it defines five stages. In GTD terms: “the natural planning model”.

  1. What is the purpose? From the video: “Some real intention. Something you need to have occur or happen. What is the main reason this thing exists? What is the primary purpose for this thing?”

    So the purpose of a project isn’t “my bike’s light must be fixed”. That’s more the title of the project. The purpose is something at a higher level. “My bike commute should be safe”, for instance.

    The purpose connects your project to the whole of your life. That might sound a bit melodramatic for mundane projects. Something simpler like “because its my job to fix misbehaving websites” is fine, too.

  2. What is the vision of success. Paint a picture in your mind of the successful end result. “My bike light works again and keeps working without any maintenance for the next three years”, for instance.

    Painting the end result helps you get ideas. Your brain automatically notices a difference between the current state and the desired future state. So it starts coming up with ideas (see the next phase).

    As an example, look at your desk if it is cluttered with papers and stuff. And then imagine it completely clean. Really imagine it. Notice how your brain starts planning automatically?

  3. Brainstorm ideas. With a good picture in your mind of what success means, your brain will start doing this automatically. Just write it down. Or use a whiteboard or so.

  4. Structure the brainstormed ideas. For instance, pick the three most relevant or important items. What is big? What is little? What is important to do? What is critical? What is a priority? What is (im)possible?

  5. Pick the next action. “This might be the most important part of the natural planning model” according to David Allen. What is the very next physical action you can take on this project.

    It needs to be very specific and visual. “Fix the light” is too vague. “Unscrew the current defective light” is fine.

There are some ideas I got out of this. Main take-away is to not just create a new project and slap some actions on it. I need to plan a project properly. At least I’ve got to write down the project’s purpose and vision. That can go nicely in omnifocus’ comment field.

Part of the vision should end up in the title of the project to make it more explicit and attractive.

And... I don’t need to store my brainstorm results (“a huge list of possible actions”) in my project. Perhaps on a paper note somewhere else? Perhaps just throw it away? Just keep the important ones and/or the actions I promised others to do. And the next action.

What I’m probably going to do:

  • Look through my current projects. Which ones are on that list for more than a year? Why didn’t I finish them? Are they ill-defined? Did they just lack priority?
  • Go through my current actions. Flag the ones that still need doing and print them out.
  • And then.... restart with a blank slate in omnifocus. That’ll take a day!

Good to be thinking about this!

Naming things: don’t use reserved words


Tags: python, django

Update: added postfixed-with-underscore examples.

I’m currently cleaning up some code. Some people just cannot spell “p-e-p-8” if their lives depended on it, apparently. Luckily I’m civilized so I only threaten with dismemberment.

Look at this gem I just uncovered:

tg_after = instance.filter.technischegegevensfilter_set.exclude(

Don’t mind about the Dutch in there. Just look at those two filter words in the first two lines. They’re even nicely underneath each other. At least they are now, I first had to fit the 159 characters long line within 78 characters, of course.

In Django, you do sql filtering with .filter(some_condition=42). That’s not what’s happening in the first line, though. There’s a foreign key called filter there! So the first filter is the name of a foreign key and the second filter is the filter method that is used everywhere in Django.

Big confusion. And big chance that someone else that reads the code messes it up somehow.

So... steer clear of common words used in your programming language or framework or whatever. Some examples:

  • Don’t use type as an name. Use customer_type or station_type or whatever. Only use type by itself if you really mean the python build-in. Alternatively you can postfix it with an underscore, so type_
  • Don’t use class. Either use the often-used klass or class_ alternative if you want to keep it short. But why not use css_class if you want to return a css class, for instance?
  • Don’t use filter for Django models. Even if you’re modelling filters that are used for groundwater extraction (as in this case). Call them WaterFilter or so.

So... you can now go and fix your code. You’ve got about 45 minutes before I’m ready sharpening my axe.

Summary of my “developer laptop automation” talk


Tags: python, django

Last week I gave a talk at a python meetup in Eindhoven (NL). I summarize Therry van Neerven’s python desktop application development talk, but I didn’t write one for my own “developer laptop automation” talk.

Turns out Therry returned the favour and made a summary of my talk. A good one!

Django under the hood: django migrations - Andrew Godwin


Tags: django, djangocon

(One of the summaries of a talk at the 2014 django under the hood conference).

Andrew Godwin wrote south, the number one migrations framework for django. It is superseded by django’s new built-in migrations, also written mostly by Andrew, hurray!

The original idea was to have a schema backend and hooks in the django ORM. The actual migration code would live outside of django in south2. In the end, everything is now in django. The original distinction between “schema backend stuff” and “the actual migrations” is still there in the code, however.

The schema backend is relatively simple and straightforward; the migration part is hard and hairy. The migration part contains: operations, loader/graph, executor, autodetector, optimiser, state. He’ll talk about some of them here.

What about the old syncdb? It is a one-shot thing: you add tables and then you add the foreign keys. When migrating, you have dependencies. You cannot add foreign keys to tables you haven’t added yet. There is automatic dependency-detecting code, now, but that was added quite at the last moment in the 1.7 beta 2...

Basic dependencies means the obvious stuff. Some examples:

  • Like creating a model before adding a foreign key to it. Most databases get fussy if you try to add the foreign key too early.
  • Create the model before creating the fields. Sounds simple, but you need to have these basics in place first in the dependency graph before you can get on to the hard cases.

Now on to the more creative dependencies.

  • For many-to-many fields you need both target models first before you add the actual M2M model that points at the targets.
  • Multi table inheritance? Create the MTI parent before the child.
  • “Unique together” constraints need to be done after adding the fields. Same for “index together”.
  • “Order with respect to” is a rarely used feature that adds an extra field with an ordering based on a foreign key field. He started dispairing when he discovered this feature.
  • Proxy models. Weird things need to happen when you actually turn proxies into real models and want to do that in a migration. It is a one-line change for a developer, but it makes for nightmares in the migration code. “You have to create the model with the same name before you can delete the model with the same name”. Yes, that’s not a typo.
  • Swappable models! Please take them away! Swappable models? For instance the User model that you can replace with a different custom model. Suddenly a migration that you already applied might need to point at a different model. Rolling back the migrations is no option, as that leads to data loss. It works fine if you do it at the start of the project.

He used a different mindset when developing django’s migrations as opposed to how he developed South. South depended on people reading the documentation. Which they often don’t do. So they could shoot themselves in the foot quite well. Instead, django’s migrations are much more bulletproof, so there is much less need for reading the documentation in detail.

There’s a main loop in the migrations code that tries to find dependencies, shifts operations to satisfy the dependency, checks if everything is fine, and loops again and again until it is right.

The way it works is by chopping all operations into tiny dependencies. Every individual field that has to be created is turned into a tiny dependency step. After the list of steps is sorted (via the dependency-resolving loop) into the correct list of steps, an optimiser goes through the list and optimises it. If a model gets created and deleted, nothing needs to be done, for instance.

This kind of reducing could be dangerous. So there’s another loop that checks which reductions/simplifications are possible. Whether there are conflicts. It is better to have no optimisation than to have a wrong optimisation.

Reduction is applied after various stages: after the automatically detected dependency code. After applying the manual dependencies. And after squashing.

Squashing: it makes your history a bit shorter. It squashes migrations into a new starting point. This is especially handy when you’re a third party app developer.

The final part of the puzzle is the graph. It builds a directed graph of all basic migrations in memory. It needs to read all the models on disk for that. It also looks in the database. There’s a table in there that marks which migrations (or rather: nodes in the migration graph) have been applied.

A squashed migration lists the graph nodes that it replaces. A squash can only be applied if all the replaced nodes have the same state. They either all are unapplied: then the squash is applied. If they’ve all been applied, the squash can be considered as applied.

There’s room for improvement!

  • The autodetector is slow.
  • The optimizer is not great.
  • Graph state building is inefficient. Very inefficient. It might take 30 seconds. Building the state itself isn’t that slow, but it simply happens thousands of times.

It is mostly a case of un-optimized code. Big, pretty dumb, loops. So: everyone’s invited to help out, for instance at the sprint.

If you want to look at the code, here are some pointers:

  • django/db/migrations/, start at _detect_changes()
  • django/db/migrations/, start at reduce()
  • django/db/migrations/
  • django/db/migrations/

His slides are at

Come work for us!

Django under the hood: internationalisation - Jannis Leidel


Tags: django, djangocon

(One of the summaries of a talk at the 2014 django under the hood conference).

Jannis Leidel tells us about django internationalisation (or should that be internationaliZation? Well, Jannis confessed to changing it to an S on purpose :-) ) Jannis used to maintain django’s translations a few years ago. Now he works on the mozilla developer network website, which has translations into more than 30 languages.

He showed a picture of the Rosetta stone. That stone turned into a tool to transmit the previously-unknown egypt hyroglyphs to our times. It was the vessel that brought lots of Egypt culture to the present time.

Isn’t what we do with our Django websites something that similarly can transfer knowledge and culture? But there is a problem...

56% of the web is English. 5% is a native speaker. Only 20% of the world population has a basic knowledge of English!

Another example? 0.8% of the web is Arabic. 4% is a native speaker. 0.1% is Hindi, with 5% native speakers.

Multilingual and multicultural... There are huge differences. The future is a global web, with lots of fresh new internet users coming online in the coming years. Users that don’t speak English and largely aren’t “western”. How do we get to a global web? There are three long words that are relevant:

  • Internationalization, “i18n”. The preparation phase of making your software at least translatable. Django has tools for this, for instance {% blocktrans %} in your templates. And gettext() in your python code.

  • Translation, “t9n”. Actually translating the marked-as-translatable strings from the i18n phase. Additionally you can translate your content. A different problem altogether is translating your project’s documentation.

    The files used for translation are normally the GNU gettext .po files. A technical format.

  • Localization (l10n). Different representations for numbers or dates and so. “5 mei 2014” versus “May 5th, 2014”.

    In django, everything is done automatically if you have USE_L10N = True in your settings file.

In django, look at django.utils.translation. It is an old part of the code. There wasn’t much web tech experience at that time. It contains a “thread local” that accesses the request, for instance. Various parts really should be replaced.

How does django detect the current language?

  • URL path.
  • Session data from a previous visit.
  • A cookie.
  • The Accept-Language request header.
  • Django’s LANGUAGE-CODE setting.

We have tons of information here, but it also means that it isn’t quite clear which language the user is actually going to see.

Most of the tools the open source generally uses come from the GNU Gettext project. It isn’t particularly bad as such, but we’re pretty much bound to gettext and cannot really use any improvements made in other toolsets.

The normal gettext string extraction workflow means running makemessages, sending the untranslated .po files over to the translators. Get them back, run compilemessages to generate the resulting .mo compiled translation file. This is very technical. A very technical solution to a non-technical problem.

It gets worse with javascript. We have to use a javascript file that contains a gettext implementation and that actually contains all the translations. It is a wonder it still works. (But, yes, it does work).

Timezones are aonother area. We use pytz for that, which works well.

The missing pieces in django. This are the bits he wants us all to add to django to support all those upcoming 4 billion/milliard (depends on your locale, I mean 4*10^9) new users.

  • One way to do this is to use more of the unicode CLDR database. We only use part of this. Lots of locale-relevant data.
  • Translation catalogs could be made handier. Merging them for instance. Visualize which strings still require translation. Progress indicator.
  • Another area: translating content. The content in the django database. There are quite a lot of add-ons that allow this. All with different approaches. But... all in all there are three basic approaches to do it. All three must be moved to django core!
  • For translating content, editing support could be handier, for instance a CompositeField to aid translation in multiple languages.
  • We can add helpers, for instance for visual translation with a sync to PO files. Just a small javascript helper that gives you a popup to translate a single string.
  • And... where is the common language selector? We all implement such a thing from scratch in all of our projects, right?
  • And what about a timezone selector? And can’t we suggest a way for browsers to pass along the user’s current timezone as part of the request? Cannot we lead that?
  • Use the hostname, too, when detecting the language.,, etcetera.

Call to action

  • Rewrite our translation infrastructure on top of Babel. Advantage: it uses much more of the unicode CLDR project, bringing us much closer to many more people’s culture.
  • Push the boundaries. We can have lots of influence as a big open source project.
  • Make i18n a first class citizen. Remove the USE_I18N setting. It should be on by default.

His slides are on

Come work for us!

Django under the hood: model _meta - Daniel Pyrathon


Tags: django, djangocon

(One of the summaries of a talk at the 2014 django under the hood conference).

Daniel Pyrathon talks about django’s Model._meta and how to make it non-disgusting. He worked on it via the google summer of code program. His task was to “formalize the Meta object”.

The Meta API is an internal API, hidden under the _meta object within each model. It allows Django to inspect a model’s internals. And.... it makes a lot of Django’s model magic possible (for instance the admin site).

What’s in there? For instance some real metadata like “model name”, “app name”, “abstract?”, “proxy model?”. It also provides metadata and references to fields and relations in a model: field names, field types, etc.

Which apps use it?

  • The admin.
  • Migrations.
  • ModelForms.
  • ... other developers. Developers have always used it, even though it is not an official API! Developers shouldn’t be using it as it is internal. You really need it however for things like django-rest-framework.

So... There’s a big need for a real, public API.

There is an important distinction between fields and related objects. A field is any field defined on the model, with or without a relation. Including foreign keys. Related objects are a special case: they are objects that django creates on objects if there’s for instance a foreign key pointing the other way. This distincion is how django likes to work internally. It does lead to a little bit of duplication regarding the API.

There are about 10 functions (“entry points”) in django that make use of _meta. And 4 properties. And there are 6 separate caching systems for the API... many_to_many, get_field, get_all_related_objects, etc.

The new Meta API’s philosophy:

  • An official API that everyone can use without fear of breakage.
  • A fast API, that also Django’s internals can use.
  • An intuitive API, simple to use. And documented.

The new API has only 7 entry points. Well, really only two: get_field and get_fields. The other five are fast cached helper functions to make the API easier to use.

There are three intuitive return types.

  • A set of field names.
  • A field object.
  • A set of cached properties, for instance a set of fields.

The new Meta API is properly tested. The old _meta was “only” tested by the entire set of django tests. The new one is explicitly properly tested in isolation.

get_fields is the main method that iterates through all the models, handling inheritance and so. In every loop through a model, the result is cached, leading to more performance.

For related objects, a complete graph of all the models with all the fields is needed. This is an expensive one-time operation which is cached afterwards.

Sidenote: what is used often in Meta is the cached_property decorator. It is a property that is only computed once per instance. It prevents lots of unnecessary re-calculations.

cached_property is included in django. You can also install a generic implementation from (Note: at the bottom of the README there, I get thanked for calling cached_property to pydanny’s (Daniel Greenfeld’s) attention. Funny :-) )

Cached_property means the five extra cached properties (for grabbing related objects, for instance) are essentially free. They don’t have any overhead as they’re computed only once.

An important concept in the Meta API: immutability. This helps prevents lots of bugs. If you return an immutable result, you can be sure it cannot be changed (of course). An advantage is that they’re quick. You can also use itertools.chain() to avoid allocating a new list. You can make a copy of everything as a list, of course.

Fun fact: it seems that the Meta API and its optimizations give django a 10% performance boost.

He showed some additional ideas for future improvements. He’ll discuss them tomorow at the sprint.

Come work for us!

Django under the hood: python templating - Armin Ronacher


Tags: django, djangocon, python

(One of the summaries of a talk at the 2014 django under the hood conference).

Armin Ronacher is the maker of many non-Django things such as Flask and Jinja2.

His initial thought was “why are we even discussing templates in 2014”? In 2011 everyone started making single-page applications. But a year later it turned out not to be too good an idea: the cloud is much faster than your mobile phone. So server-side rendering is hip again.

Django in 2005 had one of the few available template languages that were any good. It got imitated a lot, for instance in Jinja2. Which is fast.

He doesn’t really like looking at his own Jinja2 code. For him, it is pretty early Python code and it could be so much nicer. Feature-wise Jinja2 hasn’t changed a lot in the recent years. There are a couple of problems, but they’re not big enough to warrant a fix, as any fix will invariably break someone’s template.

The template language is mostly the same, but Jinja2’s and Django’s internal design differs greatly. Django has an AST (“abstract syntax tree”) interpreter with made-up semantics. Jinja2 compiles into python code (“transpiles” he calls it).

Rendering is mostly the same .You have a context object with all data for the template, you run it through the AST or the code and you get output.

The parsing of the template is widely different. Jinja2 supports nested expressions, for instance, Django not. Django parses in two stages. With a regex it filters out blocks, statements, comments and the rest. One of the problems is that a block statement needs to be on a single line. You often end up with super-long lines this way.

The parsing inside a block in Django happens in the block’s implementation. So it is very inconsistent and often re-implemented.

Extensions? Heavily discouraged in Jinja2. It is tricky to debug due to the compiled nature. It is encouraged in Django (“template tags”). Many people build them. It is easy to implement and debugging is quite doable. It is easier to implement due to the context and the render method.

A Jinja2 template compiles into a generator yielding string chunks. The django render() functions yield strings. Any form of recursive calls generate new strings. In the end, jinja can generate huge documents, django can run into problems.

Error handling is at the python level in Jinja2. jinja2 generates python code, so it simply prints a traceback if something goes wrong. Django has its own error handling that shows less information.

A huge difference is the context. In jinja2 it is a source of data. It only holds top-level variables. In django, it itself stores data. It holds all variables. Basically, it is a stack of dictionaries.

Autoescaping. In django, it is django-specific. It largely lives only in the template engine. Jinja2 uses markupsafe, which is pretty much standard in the python world. Escaping is handled in python. (Django also uses markupsafe one-directionally).

He showed some example django and jinja2 code. Fun to see the generated python code coming out of a short jinja2 template! He also showed some of the optimizations that jinja2 can do. Impressive.

Fun: he hacked the C python traceback functionality to get good tracebacks out of his templates. Imagine seeing simply {% for i in something %} in a traceback line...

Armin tried to use jinja in django for a google summer of code project. But in the end, it boils down to philosophical differences:

  • People like jinja because of its python expressions and because of its speed.
  • People like django’s templates because of the extensibility.

Why can’t django just use jinja2? Jinja needed to sacrifice certain functionality. Doing the same in Django would break everybody’s code.

The problem is solved to a sufficient degree that he thinks no big changes will happen.

Come work for us!

Django under the hood: django REST framework - Tom Christie


Tags: django, djangocon

(One of the summaries of a talk at the 2014 django under the hood conference).

Tom Christie will give a two-part talk. The first is about django REST framework, the 3.0 beta will be out tomorrow. The second part is about API design in general.

REST framework 3.0

They made a few fundamental changes in 3.0. Why did they do this?

For instance for serializers. Serialization is from an object to a serialization format like json. It is is the deserialization that is the hard part: you have to do validation on incoming data and transform it if needed and convert it to an object. How it happened till now:

serializer = ExampleSerializer(data=request.DATA, ...)
# serializer.object exists now

This .is_valid() performs some validation on the serializer and it also instantiates an object. In 3.0, the validation still happens, but it only works on the serializer, not on an object. It returns validated data instead of an object. The validated data can afterwards be saved to an object explicitly.

Another change is that there’s a separate .create() and .update() method to split out the behaviour regarding creating new database objects or updating existing ones. The previous version determined itself whether a create or update was needed and it was often mixed in a wrong way.

This change was done because of relations. It is tricky to save a django User+Profile object when the user doesn’t exist yet. A profile needs a foreign key to a saved user... The code here did a dirty trick. A trick that you needed to do by hand to if you wanted to do something similar yourself.

You can look at this as an encapsulation problem (see Tom’s blog article). You can use the scheme of “fat models, thin views”. The idea is to say “never write to a model field or call .save() directly”. You instead always use model methods and manager methods for state changing operations. The advantage is that all your model-related code is encapsulated in one place.

Regarding the user/profile problem, you could write a custom model manager for users with a custom create() method which always creates User and Profile instances as a pair.

In REST framework 3.0, you can now use a similar method as you can now have a .create() model on the serializer that controls the actual creation process. You can still use the ModelSerializer shortcut, but your validation behaviour is made visible and explicit. “unique_together” and field validation are helped by this.

Note: I missed something here as my late-2010 macbook pro crashed again with the hardware GPU bug that apple refuses to fix.

Field validation in 2.0 can be hard to track, because you don’t know if something failed because of a serializer validation or because of a database validation. A max on an integer field will only fail when saving to the database: in 2.0 the error would come out of .is_valid(). In 3.0, the serializer validations will run OK and only the database save() will fail. Nice and explicit.

There are now more methods that you can override on the serializer, like to_internal_value(), to_representation().

Serializers are used as input for renderers. Simple ones like a JSON renderer, but also an elaborate one like the HTML renderer for browser forms. The serializer output is almost exactly like regular form data, except that there’s an extra data.serializer attribute, pointing back at the serializer so that the renderer can query it for extra information.

There’s a way to give extra hints to a renderer: add a style dictionary to your fields. The contents of the dict is intentionally left unspecified, you can do what you want. It is the renderer that will decide what to do with it. Probably you’ll only use this to customize the HTML renderer.

The big 3.0 picture:

  • Easier to override save behaviour.
  • More explicit validation.
  • Better model encapsulation.
  • Less complex internal implementation.
  • Supports both Form and API output.
  • Template based form rendering.
  • Nested forms and lists of forms.

In the 3.1 version, you’ll be able to use the forms as exposed by Django REST framework as almost-regular forms in your django site.

Generic API design thoughts

There are quite some hypermedia formats coming out. HAL, JSON-LD for instance. He doesn’t think they’re compelling yet.

There are hypermedia formats that are successes.

  • Like plain old RSS. A simple link of lists, at the core. It is domain specific. It is machine readable. There are many tools that can read it.
  • HTML is the closest we’ve come to a universal platform. It is document centric, though, not data centric. It is not intended for programmatic interaction.

Keep in your mind one of your websites. In your mind, peel off the javascript. Peel off the css next. Then you still have quite a lot of document-centric structure left!

Can’t you peel back the html, too? And be left with the actual data? So: less documented-oriented but data object oriented? Say, an abstract object interface? A Data Object Model? That way a mobile application and a regular html application can interact with the same content.

(Note: I don’t know if he really meant the Data Object Model to abbreviate to DOM, which is obviously already the DOM abbreviation as known from html...)

A document object might have some meta attributes like uri, title, description. And other attributes like lists, objects, strings, nested documents. A “link” in turn can have a uri, some fields, etc.

Could you build generic client application on top of such a thing?

A “document” could have a ‘board’ attribute with the current state of a chess board, for instance. If you work with such a document’s Data Object Model, you could call methods that modify the board (play_move('d2', 'd4')) directly...

Also imagine the level of functional testing you could do on your application. Especially as opposed to what you could do with bare html!

According to Tom, it is time to start addressing the fundamental issues of system communication. You’ve got to think big for this.

So: how do we design abstract object interfaces and describe the actions and transforms they expose?

Come work for us! logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):