Django under the hood: model _meta - Daniel Pyrathon¶

Tags: django, djangocon

(One of the summaries of a talk at the 2014 django under the hood conference).

Daniel Pyrathon talks about django’s Model._meta and how to make it non-disgusting. He worked on it via the google summer of code program. His task was to “formalize the Meta object”.

The Meta API is an internal API, hidden under the _meta object within each model. It allows Django to inspect a model’s internals. And…. it makes a lot of Django’s model magic possible (for instance the admin site).

What’s in there? For instance some real metadata like “model name”, “app name”, “abstract?”, “proxy model?”. It also provides metadata and references to fields and relations in a model: field names, field types, etc.

Which apps use it?

• The admin.

• Migrations.

• ModelForms.

• … other developers. Developers have always used it, even though it is not an official API! Developers shouldn’t be using it as it is internal. You really need it however for things like django-rest-framework.

So… There’s a big need for a real, public API.

There is an important distinction between fields and related objects. A field is any field defined on the model, with or without a relation. Including foreign keys. Related objects are a special case: they are objects that django creates on objects if there’s for instance a foreign key pointing the other way. This distincion is how django likes to work internally. It does lead to a little bit of duplication regarding the API.

There are about 10 functions (“entry points”) in django that make use of _meta. And 4 properties. And there are 6 separate caching systems for the API… many_to_many, get_field, get_all_related_objects, etc.

The new Meta API’s philosophy:

• An official API that everyone can use without fear of breakage.

• A fast API, that also Django’s internals can use.

• An intuitive API, simple to use. And documented.

The new API has only 7 entry points. Well, really only two: get_field and get_fields. The other five are fast cached helper functions to make the API easier to use.

There are three intuitive return types.

• A set of field names.

• A field object.

• A set of cached properties, for instance a set of fields.

The new Meta API is properly tested. The old _meta was “only” tested by the entire set of django tests. The new one is explicitly properly tested in isolation.

get_fields is the main method that iterates through all the models, handling inheritance and so. In every loop through a model, the result is cached, leading to more performance.

For related objects, a complete graph of all the models with all the fields is needed. This is an expensive one-time operation which is cached afterwards.

Sidenote: what is used often in Meta is the cached_property decorator. It is a property that is only computed once per instance. It prevents lots of unnecessary re-calculations.

cached_property is included in django. You can also install a generic implementation from https://github.com/pydanny/cached-property (Note: at the bottom of the README there, I get thanked for calling cached_property to pydanny’s (Daniel Greenfeld’s) attention. Funny :-) )

Cached_property means the five extra cached properties (for grabbing related objects, for instance) are essentially free. They don’t have any overhead as they’re computed only once.

An important concept in the Meta API: immutability. This helps prevents lots of bugs. If you return an immutable result, you can be sure it cannot be changed (of course). An advantage is that they’re quick. You can also use itertools.chain() to avoid allocating a new list. You can make a copy of everything as a list, of course.

Fun fact: it seems that the Meta API and its optimizations give django a 10% performance boost.

He showed some additional ideas for future improvements. He’ll discuss them tomorow at the sprint.