Django under the hood: files in Django - James Aylett¶

Tags: django, djangocon

(One of my summaries of a talk at the 2015 django under the hood conference).

James Aylett talks about files in django. You’ve got Files in python. Django build its own abstraction on top of it. File, ImageFile. Separate ones for use in tests. UploadedFile (“behaves something like a file object”, it mentions in the documentation…). There are temporary file and memory variants. Custom upload handlers. forms.FileField. It might not be perfect, but it works.

Files in the ORM: what gets stored in the database is a path to a file, the file is stored on the filesystem. If you store an ImageField, you can query the width and the height of the image. You’re better off storing the width and height in the database, though, as otherwise the image has to be read from disk on every request.

“Files are stored on the filesystem”? They are stored in settings.MEDIA_ROOT by default. Storing is done by a storage backend. You can replace it to get different behaviour. You can use a different storage backend by configuring it in the settings file. Or you can override it on a field-by-field basis.

Different storage backends? You can store data in amazon S3, for instance.

If you have a reusable app that works with files, please test on both windows and linux. And test with someting remote like working with S3.

Static/media files. Originally, you only had “media” files. Since django 1.3, you also have static files: non-user-uploaded files such as your apps’ javascript/css. Splitting “media files” and “static files” is a good thing.

There’s a complication: CachedStaticfilesStorage (or ManifestStaticfilesStorage in django 1.7). It adds hashes of the files to the filename to allow them to be cached for ever. Great system. Best practice. But it depends on everything using {% staticfiles %} in a very neat way. Otherwise you have cached-forever files that you want to change anyway…

Asset pipelines. Not many people write their css and javascript in one single file. You split it over multiple files. Or you compile coffeescript to javascript. Or you use a program (webpack or browserify for instance) to combine various files in one big one. This is graet for minification and caching. You do probably need “source maps” to help your browser debug tools refer back to the original files.

(Note by Reinout: read https://github.com/faassen/bundle_example for a nice explanation!)

Now… how do you get this into your django template? Either your combiner has to read your html code and write it back again. Or you write custom code to do things like {% asset 'my-js.js' %}.

For an example of an alternative you could look at rails/sprocket. Sprocket manages the entire pipeline and can touch every file and manage everything.

In the node.js world, it is common for the web code not to touch the pipeline. They’re separate. Webpack is an interesting one. Also “gulp” which defines the pipeline in a program. This means it can be customized a lot.

For django, it is good to be compatible with what node.js is doing.

What we’d ideally need:

• Use a pipeline external to django.

• Hashes computed by staticfiles.

• Sourcemap support.

If you want to use webpack, you could look at wabpack-bundle-tracker and django-webpack-loader. The pipeline is run by webpack and it emits a mapping file. There is a template tag to resolve the bundle name to a URL relative to STATIC_ROOT.

Tip: many people know http://caniuse.com/. There’s also http://doiuse.com/, which looks at it the other way around. It looks at your website and figures out which thing you’re using that lead to problems in browsers you care about.

Image: cat cleaning itself on the valve of a water tower, picture of my in-progress ‘Eifelburgenbahn’ layout.