Reinout van Rees’ weblog

Buildout and Django: djangorecipe updated for gunicorn support

2015-06-16

Tags: django

Most people in the Django world probably use pip to install everything. I (and the company were I work, Nelen & Schuurmans) use buildout instead. If there are any other buildout users left outside of zope/plone, I’d love to hear it :-)

First the news about the new update, after that I’ll add a quick note about what’s good about buildout, ok?

Djangorecipe 2.1.1 is out. The two main improvements:

  • Lots of old unused functionality has been removed. Project generation, for instance. Django’s own startproject is good enough right now. And you can also look at cookiecutter. Options like projectegg and wsgilog are gone as they’re not needed anymore.

  • The latest gunicorn releases didn’t come with django support anymore. You used to have a bin/django run_gunicorn (or python manage.py run_gunicorn) management command, but now you just have to run bin/gunicorn yourproject.wsgi. And pass along an environment variable that points at your django settings.

    With the latest djangorecipe, you can add a scripts-with-settings = gunicorn option and it’ll create a bin/gunicorn-with-settings script for you that sets the environment variable automatically. Handy!

Advantage of buildout. To me, the advantage of buildout is threefold:

  • Buildout is more fool-proof. With pip/virtualenv you should remember to activate the virtualenv. With buildout, the scripts themselves make sure the correct sys.path is set.

    With pip install something you shouldn’t forget the -r requirements.txt option. With buildout, the requirement restrictions (“versions”) are applied automatically.

    With pip, you need to set the django settings environment variable in production and staging. With buildout, it is just bin/django like in development: it includes the correct reference to the correct settings file automatically.

    There just isn’t anything you can forget!

  • Buildout is extensible. You can extend it with “recipes”. Like a django recipe that helps with the settings and so. Or a template recipe that generates an ngnix config based on a template with the django port and hostname already filled in from the buildout config file. Or a sysegg recipe that selectively injects system packages (=hard to compile things like numpy, scipy, netcdf4).

  • Buildout “composes” your entire site, as far as possible. Pip “just” grabs your python packages. Buildout can also build NPM and run grunt to grab your javascript and can automatically run bin/django collectstatic -y when you install it in production. And generate an nginx/apache file based on your config’s gunicorn port. And generate a supervisord config with the correct gunicorn call with the same port number.

Of course there are drawbacks:

  • The documentation is definitively not up to the standards of django itself. Actually, I don’t really want to point at the effectively unmaintained main documentation site at http://www.buildout.org/.... You need some experience with buildout to be able to get and keep it working.
  • Most people use pip.

Why do I still use it?

  • The level of automation you can get with buildout (“composability”) is great.
  • It is fool-proof. One bin/buildout and everything is set up correctly. Do you trust every colleague (including yourself) to remember 5 different commands to set up a full environment?
  • If you don’t use buildout, you have to use pip and virtualenv. And a makefile or something like that to collect all the various parts. Or you need ansible even to set up a local environment.
  • Syseggrecipe makes it easy to include system packages like numpy, scipy, mapnik and so on. Most pip-using web developers only need a handful of pure python packages. We’re deep into GIS and numpy/gdal territory. You don’t want to compile all that stuff by hand. You don’t want to have to keep track of all the development header file packages!

So... hurray for buildout and for the updated djangorecipe functionality! If you still use it, please give me some feedback at reinout@vanrees.org or in the comments below. I’ve removed quite some old functionality and I might have broken some usecases. And buildout/django ideas and thoughts are always welcome.

Formatting python log messages

2015-06-05

Tags: python

I see three conflicting styles of log formatting in most of the code I come across. Basically:

import logging

logging = logging.getLogger(__name__)

logger.info("%s went %s wrong", 42, 'very')
logger.info("{} went {} wrong".format(42, 'very'))
logger.info("%s went %s wrong" % (42, 'very'))

I looked at the official PEP 282 and at the official docs.

With the help of someone’s stackoverflow question I think I understand it now.

  • Use the first version of the three examples. So:

    • The actual log message with the old, well-known %s (and %d, %f, etc) string formatting indicators.
    • As many extra arguments as you have %s-thingies in your string.
  • Don’t use the second and third example, as both of them format the string before it gets passed to the logger. So even if the log message doesn’t need to be actually logged somewhere, the full string gets created.

    The first example only gets passed a string and some extra arguments and only turns it into a real full string for in your logfile if it actually gets logged. So if you only display WARN level and higher, your DEBUG messages don’t need to be calculated.

  • There is no easy way to use the first example with {} instead of %s.

So: use the logger.info("%s went %s wrong", 42, 'very') form.

(Unless someone corrects me, of course)

Djangocon sprint: zest.releaser 5.0 runs on python 3

2015-06-05

Tags: djangocon, python

Good news! zest.releaser supports python 3.3+ now!

Now... what is zest.releaser? zest.releaser takes care of the boring release tasks for you. So: make easy, quick and neat releases of your Python packages. The boring bit?

  • Change the version number (1.2.dev0 -> 1.2 before releasing and 1.2 -> 1.3.dev0 after releasing).
  • Add a new heading in your changelog. Preferrably record the release date in the changelog, too.
  • svn/git/bzr/hg tag your project.
  • Perhaps upload it to pypi.

Zest.releaser takes care of this for you! Look at the docs on readthedocs.org for details. The short page on our assumptions about a releasable python project might be a good starting point.

Now on to the python 3.3 support. It was on our wish list for quite some time and now Richard Mitchell added it, kindly sponsored by isotoma! We got a big pull request last week and I thought it would be a good idea to try and merge it at the djangocon sprint. Reason? There are people there who can help me with python 3.

Pull request 101 was the important one. So if you want to get an idea of what needs to be done on a python package that was first released in 2008, look at that one :-)

Long-time zest.releaser users: read Maurits’ list of improvements in 4.0, released just two weeks ago.

Running on python 3 is enough of a change to warrant a 5.0 release, though. Apart from python 3, the biggest change is that we now test the long description (what ends up on your pypi page) with the same readme library that pypi itself uses. So no more unformatted restructured text on your pypi page just because there was a small error somewhere. Handy. Run longtest to check it. If it renders OK, it’ll be opened in your webbrowser, always handy when you’re working on your readme.

Djangocon: React.js workshop - Maik Hoepfel

2015-06-04

Tags: djangocon

He compares react.js to angular. Angular basically want to take over your whole page. Multiple apps on one page, mixed with some jquery, isn’t really possible. You also have to learn a lot of terminology. It is a full stack framework. And angular 2.0 won’t have a migration path from angular 1.x...

React.js only does one thing and does it well: the view. It doesn’t do data fetching, url routing and so.

React.js is quite pythonic. You can start adding little bits at a time to your code. You cannot really do anything wrong with it.

A core idea is that the html is in one corner. That’s the output. And the state is in the other corner. They’re strictly seperated, which makes it easier to reason about.

It is build by facebook. They take page load time seriously. You can even run react.js partially on the server side (“react native”, inside node.js), taking down rendering time, especially on mobile. Facebook is pretty deeply invested in it, so the chances of it disappearing are low. (As opposed to angular: google is barely using it itself).

When you start using react.js on a page, you’ll have to start thinking about components. An “unordered list of results”, a “search box”. Those kinds of components.

  • Component: basically a piece of html, handled by react. If react renders it, html comes out. It looks at its state and properties to render it (“it calls the .render() method”).

  • State: the current state of the component. The component can change this.

    If you can do without state: do it. You’re better off calculating it from something else. And if it makes sense to store the state on a higher-level component: do it.

    If a component’s state changes, you can assume that the render method gets called (it does some magic so that it only updates html if there are changes).

  • Properties: what you pass a component from the outside. Think parameters, like a list of products. The component cannot change these.

React.js code is quite simple to reason about as there’s a simple path through the code. If some method changes the state, react handles the rendering. The rendering might set up click handlers or other things. If such a click handler changes the state, react renders everything and so on and so on.

Integration with existing jquery plugins is generally possible, but you’d rather not do it. React.js tries to store state nicely in a corner and jquery plugins normally store state on the DOM itself. So watch out a bit. You might have to build some more custom html than you’d have to do otherwise as you’d just download one of the existing jquery plugins.

A note about facebook’s flux. “A flux” is just a structure for building large applications. There are multiple ones. He hopes you don’t have to build such a large application yet, as the flux you’re choosing might not be the one that wins out in the end.

If you need a good example of django and react.js, look at django-mediacat.

Question: “How do you integrate with django? Does django still render forms?” Answer: normally, you’d decouple the front-end and back-end, at least for the bits you let react.js handle. So perhaps generate some javascript variables in the django template. And grab the rest via an REST API.

Some clarification to the previous paragraph: you can choose yourself how much of the page you let react.js handle. He himself is using react.js to handle a couple of interactive elements on an otherwise-static django-rendered page.

Question: “Are you happy with react.js?”. Answer: “YES!”. It is quite small in scope. It doesn’t get in your way. Easy to get started with. You can’t do an awfully lot wrong (he has seen terrible examples with angular: it gives you much more rope to hang yourself with). You can progressively add interactivity to your page.

Tip: keep your react.js components small. Only let it handle spitting out html. Do any calculations you might have to do on the server side, behind your REST API.

Djangocon: Lightning talks

2015-06-03

Tags: django, djangocon

(One of the summaries of a talk at the 2015 Djangocon EU conference).

Pycon Namibia - Daniele Procida

In february 2015, the python conference in Africa actually took place! In Namibia! Daniele Procida talked about the plans at many conference (like at djangocon.eu 2014).

It was really good. They formed valuable relationships and learned a lot. Several of the attendees are here at the conference. They’re going to organize another conference later this year. So if you want to escape a dreadful European winter...

I’m new to django but why should I use it? - Ben Sharif

Why should you use it? You’re new, so why Django? He’s in that situation.

He’s a medical student. He’s not a doctor yet. It also means he isn’t a professional programmer. He wants to quickly prototype ideas for medical web apps. He wants something that lets him develop things fast: Django fits the bill.

He started learning python in may 2014. He tried the django tutorial, but that didn’t really work. He got pointed at web2py. Easy to get started. Almost no configuration. Through the web development.

His demo is at http://medboard.co.uk/

So... what is he missing? The django community is absolutely massive. So... tell him how he can get involved with Django with a bit more success!

Speech to text typists

(Note: The whole conference, there were screens on which a live transcription/subtitling of the talks was provided for those that needed it (non-native speakers of those with hearing problems). (So if you think that I’m typing fast by typing in all these summaries: I don’t stand a chance against those two ladies!)

They travel around the country to support deaf people. She had a top 3 of hard conferences. Now she has a top 4.

They work with machine shorthand. It works with a system of phonetics. It is a special keyboard. You use it like a piano with chords: you press multiple keys together to form words and parts of the words. The software matches the keys they’re pressing to a dictionary.

They can adjust the dictionary. That was why halfway the conference the term “wiskey” got turned into “WSGI”.

With a regular keyboard, you get a 120 word per minute if you’re really really good. With a keyboard like this, you can get to 250. That’s enough to keep up with most people (except Russell) :-)

(Note: They got a thunderous applause!)

Holiday picture from the Eifel region

DjangoBeer Florence

Two years ago he attended the djangocon in the Warsaw circus tent. Why not do something similar in Florence? They now rebooted the event and turned it into a monthly meeting about all things web.

They like (craft) beer, but they love to enjoy it responsibly. No drunkenness. The code of conduct is strictly enforced!

The even had beer tasting lessons.

http://djangobeer.com

Systers - Ana Balica

There are a lot of organizations that want to increase the number of women in IT. Djangogirls (lots of applause!), Pyladies (lots of applause!) and lots more.

Systers is a community for women involved in STEM areas (Science, Technology, Mathematics).

I wrote my first line of code 1.25 years ago - Geraint Palmer

He’s a PhD student. His first line of code was in VBA. He could do a lot with it: a for loop and a condition...

In October 2014 he started using python.

He learned enough of it to quickly start teaching python. That’s the best way to learn the language yourself. Recommended.

Google a lot. Once you don’t know the answer, google it.

Then he found a new nice text editor. It revolutionized his programming!

Afterwards he installed his first library.

Then pair programming! He learned structuring his code. He learned to use version control. This helped him a lot to grow up as a programmer.

He participated in Pycon Namibia and discovered there was a huge python community all around the world. Thank you!

Do I need to update my django project? - Mounier Messelmeni

He uses django since three years.

Why do updates matter?

  • Security and performance. Old unsupported releases don’t get security updates!
  • Bugfixes.
  • New features: every new version gets nice new features.

The good news: the current 1.8 version is a long term support version.

Python versions? If you still use python 2.6, you really should upgrade to 2.7

For every upgrade step, read the release notes and especially the added/removed section.

Make sure you have good backups of your database and your uploaded files.

It would be great if you could also look at using python 3. Look at https://caniusepython3.com/ or install caniusepython3 from pypi.

The missing method murder mistery

Great visual talk about tabs and spaces. Impossible to write down, watch the video when it comes out. Great.

Moral of the story: friends don’t let friends mix tabs and spaces.

Beter still: friends tell friends to use python 3 (that has better tab/space warnings).

Sprinting - Russell Keith-Magee

A sprint is a collection of people that turn up in a space to hack on code.

There is a sprint the next two days. What needs to be done? There are bugs. You could solve them, you could triage them, you could check patches, you can provide tests for patches or to prove bugs. You could help with the documentation.

Not being an english speaker isn’t a problem. Just get the first draft done, then other people can work on the spelling.

If you’re new to django: working on the tutorial would be great! Point out problem areas.

You can work on django core. On libraries. On programs that help django. On the django project website.

So: pick an area that interests you!

If you have questions: just ask a core developer! Or better still, ask the person next to you. The core team aren’t superheroes, we’re all in the same boat.

Djangocon Europe 2016

Djangocon Europe 2016 will be in... Budapest in Hungary!

Holiday picture from the Eifel region

Djangocon: Conformity and you - a question of style - Greg Chapple

2015-06-03

Tags: django, djangocon, python

(One of the summaries of a talk at the 2015 Djangocon EU conference).

Greg Chapple works half his time on Django projects, half on Rust projects.

Some things you’ll probably never hear: “Can’t wait to fix some pep8 violations today”. Style conventions probably aren’t the sort of thing you take into account when picking a language or a framework.

Every line of code follows some sort of style convention. Intentional or not. The style of our code can identify us. Same as with this talk: Greg speaks with a slight Dublin accent, by default.

Something to ask yourself: if you look at a piece of code inside your company, could you identify the person that wrote it?

Let’s get back to Greg’s accent and the way he speaks. How did he come by this? Where he grew up, of course. The people he’s around regularly. You can change it, though. He spend two weeks in the USA and when he came back he got funny looks for some of the US pronouncement that krept into his vocabulary.

The same with your code style! You’re influenced by the code you’re surrounded by. If you want to make changes, it takes quite some effort to do it. Learning to use list comprehension instead of for loops, for instance.

Your style should be consistent. Using single quotes in one place and double quotes on the next line, that looks bad. Spaces after a comma in some places (=pep8) or not.

If you have to switch from one project to another, you’re helped a lot if both projects use the same style. That’s where style guides come in. PEP8, jshint, flake8. Corporate guides. Look for support for those in your text editor to make it easy for you to use them. If you look at the download numbers for these text editor extensions, they’re very popular (and rightfully so).

Interesting fact: people take comments about their coding style much better if it comes from a tool as opposed to from a human being :-) Having a tool in place to tell you about code style violations is very valuable as it removes lots of possible unfriendly interactions.

A nice quote from pep8: One of Guido’s key insights is that code is read much more often than it is written. As PEP 20 says, “Readability counts”.

Greg looked at the django source code, hoping to find some nice glaring inconsistencies, but the django code is pretty good! :-) He did find some small issues and showed them as examples.

Sometimes you should allow some slight breakage to actually improve readability.

If someone else will be working on it, make sure it looks good.

If the code looks dirty and non-consistent, it doesn’t help to make you trust the actual work the code does. Doesn’t the lack of attention to how it looks mean that the actual code is not well-designed? Maybe, maybe not. But the lack of attention could easily have been there in some of the algorithms or the security declarations, right?

Your code speaks about you. You, as a professional developer. What does it tell about you?

Holiday picture from the Eifel region

Djangocon: Coding with knives (pt II) - Adrienne Lowe

2015-06-03

Tags: django, djangocon, python

(One of the summaries of a talk at the 2015 Djangocon EU conference).

Adrienne Lowe wants to share some lessons she learned while learning to code. And how to use knowledge learned in one area in another one.

Another area? Cooking! She cooked a lot. Working in restaurants, private cook, training, everything. She started a blog called coding with knives.

Three lessons she has for new coders:

  • If one tutorial doesn’t work for you, choose another one. Don’t let it reflect on your own opinion of yourself. If you’re a cook and a cookbook doesn’t work for you, you simply pick another! So simply look around for a different tutorial.

  • Reach out. Don’t be afraid to ask basic questions. The person answering also learns something: being a better teacher. So you both learn!

    In the cooking community, you exchange recipes. You help each other with tips. The same here in the django community. It is so friendly! And most of it is open source. Great!

  • Let your goal guide you. You’ll inevitable run into problems as a beginner. Don’t let that discourage you. Make sure you have a specific goal you’re working to and celebrate when you reach it. Simply accept that there’ll be some problems along the road. That’s fine as long as you reach the goal (“get a first django view working”).

How can we help new coders? Conferences like this can be daunting. Explicitly reaching out and getting newcomers to such a conference with grants helps a lot.

What helps even more: showing that we’re regular human people. Being open. We are all in this together. Many of us are insecure. And many of us have a tendency to see the best coders as heroes and ourselves as people down at the bottom of the rungs. But we’re all in this together. We are all regular human people.

Look at the keynote of Jacob Kaplan Moss (youtube). He, one of the django founders, described himself as a mediocre programmer. He wants to do away with the myth of the superhuman programmer who is 100x more productive as all the others. Programming, for the most part, is an accumulation of skill and knowledge. Just like cooking! In cooking, you have celebrity chefs that look like they have some inherent skill that ordinary people cannot reach. But... by doing it and learning it and exercising yourself and more doing it, you can become a pretty good cook.

What we should tell new programmers? You can. You do. You are. You can program. Make sure you do it. If you do it, you are a programmer.

(Note: using knowledge learned in one area in another one? There was a talk about exactly that two weeks ago at the pygrunn conference)

Holiday picture from the Eifel region

Djangocon: Demystifying mixins with Django - Ana Balica

2015-06-03

Tags: django, djangocon

(One of the summaries of a talk at the 2015 Djangocon EU conference).

Ana Balica talks about mixins. Mixins are ways to add extra behaviour into classes. Mixins aren’t anything special in python. They are just classes.

Mixins are nice for providing modularity. They should have one specific goal and they should not themselves be extended or instantiated.

You could use them for instance to work around copy-pasting code around between classes. Just make a class with the otherwise copy-pasted code.

How do you use it? Python’s multi-inheritance. A class can have multiple parents:

class Foo(SomeMixin, BaseFoo):
    pass

In django, mixins are especially useful for class based views. As an example, let’s build a mixin that checks if our users are logged in:

class LoginRequiredMixin(object):

    def dispatch(self, request, *args, **kwargs):
        if not request.user.is_authenticated():
            raise PermissionDenied
        return super(LoginRequiredMixin, self).dispatch(
            request, *args, **kwargs)


class AboutView(LoginRequiredMixin, TemplateView):
    template_name = 'about.html'

dispatch() is handy to keep in mind. Likewise .get_context_data(). (Note by Reinout: doing something with get_context_data is probably not needed).

How do you discover what’s in django’s class based views? You could look at the source code of django. But looking at http://ccbv.co.uk/ is handier.

There are a lot of mixins already written for you, ready for use in django-braces. Like the LoginRequiredMixin!

Go back to your views and start writing mixins to clean up the code!

Holiday picture from the Eifel region

Djangocon: What it’s really like building RESTful APIs with Django - Paul Hallett

2015-06-03

Tags: django, djangocon

(One of the summaries of a talk at the 2015 Djangocon EU conference).

Paul Hallett considers himself an “API fanatic”. He works for lyst, a website for buying clothes. They had an existing API, json-rpc, and wanted to have a better API that fitted better with http.

RPC (remote procedure call) means you only have one endpoint (=url) and everything is a POST request, even if you only request information.

“RPC for http is bad” is what he said. He wanted to do it with django rest framework. Django is a batteries-included webframework, django rest framework is a batteries-included API framework.

If you need to re-build some existing API as a rest API, think of it as a chance to rebuild. If it is an existing API, the program probably changed a bit from how it looked originally. Shouldn’t you clean it up a bit? Change a couple of models? The alternatively is to shoe-horn non-fitting models into an otherwise nice API...

REST focuses on Resources. In django terms, you often think “a model”. This isn’t necessarily true, it might be either more or less. A model might actually have two separate sets of attributes that you’d rather treat as two separate resources.

The core rest framework technology you need to use for this is serializers. Basically translators between django models and resources.

Permissions tip: set DEFAULT_PERMISSION_CLASSES to the admin-only one. That way, if you make a mistake, only the admin can access it. Otherwise you might leave too many API endpoints open.

Authentication: always have it. Always do something. Even if it is on the same server. Tokens, a simple string, whatever. But the best option is oauth2. Install the oauthtoolkit to set it up.

Don’t be afraid to bend the rules for http a little bit. If you can make a nice API with just a little bit of rule-bending: that’s fine.

Documentation is necessary. If you don’t have documentation, you have a bad API. You might have a great API and clear endpoints, but documentation is still key. At the bare minimum provide documentation on how to obtain an oauth2 token!

Holiday picture from the Eifel region

Djangocon: CMS Panel - Iacopo Spalletti and Tom Dyson

2015-06-03

Tags: django, djangocon

(One of the summaries of a talk at the 2015 Djangocon EU conference).

The panel consisted of Iacopo Spalletti of Django CMS and Tom Dyson of Wagtail.

For many people, a CMS is the way they get acquainted with django, so it is pretty important. Often they come from Wordpress originally.

  • What is the main philosophy of your CMS?

    Django CMS tries to be just another django app. After a while, django cms grew to support multiple roles. The content manager, the front-end designer, the main editors and the other editors. The main editors set up the content structure of the site and the other editors fill it.

    Wagtail doesn’t try to give editors as much power as django CMS (and Drupal and so) do. Often the super-powers aren’t even used. It is more aimed at django programmers. It has its own UI (instead of the standard django admin). Performance is a key priority.

  • In practice, how does the philosophy work out for you?

    Django CMS is build upon the flexibility of the django admin. They use it everywhere and it gives them a lot of power. The only pain point is the URLconf. They love django. They wouldn’t be anywhere without django.

    Wagtail also tries to make the most use of all the great tools that django gives them. A point of friction for first-time users is django’s installation. It just isn’t as easy as uploading a couple of PHP files.

  • Ultimately, every CMS has a UI. Does django need to change anything in the way it deals with the front-end?

    Wagtail is often used through its API. The front-ent is often a separate javascript application. No problems there.

    CMSs are often used to create traditional websites. Websites that need to be crawlable by search engine bots. So that often means a regular html website. And if you need it there are more than enough ways to get your data into a javascript front-end.

  • Do you have any advice for the other CMS?

    Django CMS’s lesson number one is “communicate correctly with the community”. This is especially necessary for an open source project. They had some grumbling about features that got removed from one version to another.

    Wagtail: they made a couple mistakes with the community because they didn’t respond quickly enough to pull requests at the start of the project. So watch out for that.

  • How far away are we from a tidal wave of new developers because we’ve pushed wordpress off its throne?

    Django CMS: deployment is the bottleneck. You could solve it at the service provider level, perhaps, because then you wouldn’t have to install it yourself.

    Wagtail: yes, agreed. A second point is themes. Wordpress has lots of themes. But I don’t know if that would be a good fit for us.

  • We don’t really have a commercialization story. You can’t really sell django themes at the moment, for instance.

    Wagtail: they’re mainly developing wagtail to make it easy for themselves to build websites quickly. They’re not sure themselves how it could be commercialized (in addition to making money with the websites).

Holiday picture from the Eifel region
 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):