Djangorecipe: easy test coverage reports

Tags: django, buildout

Code coverage reports help you see which parts of your code are still untested. Yes, it doesn’t say anything about the quality of your tests, but at the least it tells you which parts of your code have absolute the worst kind of tests: those that are absent :-)

Ned Batchelder’s coverage.py is the number one tool in python.

Note: I use buildout instead of pip to set up my projects. Reason: I can integrate extra automation that way. One of those extra automation steps is “djangorecipe”: a recipe is a buildout plugin. It installs django, amongst others.

My previous setup

I use nose as a test runner. Easier test discovery was the reason at the time. Python’s unittest2 is better at that, so it might be time to re-visit my choice. Especially as I just read that nose isn’t really being maintained anymore.

A nice thing about nose is that you can have plugins. coverage.py is a standard plugin. And with django-nose you can easily use nose as a test runner instead of the standard django one. So once I put a few nose-related settings in my setup.cfg, coverage was run automatically every time I ran my tests. Including a quick coverage report.

The setup.cfg:

[nosetests]
cover-package = my_program
with-coverage = 1
cover-erase = 1
cover-html = 1
cover-html-dir = htmlcov

Running it:

$ bin/test
......................................
Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
djangorecipe                   0      0   100%
djangorecipe.binscripts       42     16    62%   25, 37-57
djangorecipe.boilerplate       1      0   100%
djangorecipe.recipe          115      0   100%
--------------------------------------------------------
TOTAL                        158     16    90%
----------------------------------------------------------------------
Ran 38 tests in 1.308s

OK

An important point here is that laziness is key. Just running bin/test (or an equivalent command) should run your tests and print out a quick coverage summary.

Note: bin/test is the normal test script if you set up a project with buildout, but it is effectively the same as python manage.py test.

New situation

“New” is relative here. Starting in django 1.7, you cannot use a custom test runner (like django-nose) anymore to automatically run your tests with coverage enabled. The new app initialization mechanism already loads your models.py, for instance, before the test runner gets called. So your models.py shows up as largely untested.

There’s a bug report for this for django-nose.

There are basically two solutions. Either you run coverage separately:

$ coverage run python manage.py test
$ coverage report
$ coverage html_report

Ok. Right. You can guess what the prototypical programmer will do instead:

$ python manage.py test

That’s easier. But you don’t get coverage data.

The second alternative is to use the coverage API and modify your manage.py as shown in one of the answers. This is what I now build into buildout’s djangorecipe (version 2.2.1).

bin/test now starts coverage recording before django gets called. It also prints out a report and export xml results (for recording test results in Jenkins, for instance) and html results. The only thing you need to do is to add coverage = true to your buildout config.

The details plus setup examples are in the 2.2.1 documentation.

(An advantage: I can now more easily move from nose to another test runner, if needed).

Not using buildout?

Most people use pip. Those using buildout for django will, I think, really appreciate this new functionality.

Using pip? Well, the basic idea still stands. You can use the call-coverage-API-in-manage.py approach just fine in your manage.py manually. Make it easy for yourself and your colleagues to get automatic coverage summaries!

 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):