Eric Holscher: Getting the most out of your test suite¶

Tags: django, djangocon

How to run tests? It differs per application. “setup.py test”, “nosetests”, “run_tests.py”, “manage.py test myapp”… So we need a standard way to run python tests. The answer is “setup.py test”. Problem: it is not ready yet. It is part of setuptools/distribute (not in distutils), it will be part of distutils2. When that lands, things will be a bit earlier. Django has no support for “setup.py test” as it needs a settings file and so.

Document how to run your tests, preferably in a shell script or so. Make it clear how to run the tests.

Run your tests after every commit, for instance in a continuous integration tool (Pony build (a simple one), hudson, buildbot, devmason). A continuous integration tool needs some script to set up the environment and run the tests. Use virtualenv and install your package through your setup.py so that also your setup.py is tested.

Hudson is the best continuous integration tool. Great plugin community. It is becoming the de-facto tool. Worth wile if you have the machine resources. See Honza’s earlier talk which covers hudson, too. There’s a doesn’t-run-perfectly-yet one at http://hudson.djangoproject.com.

Handy plugins: cobertura, violations, irc. Get coverage report by running coverage your_test_command. The code coverage needs an xml report to work from. django-test-extensions helps set it up for you. django-nose also makes it easy.

Django Kong is a functional testing tool (a Django application, so it stores historical data in the database) that can check if your live website is still running by checking certain pages. It is amazing how many problems get caught this way. It is useful for finding the crazy mistakes. And it records the time every request takes, so you can get historical data on the speed of your site.

Tip: get a dashboard. (They use devmason). Display test results. Display CPU server usage statistics. You get an overview in one screen. (Personal note: wow, that all looked very neat. Also the UI and the interface workflow. Small “sparklines” here and there to give a quick historical overview. Got to make sure I find some screenshots to show internally in my company as we’re also working on a dashboard for some completely different use case.)

Devmason can serve as a reporting platform. It is open source.

Summary:

• Make sure one single command will run your tests.

• Get a continuous integration setup.

• What you don’t measure, you don’t improve. So measure to get yourself to improve what needs improving.

• Get some basic monitoring going for your sites.