Practical project automation (talk at Dutch python user group)

Tags: pun, python, buildout, thehealthagency, zestreleaser

I gave a talk at the Python User group in the Netherlands (PUN) with the goal of getting everyone to automate their projects more:

  • Practical tips and pointers you can follow up on. Also practical as I illustrate it with examples of an automation spree we had at The Health Agency in the last months.

  • Project: well, your projects are important, aren’t they? Customers pay you for well-working software or websites. Better make sure it is reliable!

  • Automation: don’t repeat yourself. Don’t do the same thing over and over again manually. The computer will flawlessly do the same task over and over again without making errors; you yourself will make stupid or grave mistakes.

Project + repeatable + automatable

A project is a bunch of code and a bunch of configuration and probably some scripts. Which you need to collect from multiple places, initialize, convert, perhaps compile and what have you.

Since the beginning of humanity as we know it there are tools for this. You need a build tool. Makefiles, scons, a.a.p. and buildout. Just use it. My advice: buildout.

Buildout allows you to write your own extensions (called “recipes”). Well, write those extensions when you have custom tasks. They’re not hard. Examples from The Health Agency:

  • z3c.recipe.usercrontab for adding entries to a user’s crontab (more info).

  • tha.recipe.logcheck for setting up a custom directory structure with a custom logcheck configuration. Yes, you can do it globally, but that makes it hard to steer it from your build tool. And more manual unless you’re building debian packages or so.

Quality control: buildbot + tests

Of course you have tests. And if you don’t, make sure you can run a python test or bin/test that tests a big part of your software.

You can do one extra step of automation that really hits the jackpot: a buildbot. Buildbot detects changes to your (svn) repository, makes a clean checkout, builds your project (here’s where the build tool comes in!), runs the tests and reports on the success. That is Quality Control with Capital Letters. (More info). No more of the following:

  • Oops, I forgot to check in that file.

  • Oops, I have some local dependency that the server doesn’t have.

  • Oops, I forgot to run the tests before checking in.

Example: at The Health Agency, an essential part of our work is health documentation which is stored in amongst others a docbook xml variant. So apart from checking all our code, we set up a buildbot for all content checkouts (link checking, xml validity checking, etc.). The feel of reliability and solidity you get is amazing.

Quality: make it easy to do the good thing

A big part of quality is in the regular small things you do. Often there’s a sloppy way and a good way to do it. Make the good way the easiest. Laziness can work for you! Use scripts and skeletons.

  • Copying some files is fine with a shell script, but once it gets bigger: just use a python script. os.walk, os.path, os.path.exists, shutil.rmtree. This prevents you from relying on your local shell script guru; instead you can do your beloved python thingy.

  • Automate away boring multi-step tasks like releasing software. Like what I did with zest.releaser. Update the version, note the release date in the CHANGES.txt file, commit those changes, make a tag, upload to pypi if applicable, switch the version and CHANGES.txt back to development. Yeah, right. You’ll forget steps from time to time and it takes 5 minutes per package. With bin/fullrelease it takes half a minute without errors. Guess what gets used? Right.

  • Or automating away a multi-step process to generate code coverage reports into one single bin/coverage call (provided you use buildout, with tha.coverage). Just make it easy to do. The call even opens your webbrowser with the coverage report. Lazy for the win! (More info).

  • Skeletons! Useful! See for instance ZopeSkel that provides ready-made templates for plone buildouts, buildout recipes, etc. Paster itself has templates for plain python packages. (Note that Paster’s documentation isn’t clear that it can actually create skeletons as it mostly talks about the wsgi server part…). Just take a look at zopeskel and work from there. Recommendation: make your own in-company skeletons. You’re bound to have internal code guidelines or preferences. With a skeleton you generate just the right initial directory structure with just the right starting-point files for your new website or python library. No more missing README.txt. No more conveniently-forgotten test setup code. Etc.

Closing comments

Automation is nice work. It ensures quality. Get into the habit. Do it also for your own private tasks. It pays off in spades.

My slides are at logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):