Practical project automation (talk at Dutch python user group)¶

Project + repeatable + automatable¶

A project is a bunch of code and a bunch of configuration and probably some scripts. Which you need to collect from multiple places, initialize, convert, perhaps compile and what have you.

Since the beginning of humanity as we know it there are tools for this. You need a build tool. Makefiles, scons, a.a.p. and buildout. Just use it. My advice: buildout.

Buildout allows you to write your own extensions (called “recipes”). Well, write those extensions when you have custom tasks. They’re not hard. Examples from The Health Agency:

• z3c.recipe.usercrontab for adding entries to a user’s crontab (more info).

• tha.recipe.logcheck for setting up a custom directory structure with a custom logcheck configuration. Yes, you can do it globally, but that makes it hard to steer it from your build tool. And more manual unless you’re building debian packages or so.

Quality control: buildbot + tests¶

Of course you have tests. And if you don’t, make sure you can run a python setup.py test or bin/test that tests a big part of your software.

You can do one extra step of automation that really hits the jackpot: a buildbot. Buildbot detects changes to your (svn) repository, makes a clean checkout, builds your project (here’s where the build tool comes in!), runs the tests and reports on the success. That is Quality Control with Capital Letters. (More info). No more of the following:

• Oops, I forgot to check in that file.

• Oops, I have some local dependency that the server doesn’t have.

• Oops, I forgot to run the tests before checking in.

Example: at The Health Agency, an essential part of our work is health documentation which is stored in amongst others a docbook xml variant. So apart from checking all our code, we set up a buildbot for all content checkouts (link checking, xml validity checking, etc.). The feel of reliability and solidity you get is amazing.

Quality: make it easy to do the good thing¶

A big part of quality is in the regular small things you do. Often there’s a sloppy way and a good way to do it. Make the good way the easiest. Laziness can work for you! Use scripts and skeletons.

• Copying some files is fine with a shell script, but once it gets bigger: just use a python script. os.walk, os.path, os.path.exists, shutil.rmtree. This prevents you from relying on your local shell script guru; instead you can do your beloved python thingy.

• Automate away boring multi-step tasks like releasing software. Like what I did with zest.releaser. Update the version, note the release date in the CHANGES.txt file, commit those changes, make a tag, upload to pypi if applicable, switch the version and CHANGES.txt back to development. Yeah, right. You’ll forget steps from time to time and it takes 5 minutes per package. With bin/fullrelease it takes half a minute without errors. Guess what gets used? Right.

• Or automating away a multi-step process to generate code coverage reports into one single bin/coverage call (provided you use buildout, with tha.coverage). Just make it easy to do. The call even opens your webbrowser with the coverage report. Lazy for the win! (More info).

• Skeletons! Useful! See for instance ZopeSkel that provides ready-made templates for plone buildouts, buildout recipes, etc. Paster itself has templates for plain python packages. (Note that Paster’s documentation isn’t clear that it can actually create skeletons as it mostly talks about the wsgi server part…). Just take a look at zopeskel and work from there. Recommendation: make your own in-company skeletons. You’re bound to have internal code guidelines or preferences. With a skeleton you generate just the right initial directory structure with just the right starting-point files for your new website or python library. No more missing README.txt. No more conveniently-forgotten test setup code. Etc.

Closing comments¶

Automation is nice work. It ensures quality. Get into the habit. Do it also for your own private tasks. It pays off in spades.

My slides are at http://www.slideshare.net/rvanrees/practical-project-automation