Practical project automation (talk at Dutch python user group)
I gave a talk at the Python User group in the Netherlands (PUN) with the goal of getting everyone to
automate their projects more:
- Practical tips and pointers you can follow up on. Also practical as I
illustrate it with examples of an automation spree we had at The Health
Agency in the last months.
- Project: well, your projects are important, aren’t they? Customers pay
you for well-working software or websites. Better make sure it is reliable!
- Automation: don’t repeat yourself. Don’t do the same thing over and
over again manually. The computer will flawlessly do the same task over and
over again without making errors; you yourself will make stupid or grave
Project + repeatable + automatable
A project is a bunch of code and a bunch of configuration and probably some
scripts. Which you need to collect from multiple places, initialize, convert,
perhaps compile and what have you.
Since the beginning of humanity as we know it there are tools for this. You
need a build tool. Makefiles, scons, a.a.p. and buildout. Just use it. My advice: buildout.
Buildout allows you to write your own extensions (called “recipes”). Well,
write those extensions when you have custom tasks. They’re not hard.
Examples from The Health Agency:
- z3c.recipe.usercrontab for adding entries to
a user’s crontab (more info).
- tha.recipe.logcheck for
setting up a custom directory structure with a custom logcheck
configuration. Yes, you can do it globally, but that makes it hard to steer
it from your build tool. And more manual unless you’re building debian
packages or so.
Quality control: buildbot + tests
Of course you have tests. And if you don’t, make sure you can run a python
setup.py test or bin/test that tests a big part of your software.
You can do one extra step of automation that really hits the jackpot: a
buildbot. Buildbot detects changes to your (svn)
repository, makes a clean checkout, builds your project (here’s where the
build tool comes in!), runs the tests and reports on the success. That is
Quality Control with Capital Letters. (More info). No more of
- Oops, I forgot to check in that file.
- Oops, I have some local dependency that the server doesn’t have.
- Oops, I forgot to run the tests before checking in.
Example: at The Health Agency, an essential part of our work is health
documentation which is stored in amongst others a docbook xml variant. So
apart from checking all our code, we set up a buildbot for all content
checkouts (link checking, xml validity checking, etc.). The feel of
reliability and solidity you get is amazing.
Quality: make it easy to do the good thing
A big part of quality is in the regular small things you do. Often there’s a
sloppy way and a good way to do it. Make the good way the easiest. Laziness
can work for you! Use scripts and skeletons.
- Copying some files is fine with a shell script, but once it gets bigger:
just use a python script. os.walk, os.path, os.path.exists, shutil.rmtree. This prevents you
from relying on your local shell script guru; instead you can do your
beloved python thingy.
- Automate away boring multi-step tasks like releasing software. Like what I
did with zest.releaser.
Update the version, note the release date in the CHANGES.txt file, commit
those changes, make a tag, upload to pypi if applicable, switch the version
and CHANGES.txt back to development. Yeah, right. You’ll forget steps from
time to time and it takes 5 minutes per package. With bin/fullrelease
it takes half a minute without errors. Guess what gets used? Right.
- Or automating away a multi-step process to generate code coverage reports
into one single bin/coverage call (provided you use buildout, with
tha.coverage). Just make it
easy to do. The call even opens your webbrowser with the coverage report.
Lazy for the win! (More info).
- Skeletons! Useful! See for instance ZopeSkel that provides ready-made templates
for plone buildouts, buildout recipes, etc. Paster itself has templates for
plain python packages. (Note that Paster’s documentation isn’t clear that
it can actually create skeletons as it mostly talks about the wsgi server
part...). Just take a look at zopeskel and work from there.
Recommendation: make your own in-company skeletons. You’re bound to
have internal code guidelines or preferences. With a skeleton you generate
just the right initial directory structure with just the right
starting-point files for your new website or python library. No more
missing README.txt. No more conveniently-forgotten test setup code. Etc.