Practical project automation¶

Tags: python, django, pygrunn, zestreleaser

I got a request for repeating an earlier talk about practical project automation at the PyGrunn conference. A conference in Groningen in the north of the Netherlands about python and related subjects. Sounded great, so I set out to update the talk. The basic structure could remain intact, as I’m still practicing what I preached at that older talk even though I’m at a new job right now. So it is really valid :-)

Projects are more than just a collection of code. Projects are important. Projects are only finished when they’re deployed. On the server so that the website is visible to the customer, for instance. Or installed on the customer’s laptop. So one way or another you’ll have to grab together all the pieces and install it somewhere. But you also have to do that yourself on your development laptop, right? And your colleagues, too. How many pages of instructions do you want to give to your colleagues? You’d better be automating this! And… automation helps heaps regarding quality.

The cobbler’s children go barefoot. We write software for our clients, but our own processes are non-automated manual and menial steps. Menial is bad, as it is boring. So you cut corners. And you make mistakes. So automate away those boring bits and gain more time for the fun stuff!

The first and most essential step is build automation. Building your software. Collecting all the dependencies and installing them together with your project. Preparing everything, generating documentation, running tests. Whether it is ./configure;make;make install, ant/maven or python’s buildout. You need basically a single command to get your project all set up. That’s the only way to make sure you and your colleagues can reliably and comfortably get started on a project. And the only way you can rely on a proper installation on the webserver or your customer’s laptop.

Those build tools are great points for hooking in more automation. All build tools are extensible. Buildout, for instance, has so-called “recipes” to extend it.

For instance, with this buildout snippet …:

[apacheconf]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/mysite.apache.conf.in
output = ${buildout:directory}/etc/mysite.apache.conf

… the following example apache config file snippet …:

<VirtualHost *:80>
  ServerName mysite.example.org
  CustomLog ${buildout:directory}/var/log/access.log combined
  ErrorLog ${buildout:directory}/var/log/error.log
  DocumentRoot ${buildout:directory}/var/www
  ...
  RewriteRule ^(.*) http://localhost:${django:gunicorn_port}$1 [P]

… is automatically and reliably converted to:

<VirtualHost *:80>
  ServerName mysite.example.org
  CustomLog /srv/mysite/var/log/access.log combined
  ErrorLog /srv/mysite/var/log/error.log
  DocumentRoot /srv/mysite/var/www
  ...
  RewriteRule ^(.*) http://localhost:10003$1 [P]

So the filesystem paths and port numbers are all stored in just one place and reliably inserted into the actual apache configuration. Oh wait, we do need to make sure that local apache logfile directory exists, otherwise apache won’t start. But, don’t worry. We’ll make sure buildout just generates that directory for us automatically:

[mkdir]
recipe = z3c.recipe.mkdir
paths =
    ${buildout:directory}/generated
    ${buildout:directory}/var/geotiffs
    ${buildout:directory}/var/log
    ${buildout:directory}/var/media
    ${buildout:directory}/var/sqlite
    ${buildout:directory}/var/static

When we can set up our project in such a way, running tests automatically also is possible. Just grab a copy of the jenkins continuous integration server and play with it. Set it up to run the available tests on your software every time you change something in your version control system’s repository. This prevents errors like hardcoded local filename paths and files that you forgot to commit.

And, again, extend. Your jenkins can also be extended. It runs tests just fine, but it can also do more with a couple of plugins. Coverage testing! Look up coverage and hook it up in jenkins. You’ll get a nice report on the amount of lines in your codebase that’s covered by tests. A great quality measure!

Likewise, I’d recommend hooking up pep8 and pyflakes into jenkins for even more code quality monitoring. But they’re also very useful stand-alone: automated pep8 code style checking and pyflakes’ excellent detection of missing/extraneous imports and unused/missing variables. And if you’ve got javascript: make sure your javascript code is always automatically checked with jslint or jshint. It helps a lot with preventing errors in internet explorer.

An important extra mind set to keep in mind regarding automation: make it easy to do the right thing. Human beings follow the path of least resistance. If it is easier to cut corners, corners will be cut. If it easier to call some script that does the right thing, the right thing will be done.

For instance setting up a project. Do you take the time to add a proper documentation generation setup? Do you add test setup code even though you don’t have tests yet? Do you add tests later if there’s no test setup yet?

So generate a new project automatically with all these goodies already in place. See the movie below for an example. Use so-called “paster skeletons”. See also one of my blog entries about gathering and distributing organizational knowledge with those kind of skeletons.

Paster skeleton demo (with nensskel):

Releasing a package is also something where corners are cut. Do you make proper tags all the time? Do you add a new header in the changelog for the new version? Do you even update the version number? And both in the changelog and the setup.py? Automate those boring bits away and do the right thing, all the time. Use zest.releaser.

zest.releaser demo:

So, in summary:

• Make it easy to do the right thing.

• Automate your builds. This is the number one step you really really really need.

• Automate your tests and get a test server like jenkins to run them automatically.

• Extend your build tool and your test server. Those are handy points to hook in extra automation steps: don’t let that opportunity pass!

Here’s the presentation (minus the two movies shown above):