Sisyphus, which they made at paylogic, is a semi-distributed job scheduling system.
Originally they used cronjobs. It is very robust and reliable. But it has limitations:
So… they wanted something new! Sisyphus. The features:
The system depends on a central SQL database, so that part is not redundant. But that central database is the most critical part of Paylogic, so if the main database goes down, jobs that don’t run is hardly a priority :-) So this dependency is fine.
Sisyphus is divided into separate parts, like scheduler, dispatcher, configurator, etc.
Nice detail: sisyphus is actually run as a cronjob on all the active hosts (polling for jobs). If a job is available, a “guardian” claimes a process using a database lock. The job is run in a subprocess. The guardian keeps track of the job and takes care of the bookkeeping.
The current state: they are using it in production at paylogic. There are still some rough edges that need attention, however. It is not open source yet, but it will be.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):