How I structure plone projectsΒΆ

Tags: instancemanager, plone

There is a rule number one: everything must be stored in svn. If you have pieces of configuration outside of svn, you're in for pain. I've had to work my way through Products/ directories trying to figure out which versions were in there. version.txt helps a bit, but once it is a version of which you cannot find a .tgz, you're done for. Or, if it is an svn checkout: is it safe to update it or does it need to be a specific revision?

Donjon (main tower structure)

Which pieces of configuration should you include? Well, which products are installed. Which plone version. The apache config file(s). Perhaps an awstats config. A cronjob for /etc/cron.d. Those config files are simple: just store them somewhere in your svn, I'll get to the location later. For the list of products and the plone version you've got basically three choices:

  • A "readme" file that lists your plone version and add-on products. You'd better keep those up to date!
  • Put the zipfiles in some svn directory of your choice. Your svn repository won't really like that, neither the backup of your svn repository.
  • Put the list of products (and download locations) in some script that downloads and installs them.

I use the last method, by using instancemanager. Basically a config file like this (URLs shortened to keep 'em on one line):

archive_sources = [
    {'source': 'qPloneGoogleSitemaps-0.3.0.tar.gz',
     'url': 'http://surf..../qPloneGoogleSitemaps-0.3.0.tar.gz',
    {'source': 'RedirectionTool-1.1.tar.gz',
     'url': 'http://plon.....s/1.1/RedirectionTool-1.1.tar.gz',

Instancemanager will reliably download those files and put them in your Products directory. Works for bundles, too. A good custom shell script or stuff like buildout can also help here. Nobody has to wonder which products and which versions to use: just look at (or use...) the config file.

For a small project with just one custom product, I normally add an instancemanager directory inside that custom product. Which, of course, is in your svn repository. In there, I put the instancemanager config with the specification of the project (other add-ons, plone version, zope version). Perhaps another directory for some documentation and an apache config file.

For a big project, you'll probably end up with a couple of specific svn checkouts of add-on products and a set of custom products. Svn bundles to the rescue! I normally make sure I only have to grab one svn checkout to set me up completely. Inside that checkout I've got the following directories:

An svn bundle of the external add-on products for which I need a specific checkout instead of a regular zip file. Perhaps a branch on which I'm doing fixes. Or a specific revision. The exact configuration is again explicitly in svn in this way!
An svn bundle of the add-ons I'm developing myself. Project-specific stuff. It somehow is handy to differentiate between "products" and "externalproducts" as you've got to take greater care with changes in products that other people are using than with changes in your own project-specific products. Put your DIYplonestyle theme in here, for instance.
Here's where I put the configuration for bigger projects. I normally subdivide it further into subdirectories
Instancemanager configuration files. Often a for regular development and a specific with the current stable production server's configuration.
The apache configuration files.
Some cronjobs you want to put in to /etc/cron.d
Documentation for the whole project. Explanation of how to set up the server. Backup/restore documentation.

This way of setting up projects slowly developed during a few years of setting up projects like this. We also use it at zest software for the majority of our projects. One thing that's not fully crystallized yet is how to deal with production bundles and development bundles. The needs for this tend to differ wildly per project. Some only do an update once every two months, others try out a quick change in preview and put it into production right away as they need a more rapid cycle.

I hope this is of help to some people. It would be interesting to hear of other setups or recipes. If you don't have a blog yourself, feel free to email me.

(Old imported comments)
"should work just great" by Reinout van Rees on 2007-05-31 11:31:38
Sounds like a well-working method. I assume that you can use it both ways: from production to development, from development (where you changed the products a bit) to testing, from testing to development, etc.?

In any case, automating it is the key!
"automation" by Robrecht van valkenburg on 2007-05-31 10:19:10
That is why we have created a automated build script that does exactly that. This will download everything that is listed within it's functions (it's simple bash scripting realy). The script can also be used to install eggs into your zope site and things like that.

This will recreate the enviroment exactly as installed on a production server or test machine and because it's fully automated it's faster then doing it manualy and you will have the exact same versions.

"works when you work in one location" by reinout on 2007-05-31 10:10:20
Your solution could work if you work in one location: on your production server. If you need to build up a local site on a development machine, you'd need to download all those product again. (Or zip up your server's lib dir). Hard to keep it synchronised. My main idea was to automate this :-)

If you've got a lib/quills-1.5 in there, do you know if that's the official .tgz release or a copied-over svn checkout or some modified version. With some conventions for this, it won't be a real problem, of course.

"symlinking :-)" by reinout on 2007-05-31 10:01:10
The solution is simple: I add a symlink from ~/.instancemanager to the ~/svn/myproject/integration/instancemanager/ file.
"Versions" by Robrecht van Valkenburg on 2007-05-24 10:13:38
Why use a strict versioning file (version.txt). It's easier to place the "real" addons (including Plone from a Zope perspective) in $INSTANCE/lib with full version number e.g.

and just make a symlink to the active version. The advantage of this is you can see wich version you use with a simple ls and you can very easily rollback a botched upgrade. Diskspace usage will be minimal and as soon as you know it's stable you can remove the old source in $INSTANCE/lib

Of course you should keep every version of the source in svn (or similar)

Robrecht van Valkenburg
"How do you tell instancemanager where the configuration file is?" by Erik Forsberg on 2007-05-22 20:58:27
Just downloaded instancemanager 1.0RC. Certainly looks interesting, but I can't find out how to do like you like to do - put the project-specific instancemanager configuration file in its own directory under "integration/instancemanager". I'd be happy if you could tell me!
"Useful stuff, thanks for sharing" by Tom Lazar on 2007-05-17 13:05:24
i think it's important that we share more of this 'show-and-tell' kind of stuff. It gives beginners something to orientate themselves and the more seasoned developers inspiration for their own setup.

sharing nifty hacks and coding practices is all very nice, but it's really equally important to have a solid setup in which to perform all those nice hacks.

personally i just work with the combination of svn bundles and hand-written shell-scripts for setup: 10% sophistication with 90% results... ;-)
"My Plone projects" by erral on 2007-05-16 21:56:42
Hi Reinout:

I follow your blog regularly, and I'm very interested in knowing how the hell people manages to have their Plone, Zope and Python installations ordered, and your blog post has been really helpful.

At our company, we have no big Plone background (our first serious Plone development was done last year, well "development", it was mostly a templating thing with some Archetypes based objects) so we are a bit "old school" Zope guys, with many Plones in our development Zopes, so just reading how people organizes themselves is great.

Since I read about instance-manager for the first time, I've always had interest on getting it working and understanding it, but I haven't had enough time, and I'm still with my plain Zope instance and myriads of products at $INSTANCE_HOME/Products, but now that it has started to grow, it's not maintainable at all.

Now that I have at least one reference of how you organize your Plone instances, perhaps I can start thinking to do something similar. Keeping a big part of this information in instance-manager's configuration file checked-in in an svn repository, helps to reproduce it if you have to delete your instances, I have always been reluctant to have dozens of Zope instances in my $HOME, because of the nightmare I thought it was, but I am sure that this is the best strategy (one Plone - one instance), specially with the Zope 3 based new Plone programming patterns, with ZCML files that change the configuration of an entire Zope instance instead just a Plone instance like installing Products from quickinstaller do.

Instance-manager, buildout, workingenv, I have tested all of them, but haven't used seriously in any deployment, so I haven't decided which one to use, but I have a clear idea that I must start using one (or more) of them.

Thanks again for your blog post !

Mikel Larreategi
CodeSyntax - logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):