![[Valid Atom]](http://www.atomenabled.org/feedvalidator/images/valid-atom.png "Validate my Atom feed"){kind=small}
This is an Atom formatted XML site feed. It is intended to be viewed in a Newsreader or syndicated to another site. Please visit Atom Enabled for more info.
This is an Atom formatted XML site feed. It is intended to be viewed in a Newsreader or syndicated to another site. Please visit Atom Enabled for more info.
Summary: yep, we (Nelen & Schuurmans) need a new Python/Django programmer! Location: Utrecht, the Netherlands, right in the center of town. The work subject? Geodjango and water management: that's the shortest summary.
Five months ago I wrote about an upcoming job opening. We got, if I remember correctly, nine reactions, eight of which mentioned "yeah, I saw on Reinout's blog/twitter...". The power of twitter and blogs wasn't demonstrated that vividly to my colleages yet :-) The end effect was that when we invited people to come round for a talk, they were introduced internally as "oh, some of Reinout's friends are coming"... That meant I had some explaining to do during lunch :-)
Business is booming and there's yet another big project coming our way. Help! We really need to crank up the output. At the moment we're three people doing the practical day-to-day Django coding with another one coming up in a week. Cooperative, productive, friendly. We could really use another pair of hands (and a brain, of course).
The timeframe? What I heard is a closing date of 23 September. But we might start inviting people over sooner as there really are a couple of projects lining up. So don't wait too long.
The official stuff? The official vacature is on our website. Yep, that's in Dutch. I'm afraid that speaking Dutch is handiest to get settled into the company. We might be open for an experiment, though :-)
You can always mail me if you've got further questions, of course.
There was one additional problem I saw with yesterday's windows installation: simplejson!
Totally unexpected. The good thing is that simplejson is shipped with python 2.6 and higher. The sane thing is that they named it "json" instead of "simplejson".
The bad or at least unexpected thing: I cannot find a windows installer for simplejson for python 2.6! Which means my "import simplejson" won't work anymore. And those have to be in as the software gets installed on python 2.5 sometimes.
For windows + python 2.6 compatibility I now have the following in at least 10 different files:
try:
import json # Python 2.6+
except ImportError:
import simplejson as json # Python 2.5-
And I've had to remove simplejson as a dependency. I don't yet know a good solution for that. I don't think you can have an optional dependency based on the python version with the current distribute/setuptools. Of course you could do some more if/else trickery in there as a setup.py` is just a python file after all.
Anyone got a good hint? Is there something I overlooked?
My (geo)django installation on the customer's windows 2008 box went way better than I expected. "Mostly smooth" is what I'd call it. My windows experience is a tad rusty after not using windows productively for about 13 years or so. And I ran into all sorts of problems when trying it out locally. So I expected much worse.
Trying everything out beforehand and assembling the right set of packages and killing off all DLL problems helps. See my earlier article about all the weird problems I encountered). It is hard work to find the cause for some obscure DLL problem, but it beats having to debug it when the customer is sitting next to you :-)
The actual installation went without a hitch. Everything installed just fine. I had written down what I had to do beforehand: that paid out in spades. I only saw two serious problems in the end:
In any case: the software installed just fine. And the oracle database connection works just fine. And the customer was happy! And I'm too :-)
One of our customers demanded a windows installation of his geodjango application. None of our active developers are well versed in that. My last personal well-maintained windows installation was windows 3.1. There.
So I spend two days getting it working on a temporary windows 2008 test machine and on a local virtualbox. Those were two days I'd rather been spending on some productive coding, to be honest. Finding problems and solving them have their own merit and somebody needs to so it so I girded myself and got down to work. I'll list some of the bad and good things below, including solutions.
I did not have to worry about the actual Django project we were shipping. We're using buildout to make our builds and deployments repeatable. On a regular ubunbu box, deploying a new site is a non-event: easy as pie and faultless, mostly. You know which "aptitude install" commands to run to get the external dependencies. And buildout grabs all the regular python packages you specified and prepares the Django instance, apache config and all the rest.
Buildout works the same on windows: it grabs all the stuff and configures everything I told it to configure. And as I created the project from our internal skeleton/code template, it came out of the box with that at-least-working apache config for windows I distilled from the last windows project. Happy.
Problem: no internet access on the server. Intranet. Well-protected. So I need to bring a CD with all the code. Easy to do with buildout if you use zc.sourcerelease. The documentation isn't clear, but once you installed zc.sourcerelease somewhere, you can call bin/buildout-source-release https://url/to/your/project/tags/1.7 buildout.cfg. So the url is your svn url. That works best in my experience. And the second parameter is the buildout config (in our case deploy.cfg as that's the production config). And give it a -n project-1.7 parameter to get a better name than 1.7.
The result is: project-1.7.tgz. Your complete buildout including all the eggs and downloads it would otherwise have to grab from the internet. Extract it and run python install.py (instead of the normal python bootstrap.py and bin/buildout calls). Ready. Wow.
Installing on windows starts easy. I looked at the geodjango install page and ask a colleague for the set of .exe and .zip files from his last windows deployment. Ok: python-2.6.4.exe and double-click. Same for apache. And matplotlib. And numpy. And an oracle connector.
And lxml: ouch, the latest version doesn't have a python2.6 installer! I found a slightly older one that did. Another problem solved.
Some stuff is different. Some oracle "instantclient" (a set of DLLs) must be placed in some directory that you then add to the system path. Same with mapnik.
And the apache mod_wsgi module is shipped as a .so file that you have to copy somewhere deep in apache's directory structure. Manually.
Ok, fine so far. Let's fire up apache. Boom. Ok, let's try django itself on the terminal. Boom. Uh oh...
When firing up django, it would complain that it couldn't load the oracle database connector. Some dll had a hickup. Dependencywalker is a great, simple tool that inspects a DLL for you and tells you which DLLs that it depends on are missing. I don't remember whether it was linkinfo.dll or msvcr90.dll or msvcr71.dll anymore. The last two are redistributable files from microsoft visual studio. You can actually grab a .exe installer for them. Only, which of them? The 2008 or the 2009 or 2010? It doesn't say which one includes which number...
I'm actually not sure anymore what solved this issue. I installed at least two of those redistributable file installers. And I swapped around between an oracle 10 and 11 connector and matching "instantclient" libraries. I think the 11 one worked in the end. I sure hope it goes well on the client's server!
Geodjango depends on the GEOS library. Also in combination with oracle. When trying to connect to the spatial database, I'd get an ImproperlyConfigured: Could not import user-defined GEOMETRY_BACKEND "geos". error. Well, that's just another library to install, right?
Problem: there's no windows installer for geos. Only compile instructions. What? Can't be true. So I asked around on two mailinglists and on irc and got great help. What the situation is?
Next DLL problem on the list: matplotlib. After finally getting over the oracle + geo database hurdle, I could really start up Django. Only to have it die almost instantly once a graph had to be shown. Without a clear error message:
from matplotlib._path import affine_transform ImportError: DLL load failed: The specified module could not be found
Well, which DLL?!? I had to go into the source code to get it to print a better error message. The ImportError exception was silently swallowed and a different import tried which also failed with a different error... The real error message gave me more information about which DLL was missing.
Dependencywalker to the rescue again. This DLL was missing a linkinfo.dll. Another one of those microsoft DLLs. Not available, though. And googling it gives a metric ton of virus links and warnings.
Some more googling showed that more recent matplotlib releases were compiled in a more reliable way with ms visual studio instead of the mingw compiler. So I grabbed a 1.0.0 release which even worked.
The python imaging library isn't clear on how it is supposed to be imported. "import Image", "import PIL.Image", "from PIL import Image". You see all of them. But that's apparently a problem on windows!
Python (and even apache plus mod_wsgi) completely died with a AccessInit: hash collision: 3 for both 1 and 1 error message. It turned out to be PIL. "Image" and "PIL.Image" are the same object, but with a different name. I have no idea why this would be different between windows and unix, but whatever.
The solution is to put this hack in your django's settings.py (see the comments on that article linked above:
import sys import PIL.Image sys.modules['Image'] = PIL.Image
I'm handling Django and css/javascript files with django-staticfiles. This makes it easy to ship javascript and css files (and images) with your Django applications.
The end result, for me, is quite a number of css and javascript files.
Ehm? That's not good for the responsiveness of the website, as it increases the number of server requests that need to happen. And the number of requests is nowadays often a big percentage of the time it takes to load a page.
The basic solution: combine the javascript and css files into one javascript and one css file. (Well, css files need to be split a bit according to their media="screen, presentation" or media="print" settings).
There are a number of Django solutions. I went with django_compressor as I found a snippet of code that showed how to integrate django_compressor with django-staticfiles. (Warning as I made that mistake: one has an underscore, one has a dash in the name...)
The basic usage in templates is simple. I'm using a custom base UI django application for all our websites with a base template. I just had to add django_compressor's statement in there around my css/js blocks and I was done:
<!DOCTYPE html>
<html>
<head>
<title>
{% block title %}
Sample title
{% endblock title %}
</title>
{% load compress %}
{% compress css %}
{% block css %}
{# Blueprint css framework. #}
<link rel="stylesheet"
href="{{ STATIC_URL }}blueprint/screen.css"
type="text/css"
media="screen, projection" />
<link type="text/css"
href="{{ STATIC_URL }}jquery-treeview/jquery.treeview.css"
rel="stylesheet"
media="screen, projection" />
<!-- And a couple of other stylesheets, of course -->
{% endblock css %}
{% endcompress %}
{% compress js %}
{% block javascript %}
<script type="text/javascript"
src="{{ STATIC_URL }}jquery/jquery.js"></script>
<!-- Yeah, and more js, of course -->
{% endblock javascript %}
{% endcompress %}
...
And in the settings.py:
# Storage engine to be used during compression COMPRESS_STORAGE = "staticfiles.storage.StaticFileStorage" # The URL that linked media will be read from and compressed # media will be written to. COMPRESS_URL = STATIC_URL # The absolute file path that linked media will be read from # and compressed media will be written to. COMPRESS_ROOT = STATIC_ROOT
Note that all three statements depend on the django-staticfiles setup.
Bonus for django_compressor: it not only combines the js/css, but it also gives the static file that it generates a unique name that depends on the timestamp of all the combined resources. So every time something changes in one of the source files, a new combined file with a new name is generated. So you can cache it forever. This means that your users will only have to download one css and one js file once, basically. A huge performance boost.
A bit of simple apache config that I added for this caching:
<Location /static_media/> # This is the regular django-staticfiles directory. # I set caching for one day here. That works well enough # with our update frequency at the moment. ExpiresActive On ExpiresDefault "access plus 1 day" </Location> <Location /static_media/CACHE/> # This is the location where django_compressor places # the combined files it generates. Every one of them # can be cached till the day the coyote finally catches # the roadrunner. ExpiresActive On ExpiresDefault "access plus 10 years" </Location>
(You can get much more fancy with varnish caching and more cache control headers, but this works well enough for now. I'm all ears for better setups, though :-)
Side note: using django_compressor means a bit of settings.py boilerplate in your projects. And some apache config boilerplate. Why don't you make it easy for yourself and read up on what I wrote about software skeletons for easy creation of all your boilerplate?
I released checkoutmanager three days ago. An easy tool for managing all your svn/hg/bzr/git checkouts. After mentioning it on my blog, I got quite some positive reactions, bug reports and feature requests. Yeah! Nice to see that it's useful!
Bug reports and feature requests mean I got to make some changes and fixes and new releases, of course :-) So 1.1 and 1.2 are out. Some of the bigger changes:
(Update: it works on windows, too, now: 1.1 is out)
All my life is stored in version control, basically. Several django applications, several websites, some personal stuff, my PhD thesis: everything is in subversion/mercurial/bzr/git. I just counted them: I've got 86 checkouts at the moment :-)
Doing an "svn up" or "hg pull -u" in all of them is boring. And re-building the whole set of checkouts on a fresh machine is hell. And I knew I was about to be changing jobs at the end of 2009. Which meant a change of laptop. Which meant I wrote a small program to manage my checkouts: checkoutmanager. A simple easy_install checkoutmanager is enough to install it.
The basis is a ~/.checkoutmanager.cfg configuration file:
# Sample config file. Different sections per base location
# and version control system. Splitting everything all over
# the place in multiple directories is fine.
[recipes]
# Buildout recipes I work on.
vcs = svn
basedir = ~/svn/recipes
checkouts =
svn://svn/blablabla/trunk
svn://svn/another/trunk differentname
http://host/yetanother/trunk
https://host/yetanother/branches/reinout-fix
[dotfolders]
# Folders that end up as dotted configfolders in the root.
# Note that there's a directory name behind the repository
# location, separated by a space.
vcs = bzr
basedir = ~
checkouts =
lp:emacsconfig/trunk .emacs.d
sftp://somwhere/subversion/trunk .subversion
So you've got a couple of sections. Every section has a base directory where everything is placed. And a version control system: so one vcs per directory. And a couple of repository locations.
What I normally do every morning when I get to work is checkoutmanager up. This grabs the latest versions of all my checkouts from the server(s). So an svn up for my subversion checkouts, a hg pull -u for mercurial and so on.
From time to time, I'll do a checkoutmanager st to show if I've got some uncommitted files lying around somewhere. Very handy if you've worked in several directories throughout the day: it prevents you from forgetting to check in that one bugfix for a whole week.
A new project means I add a single line to my config file and run checkoutmanager co.
Checkoutmanager allows you to spread your checkouts over multiple directories. It cannot mix version control systems per directory, however. As an example, I've got a ~/buildout/ directory with my big svn website projects checked out there. And a directory with my svn work python libraries. And a ~/hg/ dir with my mercurial projects. And I've made checkouts of several config directories in my home dir, such as ~/.emacs.d, ~/.subversion and so on. Works just fine.
I'm real happy with checkoutmanager. It is simple and it works. What more can you ask for? So I invite you to try it. Just run "checkoutmanager" and you'll get a full explanation and even a sample config file.
This is the last post in my software releases series (at least, the last as I originally planned the series).
In the last 5 articles, you've seen a couple of places where there's some irritating boilerplate. What's supposed to go into your setup.py? How did you start off with a buildout.cfg again? Oh, don't forget the CHANGES.txt and a basic README.txt. The solution for that is something I normally call a skeleton. I mentioned skeletons earlier in a Dutch python user group talk on practical project automation.
A skeleton is a basic project structure with a README, a couple of directories, some python files to start off with and whatever else you want to put in there. You give it some parameters (project name, website address, whatever) and it'll fill in the blanks with those parameters.
Django has a manage.py startapp, but there's a generic python tool called PasteScript. PasteScript's documentation almost exclusively talks about running wsgi applications, but it actually can create project skeletons for you. The best way to get started is to look at ZopeSkel, which has a large collection of skeletons. Download it and see what it can do.
There are two core advantages to using skeletons either for yourself or within your community or within your company:
Want to start your own skeleton? What I'd recommend is to start with a ZopeSkel download and to look at their code and how to set it all up. Then start your own. I worked for Zest software and started "zestskel" there. I worked for The Health Agency afterwards and started "thaskel" there. Now I work at Nelen en Schuurmans so I've started "nensskel" here :-)
Some examples of what I get when I create a new django site with nensskel (after giving it a project name):
So: make it easy to do the right thing. Let laziness work in your favour. Start a skeleton today!
Djangocon.eu 2011 is going to be organized in... Amsterdam!
I opened the evening with a summary of last month's djangocon.eu conference. I won't write down a summary of the summaries that I already made of all the talks :-) Some main points:
Some links I gave:
Subtitle: "an academic Jobboard with Django. Also featuring: Plone and SOLR". Academictransfer is a spin-off from the VSNU, the combined Dutch universities. Academic transfer has a lot of academic jobs in the Netherlands and they're expanding abroad. It is really focused on academic jobs.
Pareto, his company, uses Plone a lot. Plone is a really good content management system, but people sometimes abuse it for tasks for which it is not suited. Since a few years, they use Django for those kind of non-Plone-suited projects.
The old side was a 10-year old coldfusion site. A classic case of vendor lock-in of the company that build them the original site. Performance problems and big problems at every change request.
They originally asked Pareto for a Plone website ("it is a lot of content"). Plone is a good CMS, but this is mostly a database-driven site, so they advised Django.
They also use SOLR. SOLR is a high-end open source search engine (from the apache world). Till now, this has proven to be a good choice.
They use buildout (YEAH! See my buildout articles). It is their workhorse for setting up Django websites. With a buildout config, you can have a fresh, running django website within a minute. (Note: read Jacob Kaplan-Moss's article about Django and buildout).
People are looking for a job, so the search functionality is the most important part of the site. SOLR works great. It is good at searching with facets. A facet can be an organisation or a keyword or an academic field for instance.
There's login and personalization. Storing your queries; getting mails with new vacancies. Universities also can have their own page in the site. Most of the website is made with Django, but Plone is also used. An apache config divides up the requests between the both of them.
The UI is reused between Django and Plone. They did this by having Django ask plone for a base template. What plone returns is a plone browser view, but it is a Django template with Django's tags! The base template is downloaded with a cron job every night as the Plone site doesn't change that much. Another option would have been deliverance. Nice system.
A localization problem: Plone and Django use different methods for setting the language. Django stores it in the session and Plone uses a separate i18n cookie. The solution was to do the language switching on the Django side and get Django to set the Plone cookie, too. (An alternative could have been some django i18n middleware).
User management is Django-only. In the Plone pages, you want to show the username, too, though. The solution was to add an ajax call to the Plone interface that asks Django for the username :-)
The whole website is cached with varnish (which is basically the standard in the Plone world). For the username stuff, they had to use aggressive cache invalidation. An alternative they're going to pursue is SSI (server side includes). A main reason is that such ajax injection goes against the webrichtlijnen (Dutch accessibility guidelines).
They want to show some Django content, like recent job entries, in the Plone homepage. The solution: load those entries via RSS/Atom feeds in Plone portlets. You have to make those feeds anyway.
Website statistics are of course important. The counters are stored in a Redis store. Redis is a very simple noSQL database that's basically only a key/value store. Lightning fast. It uses http for connections. He had some doubts about that, but it turns out to be fast, too. Google analytics wasn't an option as academic transfer had some custom requirements. Especially very custom "funnels". Funnels are specific sequential steps that a customer can take through the site.
Some extra features:
Question: when would you use Django, when Plone? Answer: for this site, they'd have gone with all-Django. Plone is only used for only 15 pages. But if your site needs real CMS functionality, use Plone. So if you need workflows, per-page and per-folder security, stuff like that. Don't build anything like that yourself: it is already there in Plone.
When you update your site, there are a lot of manual steps you need to do. That's not DRY (don't repeat yourself). Some options:
(I mentioned a djangocon talk about capistrano.)
Dynamic models are not defined in your python code. For instance, they're created on the fly at runtime. You could create a flexible admin this way. Or you can dynamically prototype models without having python knowledge.
Or you can, before a syncdb, create dynamic copies of your old data models so you can still access them. Or, what he uses it for, generate additional models for localization.
How? Something like:
Person = type('Person', models.Model, attrs={..field...})
In the end, it is more involved of course. Most of django's machinery still works just fine with those classes. You can run syncdb just fine from within your code, creating the tables in your database.
So: do you have a usecase for this? He said there was a nice article about this subject on the Django website.
Alper's been programming Django for 5 years. Recently he's working with google's app engine.
Django is good for content-heavy websites and for complex websites that are composed of many apps.
App engine is made for scalability and availability.
The good things about app engine:
The bad things:
Some tips: use indexes, memcache and create json endpoints.
It is ideal for getting stuff up and running quickly. Instant gratification. He wouldn't really tie his whole company to google at the moment.
They got way too much traffic to one of their sites (http://www.fashiolista.com/) as it got waaaay to popular.
A default approach is to cache pages in Django: the @cache_page decorator.
NGINX and memcached are also a great combination.
Their problem: most of the content is user-specific and thus uncacheable.
The solution: combination of SSI (server side includes) and javascript. Memcached stores an anonymous page and the javascript knows how to turn it into a personalized page. SSI mixes in a bit of data with personalization instructions into the page. Javascript then applies those instructions. This SSI mixed-in data is just a very small Django request.
He showed result statistics that were pretty sweet.
A bigger tutorial and code will be up soon at http://mellowmorning.com
His goal was to find an object oriented way to create reusable views for his project. Django had almost the same goal for its "generic views".
He used the __init__() method and django is starting to use the __call__() method. The __call__() calls the actual methods that you can override in subclasses. The url pattern is a problem. Thread-safety is an issue, they're still thinking about an optimal solution.
An advantage of his __init__ method is that the url pattern can stay the same. There are some disadvantages with subclassing and returning self.
Search the django developer mailing list for "class based generic views in 1.3" to get the full discussion.
Benoît is one of the CouchDB developers and the maintainer of CouchDBkit and a couple of other packages.
Why go with a NoSQL database? Often our data isn't 300GB and the data fits one machine. Scalability isn't always needed. And regular SQL databases are often fast enough, right? Some reasons for NoSQL:
Some CouchDB characteristics: map/reduce; REST interface; document oriented database; append-only; replication.
Document oriented, what's that? It means schema-less json with attachments. So you can put any json you want into it, you don't have to define a schema beforehand. There's a web frontend called Futon to introspect your data (basically phpmyadmin for CouchDB).
There's more. For instance sharding. And geocouch, full r-tree.
CouchDBkit is a CouchDB python framework. It is a simple client and it also comes with a django extension. The django extension allows you to define models (if you want) so that you also can generate forms for it.
For the next version, he wants admin integration, some new mappings (objects with annotations), eventlet support, multidb support.
Question: the current couchdb version is 0.11 or so. Should we wait for 1.0? Answer: just use the 0.11, it is stable.