Reinout van Rees’ weblog

Experiment: cheap small camera for model railroad

2016-04-01

Tags: modelrailway, eifelburgenbahn

As an experiment, I bought a cheap (€9.08!) mini camera off Aliexpress.com (search for “y2000 mini camera). I wanted to see whether I could use it to make on-board model railway video. I didn’t expect much; my main worry was whether the camera could handle close objects. I want so see things 20cm - 2m away, not 2m - 200m!

A bit of ducttape:

http://abload.de/img/screenshot2016-04-01a2wsva.png

... and let’s roll!

http://abload.de/img/screenshot2016-04-01atuqjx.png

The quality is not very good, but it sure is a lot of fun for 9 Euros :-) I especially like the view of my viaduct at 2:15.

Model railway viaduct

2016-03-20

Tags: modelrailway, eifelburgenbahn

At home, I’m building a model railway. I haven’t shown anything about it on my blog, yet. I’m planning to show something from time to time, though. And I’m planning to start a new section of my website, dedicated to my railway, once I’ve figured out a good way to host images here :-)

Very very quickly: I’m modelling my railway after a single track secondary line in 1970 in Germany (the Eifelquerbahn for those familiar with German railways). It is located in the attic, which means I needed some creative solutions.

One of those creative solutions was “how to span the 2 meter gap over the staircase”.

Well, from below the stairs, it looks like this:

http://reinout.vanrees.org/images/2016/viaduct-from-downstairs.png

A reasonably light-weight two-part plywood box construction 10-20cm width and some 40cm high. 2m long, it is only supported at the ends. Plenty strong, as the entire backplane is solid 12mm thick plywood. A door hinge connects the two parts in the center. At the top a bolt keeps the two parts aligned correctly.

And... the reason why I’m posting this blog post: I’ve just finished the landscape of this section! Here’s a video:

http://reinout.vanrees.org/images/2016/vimeo-viadukt.png

Some comments:

  • The landscape might be ready, but the train still needs to be weathered (=made dirtier). And the coaches need passengers and lighting.
  • The train is running through a stop-showing signal. Sorry, I haven’t connected the signal yet.
  • Yes, it is a prototypical push/pull train. I’m not being lazy by not running the locomotive around :-)
  • If you can read dutch, I’ve got a lot of info on the Beneluxspoor forum
  • I’ve started in German on Stummiforum.

At least it is something different from my regular programming-related stuff here on this blog :-)

Buildout/setuptools speed improvement

2016-03-17

Tags: python, buildout, django

Sometimes buildout can seem to hang almost forever, even though you’ve set a timeout. It happens when installing the project itself.

  • Buildout effectively calls python setup.py develop on your project.
  • Setuptools then first calls os.walk() and builds a complete list of files.
  • Only then does it read your MANIFEST.in and some setup.py settings to determine which files to exclude and include.
  • If you have lots of files inside your directory, this might take a long time.

The key thing here is that it first builds a complete list. This includes that BUILDOUT_DIR/var/something directory with 245.934 files. And the BUILDOUT_DIR/node_modules/ with your full npm-downloaded stack of javascript with 10.000 files. And the BUILDOUT_DIR/var/project/ symlink to some slow windows share.

Ouch. A buildout that normally takes half a minute can take two hours this way...

The setuptools bug reports:

Intermediate solution: place this monkeypatch_setuptools.py next to your setup.py:

import os

TO_OMIT = ['var', '.git', 'parts',
           'bower_components',
           'node_modules', 'eggs',
           'bin', 'develop-eggs']

orig_os_walk = os.walk

def patched_os_walk(path, *args, **kwargs):
    for (dirpath, dirnames, filenames) in orig_os_walk(path, *args, **kwargs):
        if '.git' in dirnames:
            # We're probably in our own root directory.
            print("MONKEY PATCH: omitting a few directories like var/...")
            dirnames[:] = list(set(dirnames) - set(TO_OMIT))
        yield (dirpath, dirnames, filenames)

os.walk = patched_os_walk
# ^^^ This only modifies os.walk for the duration of calling setup.py

And then import the monkeypatch right at the top of your setup.py:

from setuptools import setup
import monkeypatch_setuptools

version = '0.3.dev0'

setup(
    name='your-package',
    version=version,
    ...

It works quite well! Some notes:

  • Adjust the TO_OMIT to your needs and local conventions.
  • There’s a check if '.git' in dirnames in the monkey patch. I use that to detect whether os.walk is currently in a directory with .git, which normally means our own base directory. So it’ll only strip out directories in there and not somewhere else. Just a small safety valve. You’ll have to adjust it if you use mercurial or something else.

Dutch django meetup (feb 2016) summaries

2016-02-23

Tags: django

Dutch django meetup revival - Erik Romijn

Sponsored by http://byte.nl, organized by the Dutch django association (django vereniging). The association was founded to organize the 2011 Django Europ conference. They’re also behind the “django under the hood” conferences.

They’re also supporting django girls workshops and even Python Namibia 2015.

If you want to become a member, mail bestuur@djangovereniging.nl

If you want to help with future meetups, contact him!

Migrating nu.nl to Django - Besma Mcharek-Boussif

Besma works for Sanoma, the company behind http://www.nu.nl/, a very big news site in the Netherlands. The real title of the talk is “nu.nl performance and our journey in the django realm”. She’s a django backend developer. She’s also a co-organizer of PyLadies meetings.

Nu.nl is the biggest Dutch news platform. 10^9 pageviews per day. They specialize on getting the news online fastest. Mobile and desktop web sites plus mobile apps (ios, android and windows) for tablets, phones and smart watches.

There’s now an ios team, an android team and a web/API and a CMS team. The API is also used by the mobile apps, btw.

Regarding development: they try to use a devops mind set. It is almost continuous integration. About 10 developers. You should be able to work on any part of the system. Development tools: jira, bamboo, hipchat. Monitoring is done with GA and New Relic.

The architecture is divided in layers. CMS, API, frontend. You of course always have to keep them in mind. Editors use the CMS (content management system) to enter content into the database. The database is used by the API (with a cache in front) to feed content to the mobile apps and the website. The website has its own cache.

The web/API team has three kinds of output:

  • Serving static pages for the main website.
  • Push notifications for big news.
  • Serve the raw data for certain API users (for use by third parties to use in hospital news monitors, for instance).

Past/present/future: the whole stack used to be written in php. Development and maintenance was by an external party. Not a good idea, especially when the traffic became crazy.

Currently most of the stack is in django, developed and maintained by the their own team. Only the CMS (for the editors) is still made in PHP.

Future: trying to get all of the stack in django, especially the CMS part. Also to get also this part of the stack in-house.

The site is made using a block layout. Business and editors decide on the layout. Editors supply the blocks with content The resulting layout is served to the web front-end and to mobile.

There are still challenges in the current setup: scale, speed and API errors. “Scale” means “big news items so lots and lots and lots of people come to the site”. How do they deal with this?

  • Django security.
  • Django rest framework as data provider.
  • Lots of cache servers (mostly Varnish).
  • Rigorous testing (also manual). Especially the API is almost fully unit-tested. If there is a bug, she writes a test. Period. She doesn’t accept new functionality without tests.

Recently they had a performance sprint:

  • Updated python to 2.7 (their hosting party sadly only supported 2.6, originally).
  • Updated from apache to nginx.
  • Disabled django.views.static.serve.
  • Using “serpy” as the API’s serializer. They found it to be faster than Django Rest Framework’s default one.

One of the biggest future challenges is to add more personalisation. This means pages are no longer static, which means a performance problem. Also the API will have to be changed.

Djangocon Europe - Baptiste Mispelon

He’s the chair of the upcoming Budapest Djangocon Europe. The tickets are almost sold out, so if you want to go, be quick! They can also still use sponsors.

Django Girls Den Haag - Martin Matusiak

The python community is well-known for being friendly and cozy.

Sometimes when working on a shitty piece of software with lots of dirty legacy, you yearn for a nice codebase. The python community is that nice codebase. We’re in a good shape. We’re a nice, comfortable community to be around.

A django girls workshop is aimed to take non-programmers (especially woman as they’re under-represented) and existing developers (as mentors) and help the women to learn to program and learn to love to program.

Lots of people show up because they want to learn what their colleagues do. Others look for a career change.

“What comes out of a workshop” is programmers. Beginning programmers. And also mentors for them! Also for the more advanced programmers it is interesting as you can learn to mentor people. Being a coach is quite relaxing, btw.

The interest for the upcoming meeting in The Hague is overwhelming. 100 applications when there’s only room for 60!

They hope it’ll be organized in more cities. In the Netherlands it is regularly organized in Groningen and Amsterdam.

If you want to organize it: there is a ready-made organization manual!

(Note to self: my daughter wants to learn python programming with three of her classmates...)

Djavascript that doesn’t make you cry - Sergei Maertens

Sergei works at Maykin Media as a backend developer. (The presentation is online on github, btw).

The most used javascript version is ecmascript 5.1 from June 2011. Ecmascript 6 (“ES6”) came out in June 2015, but not everything supports it yet.

Some problems that make you cry in javascript (the 5.1 version):

  • Global scope polution.
  • Loading/bundlign files. django-compressor/django-pipeline help here. But it is another thing to set up maintain in your project.
  • Unit testing.
  • Most will do jquery development. Jquery isn’t javascript.

ES6 features that’ll make you happy:

  • Proper modules. import $ from 'jquery';

  • To make something available to other modules, you have to explicitly export it and import it in another. This solves the polution and file-loading issues.

  • Native classes! Object oriented programming is now more within reach. It is mostly syntactic sugar around the existing prototypical inheritance, but ok.

  • Async javascript. You can already use callbacks (the nodejs way, “callback hell”). Or promises (now finalized in ES6). And coroutines (very much like asyncio from Python).

    Promises will either return a response or an exception. There is no other possibility. You call something (which you call a promise) and then call “then” on it. When the promise-handler is finished, it’ll return and execute what’s in the .then().

    Remember to catch exceptions, as they’ll be eaten quietly and suppressed. Bah.

  • Arrow functions. Basically something that looks like a python list comprehension.

  • New let and const var-like keywords. let is more limited, it limits the scope of the variable to the current loop, for instance. const is meant not to be changed. Browsers don’t really support that yet, though.

  • Template strings. Hello ${world.name}

  • A useful full list of ES6 features: https://github.com/lukehoban/es6features

A recommended tool: JSPM/Systemjs, see http://jspm.io/ . (“Do we need a new javascript package manager? Yes!”) It handles transpiling ES6 to ES5. It handles bundling. It is customizable by overriding package.json. It loads ES6 modules, commonjs, AMD, all in one codebase: you can use three styles of imports in one codebase.

Now... how do we use it with django? With Sergei’s own django-systemjs, of course :-) It integrates with django’s template language. It handle both DEBUG = True mode and production setups. It has a python manage.py systemjs_bundle management command.

He showed something from ponyjs (https://github.com/ponyorm/pony). It is a model layer aimed at Django Rest Framework. It uses ES6 and its Promises.

Just one comma

2016-02-10

Tags: django, python

In Django, if you say managed = False in a model’s Meta, you tell Django not to touch the database table. So: no automatic database migrations, for instance.

Now, what is the problem if you have managed = False, and Django does do migrations?

Some of you will have seen the error already.

The comma after False is the problem:

>>> a = False,
>>> a
(False,)
>>> bool(a)
True

The comma turns it into a tuple. And a non-empty tuple evaluates to True!

I found it quite funny. My colleague was torn between “extremely relieved” and “extremely annoyed” :-)

“Complex” and “complicated”

2015-12-21

Tags: python

I’m not a native english speaker, so sometimes I need to refresh my memory as to the exact meaning of a word. In this case the exact difference between complex and complicated.

I found an english.stackexchange.com page with some nice answers. According to the first answer:

  • Complex is about the number of parts. The more (different) parts, the more complex.
  • Complicated is about how hard/difficult something is.

Another answer pointed at the relevant part of the Zen of Python:

Simple is better than complex.
Complex is better than complicated.

So when programming, you can divide up a task or problem into separate parts. The more parts, the more complex. Simple is better than complex, so don’t add too many parts if you don’t need them.

On the other hand, if you want to do everything in one part, it might end up being too difficult, too complicated. Making it more complex (more parts) is better in that case.

It is a trade-off. And it takes experience and a bit of Fingerspitzengefühl to make the right trade-off.

How to learn it? Talk with more experienced programmers is one way. Another way is to explicitly review your programs yourself with those two lines from the Zen of Python. I’ll show them here again to drive them home:

Simple is better than complex.
Complex is better than complicated.

Ask yourself questions like this:

  • Do I still understand my own program? Did I make it too complicated?

  • Did I split it up enough? Or are there 400-line functions that I might better cut up? Into a more complex, but clearer, whole?

    Well-separated parts might make reasoning about your complicated problem easier.

  • Did I split everything up too much? Do you have more parts than your brain can handle?

    Your brain’s CPU has room for some 7 variables. Too many parts might make it impossible for you to work on your own code :-)

Hit by a car for the first time

2015-12-08

Tags: bike

Today was the first time I got hit by a car. On my (recumbent) bicycle. First things first: nothing is wrong :-)

If you look on this map, I crossed the “Winthontlaan” right in the center of that map from right to left. Contrary to that map, the cyclists have the right of way crossing that road (since half a year or so).

I crossed the road. The first half of the road was full of cars, as it often is late in the afternoon. So cars coming from the right don’t have a very good view on the cyclists that want to cross. The driver didn’t notice me. When I was crossing, there was quite some room between me and an approaching car, so I assumed he would stop. Room enough. And cars normally drive slowly there because of the unclear situation. This car did, too.

Only... he kept on driving. Weird to experience it. You see a car approaching and it doesn’t stop. There were no alarms going off in my head, only cool calculated distance calculations :-) I remember concluding that I would get hit.

Also weird: I don’t remember exactly what happened when I got hit. I just ended up on the ground. The whole process probably only takes three seconds, so no wonder you don’t remember the exact process.

The car only hit the back of my bike near the bag with my laptop backpack. From the right side. The total center of gravity was probably a little bit above the point where it hit, so I probably got lifted a little bit and ended up on the right side of my body.

  • No personal damage, I got up right away and everything checked out OK.
  • The owner of the car stopped immediately and first inquired of my health (“that’s the most important thing”). Very kind, correct and helpful!
  • We then checked out my bike. We had to adjust the angle of the front wheel and wriggle the brakes a bit, but that was about it.
  • I got the car owner’s email address in case I did find some additional daamge. I’ll mail him this summary if I don’t find anything when I check out my bike tomorrow :-)
  • I totally forgot to inquire after the car, weird enough. I asume it has no damage, though, as it hit my bike in a nice soft padded spot.

So: no harm done. And a gallant car owner that helped me afterwards. Not a bad way to get an accident :-)

An hour later I started to get sore on my left leg. I probably hit my steering bar with it or so. I know for sure I’ll feel it tomorrow. Nothing wrong, just sore.

Oh, I have gotten one thing out of the accident: I now know what was bothering me with my bike for the last week. My bike felt a little bit “squishy” when steering. Something somewhere was loose. I just couldn’t find it. After the accident it worsened a bit and I finally spotted which nut I have to tighten. Funny.

As a former dutch soccer player used to say: every drawback has its advantage.

It also has the advantage of cheaply instilling in me the need for a bit more defensiveness in my driving. Not that I’m driving recklessly, far from it. I even stop for red traffic lights, unlike 90% of the dutch population :-)

Some legal/dutch comments for those that live in a country where cycling is not ubiquitous:

  • I simply had the right of way, so I had the right of way. Sounds logical, but the stories I sometimes hear from the UK sometimes make the dutch situation sound like a luxury. “But it was a cyclist” seems to make everything right (for the car owner) in the UK, it seems.

  • It was dark, but I had my lights on. And that’s a luxury, as 40% of the cyclists don’t have a light in the Netherlands. So I was very visible (compared to most other cyclists). I have a good (85 Euro) headlight.

  • Yes, I have a recumbent bicycle that’s lower than “normal” bicycles. That puts my head more or less on the same level as that of car drivers. Normal bicyclists are higher than cars. I assume I cannot be blamed to be more or less car-height? Especially as I had my lights on. And my light is at the same level as most other bikes.

  • If there would have been damage (which there wasn’t), the money would have to be payed by the car driver unless the cyclist was very, very clearly and recklessly at fault. This way, the law protects the most vulnerable traffic participant. If you drive a car, you technically drive a 1000 kg killing machine, so you have more legal responsibility than a mere cyclist or pedestrian.

  • I have cycled since I was a young child. At the age of 9 I already cycled 1.5km to school on my own. Nowadays I cycle 9km every morning and afternoon. I’m 42 and this is only the first reasonably serious accident I have. I had one crash with another cyclist at almost the same point. The other cyclist totally cut the corner and hit me straight on even though I was driving neatly on the right side of the road.

    So: cycling in the Netherlands is very, very safe! And I enjoy it every day.

Fixing SSL certificate chains

2015-11-23

Tags: python, django

This blog post applies when the following two cases are true:

  • Your browser does not complain about your https site. Everything seems fine.
  • Some other tool does complain about not finding your certificate or not finding intermediate certificates. What is the problem?

So: your browser doesn’t complain. Let’s see a screenshot:

Browser address bar with a nice green closed lock, so ssl is fine

Examples of the errors you can see

Some examples of complaining tools. First curl:

$ curl https://api.letsgxxxxxxx
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

curl has the right error message: Invalid certificate chain.

Let us look at wget:

$ wget https://api.letsgxxxxxx
--2015-11-23 10:54:28--  https://api.letsgxxxxx
Resolving api.letsgxxxxxx... 87.233.157.170
Connecting to api.letsgxxxxxx|87.233.157.170|:443... connected.
ERROR: cannot verify api.letsgxxxxxx's certificate, issued by 'CN=COMODO RSA
  Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB':
  Self-signed certificate encountered.
To connect to api.letsgxxxxxx insecurely, use `--no-check-certificate'.

wget is right that it cannot verify .... certificate. But its conclusion Self-signed certificate encountered is less helpful. The certificate is not self-signed, it is just that wget has to treat it that way because the certificate chain is incorrect.

If you talk to such an https URL with java, you can see an error like this:

javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException:
PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

This looks quite cryptic, but the cause is the same. SunCertPathBuilderException: CertPath sure sounds like a path to a certificate that it cannot find.

A final example is with the python requests library:

>>> import requests
>>> requests.get('https://api.letsgxxxxxx')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File ".../requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File ".../requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File ".../requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File ".../requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

How to determine what’s wrong

So... you yourself discover the problem. Or a customer calls that he’s getting an error like this. Even though everything seems right if you test the https site in the browser.

Solution: go to https://www.digicert.com/help/

If that site says everything is completely right, then you’re done. If it still complains about something, you’ve got work to do.

Most of the checkmarks are probably green:

Green checkmarks in front of many common SSL checks

In cases like this, the problem is in the certificate chain at the bottom of the page. Here’s an example of one of our own sites from a few months ago:

Broken chain icon indicating the exact problem spot

Note the “broken chain” icon halfway. Just follow the chain from top to bottom. Everything has to be perfect. We start with the *.lizard.net which is issued by GeoTrust SSL CA - G2.

The certificate GeoTrust SSL CA - G2 in turn is issued by GeoTrust Global CA.

The problem: the next certificate in the chain is not about GeoTrust Global CA, but about GeoTrust SSL CA, which is different. Here the chain breaks. It does not matter that the fourth certificate is about the GeoTrust Global CA we were looking for. The chain is broken. The order in which the certificates are placed must be perfect.

After fixing the order of the certificates in our certificate file, the problem was fixed:

Chain icons indicating that the chain is unbroken

Why is a chain needed?

There are lots of certificates in the wild. All the browsers (and java, and your OS and...) often only store a handful (well, 20+) “root certificates”. All the other certificates have to trace their origin back to one of those root certificates.

That is where the intermediate certificates come in: they’re a cryptographically signed way to trace the validity of your certificate back to one of the known-good root certificates.

How to fix it

  • If you’re handling certificates yourself, you ought to know which files to edit. The main problem will be getting the right intermediary certificates from the issuing party. Often you only get “your” certificate, not the intermediary ones. Ask about it or google for it.

  • Often you won’t maintain those certificates yourself. So you have to get your hosting service to fix it.

    If you let someone else take care of the certificate, point them at https://www.digicert.com/help/ and tell them to make sure that page is completely happy.

    In my experience (=three times in the last two years!) they’ll mail back with “everything works now”. But it still won’t work. Then you’ll have to mail them again and tell them to really check https://www.digicert.com/help/ and probably provide screenshots.

Good luck!

Nginx proxying to nginx: getting gzip compression to work

2015-11-19

Tags: python, django

At work we use gunicorn as our wsgi runner. Like many, gunicorn advises you to run the nginx webserver in front of it. So on every server we have one or more websites with gunicorn. And an nginx in front.

Nginx takes care, of course, of serving the static files like css and javascript. Some gzipping of the results is a very, very good idea:

server {
    listen 80;
    server_name my.great.site;
    ...
    gzip on;
    gzip_proxied any;
    gzip_types
        text/css
        text/javascript
        text/xml
        text/plain
        application/javascript
        application/x-javascript
        application/json;

    ....
}

Two notes:

  • The default is to only gzip html output. We also want javascript and json. So you need to configure gzip_types.

    (I copy-pasted this from one of my config files, apparently I needed three different javascript mimetypes... Perhaps some further research could strip that number down.)

  • gzip_proxied any tells nginx that gzipping is fine even for proxied requests.

Proxied requests? Yes, because we have a lot of servers and all external traffic first hits our main nginx proxy. So: we have one central server with nginx that proxies requests to the actual servers. So: nginx behind nginx:

server {
    listen   443;
    server_name my.great.site;
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://some-internal-server-name/;
    }
    ssl on;
    ssl_certificate ...
    }

Pretty standard “I listen on 443/https and proxy it on port 80 to some internal server” setup.

Works like a charm. Only drawback: gzipping does not work.

The reason? nginx defaults, in this case.

  • The gzip module has a gzip_http_version configuration parameter with a default of 1.1.

    Which means that http 1.0 requests are not gzipped, only 1.1.

  • The proxy module has a proxy_http_version configuration parameter with a default of 1.0.

    Which means that proxied requests are send from the main proxy to the actual webserver with http 1.0.

These two don’t match. There are two solutions:

  • Set gzip_http_version 1.0 in the nginx configs on your webservers. This switches on gzip for the http 1.0 connections coming from the proxy.
  • Set proxy_http_version 1.1 on the main proxy so that it sends http 1.1 connections to the webservers.

My choice originally was to do the first one. But a bug report came in for another site and now I’ve switched it on on the main proxy so that all the sites get the benefit.

Note: you might want to make different choices. Perhaps you have a caching proxy halfway? Perhaps you want the main nginx on the proxy to do the gzipping for you? Etcetera. Check whether the above tips apply to your situation :-)

Buildout 2.5.0 has much nicer version conflict reporting

2015-11-16

Tags: python, django, buildout

We use buildout for all our django projects. Nothing wrong with pip, but buildout has extension possibilities build-in (for creating directories, installing user crontabs, local development checkouts and many more) that are quite helpful. And it works better when you need to use system packages (gdal, mapnik, etc).

One area where buildout could use some improvement was the version conflict reporting. Let’s say you have pinned django to 1.6.6 (old project that I’ll upgrade to 1.8 this week) and you add the django debug toolbar. This is the error you get:

The constraint, 1.6.6, is not consistent with the requirement, 'Django>=1.7'.
While:
  Updating django.
Error: Bad constraint 1.6.6 Django>=1.7

First things first. An easy one is to improve the wording of the message:

While:
  Installing django.
Error: The requirement ('Django>=1.7') is not allowed by
your [versions] constraint (1.6.6)

Now... so there is some package that requires at least django 1.7. But which one? Buildout did not tell you. Which would mean you’d have to grep in all your requirements’ sub-requirements for which package actually requires the offending “django>=1.7”...

I’ve now added some internal logging that stores which package required which dependency. After an error occurs, the list is searched for possible matches.

With this change you’ll get a much more helpful output right before the error:

Installing django.
version and requirements information containing django:
  [versions] constraint on django: 1.6.6
  Base installation request: 'sso', 'djangorecipe'
  Requirement of djangorecipe==1.10: Django
  Requirement of djangorecipe==1.10: zc.recipe.egg
  Requirement of djangorecipe==1.10: zc.buildout
  Requirement of sso: django-nose
  Requirement of sso: django-mama-cas
  Requirement of sso: django-debug-toolbar
  Requirement of sso: django-auth-ldap
  Requirement of sso: Django<1.7,>=1.4.2
  Requirement of lizard-auth-server: django-nose
  Requirement of lizard-auth-server: django-extensions
  Requirement of lizard-auth-server: Django<1.7,>=1.6
  Requirement of django-nose: Django>=1.2
  Requirement of django-nose: nose>=1.2.1
  Requirement of django-mama-cas: requests==1.1.0
  Requirement of django-debug-toolbar: sqlparse
  Requirement of django-debug-toolbar: Django>=1.7
  Requirement of django-auth-ldap: python-ldap>=2.0
  Requirement of django-auth-ldap: django>=1.1
  Requirement of translations: Django>=1.4
  Requirement of django-extensions: six>=1.2
While:
  Installing django.
Error: The requirement ('Django>=1.7') is not allowed by
your [versions] constraint (1.6.6)

This makes it much easier to spot the cause (in this case django-debug-toolbar).

There are some unrelated packages in here because I’m doing a textual comparison. The advantage is that it is very robust. And extracting the right package name from requirements without messing things up is harder to get right and takes more code.

So... if you use buildout, give version 2.5.0 a try!

 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):