Reinout van Rees’ weblog

Dying NTP deamons on vsphere vmware machines


Tags: django

We (Nelen & Schuurmans) have quite some servers. Most of them are vmware virtual machines in a vshpere cluster.

Once in a while, one or more of the machines got reported by our monitoring tool (zabbix) as having a time drift problem. Weird, as we have NTP running everywhere. And weird if you look at django logfiles and see a negative jump in time all of a sudden.

We run ntpd everywhere to keep the time in sync with two windows domain servers. Every time a server drifted, the ntpd daemon turned out to have died. Without leaving any trace in any logfile.

ntpd kills itself when the time drift is more than 20 minutes or so, assuming that it hurts more than it helps. There’s a switch to prevent this self-killing behaviour, but ntpd killed itself anyway.

In the end, an external sysadmin found the problem:

  • One of the physical vsphere host machines (big server, lots of blades) was mis-configured: the ntp daemon on the host machine itself was configured, but it was not configured to automatically start when you start up the server...
  • This host machine started to drift its time, naturally.
  • Several actions vsphere does on a VM result in a very very short period where the VM is frozen. Actions like “full backup”, “snapshot” and “automatically moving from one host machine to another for performance reasons”. Very short, but vmware does adjust the time inside the VM. It keeps track of how long the quick action took and adjusts the VM’s time accordingly.
  • It adjusts the time relative to the host machine’s time. So if an action took 1 second, the second is added to the host machine’s time and the result is set as the VM time. All is still well if the VM stays on the same host.
  • If the action includes moving the VM to a different host... And that host is the one with the drifted time.... If the host machine’s time has drifted by an hour, the VM that gets moved to that host suddenly gets its internal time moved by an hour...

So it was a combination of host machines with a drifted time and the fact that vmware adjusts the VM’s time after certain actions.

Writing it down as it might help someone googling for this problem :-)

Don’t import (too much) in your django settings


Tags: django

One of our production Django sites broke this afternoon with a database error “relation xyz doesn’t exist”. So: a missing table.

Why 1

I helped debugging it and eventually found the cause by doing a select * from south_migrationhistory. This lists the south migrations and lo and behold, a migration had just been applied 25 minutes earlier. The migration name suggested a rename of tables, which of course matches the “missing table” error.

Why 2

Cause found. But you have to ask yourself “why” again. So: “why was this migration applied?”.

Well, someone was working on a bit of database cleanup and refactoring. Naming consistency, proper use of permissions, that sort of thing. Of course, locally in a branch. And on a development database. Now why did the local command result in a migration on the production database?

Why 3

So, effectively, “why don’t the development settings work as intended”? We normally use as the production settings and a that is used in development. It imports from and sets the debug mode and development database and so.

This project is a bit different in that there’s only a It does however try to import This is generated for you when you set up your project environment with ansible. A bit less clear (in my opinion) than a real .py file in your github repository, but it works. We saw the generated localsettings file with development database and DEBUG = True. This wasn’t the cause. What then?

Normally, calling django’s diffsettings command (see the django documentation) shows you any settings errors by printing all the settings in your config that are different from Django’s defaults. In this case, nothing was wrong. The DATABASES setting was the right one with the local development database. Huh?

The developer mentioned one other thing he changed recently: importing some django signal registration module in the ````. Ah! Django’s signals often work on the database. Yes, the signals in this module did database work, too.

So the effectively looked like this:

DATABASES = { .... 'server': 'productiondatabase' ....}
import my_project.signal_stuff_that_works_on_the_database
    from .localsettings import *
    # This normally sets DATABASES = { .... 'server': 'developmentdatabase' ....}
except ImportError:

The import of the signal registration module apparently triggered something in Django’s database layer so that the database connection was already active. The subsequent change of the DATABASES config to the local development database didn’t have any effect anymore.

diffsettings just shows you what the settings are and doesn’t catch the fact that the DATABASES isn’t really used in the form that comes out of diffsettings.

Why 4

Why the import, then?

Well, it has to be executed when django starts up. The settings file looked like a good spot. It isn’t, though.

The traditional location to place imports like this is the or file. That’s why the admin.autodiscover() line is often in your, for instance.

So... put imports like this in or instead of in your settings file.

Why 5

Digging even deeper... isn’t this sort of weird and ugly? Why isn’t there a more obvious place for initialization code like this? Now you have to have the arcane knowledge to somehow know where you can import and where not, right?

The answer: there is a good spot, in django 1.7. The AppConfig.ready() method! Quote from the documentation: Subclasses can override this method to perform initialization tasks such as registering signals. Bingo!

Forward movement


Tags: personal

Over the years, one phrase started to help me a lot: “forward movement”. It is a question I can ask at the end of a day: “did I have move forward today”?. The subject of the question is often a project or a goal. The question itself is quite liberating.

Liberating? Yes. You cannot do your life’s work in one day. You cannot finish a model railway in one day. You cannot paint an entire house in one day. But you can move forward on one or more of them! “Did I put in time and effort to move one of my higher-priority items forward?” Or even “did I do a load of small things that in themselves aren’t terribly important but that kept nagging at me”?

I have three main areas/goals I want to put some real effort into this year. A year is a long time. You can do a lot in a year, provided you don’t try to do it all in the last few days of the year!

One of the three goals is the deliberately vague-sounding “lose mass”. My body mass should go down. But also the amount of paper/books/saved items/junk in the house should go down. And the backlog of I-should-read-it-one-day magazines. And... And... And... Lots of things can be placed under this banner.

Now... I can aim for perfection in one of the areas. Or I can aim at progress. Forward movement. “Practice progress over perfection” is a quote I recently saw. Nice.

Some progress I made:

  • I went to my thursday evening swordfighting lessons again for the first time in three months. Sore muscles all over, so that ought to help a tiny bit in getting my body mass down. Someone stabbing you with a knife or aiming a sword at your stomach tends to make you pretty active :-) I’m always drenched with sweat afterwards.

  • Yesterday I emptied out my overflowing email inboxes.

  • This morning I emptied my backpack (which I use for my laptop) of three months of accumulated papers and assorted bits and pieces. This gave me a surprisingly good feeling. Opening the bag and looking at the tidy interior instead of at a pile of cumpled papers...

    The mess apparently had “mass” as it put a weight on my shoulders (figuratively) just by me looking at it every morning and afternoon.

  • I cleaned out my desk. And found a couple of scribled down TODO items that I’ve now placed in my GTD system. Again: a bit of weight left my shoulders this way. Unconsciously you know there might be unfinished business lurking in the pile of papers on your desk.

I’m still amazed at the joy I now have in opening my backpack :-) Forward movement is great.

Updated instructions to fix macbook pro with black screen


Tags: python, apple

Last week I wrote about my macbook pro and its screen that suddenly stayed black after waking it from sleep. Including instructions on how to fix it. I now have improved instructions...

The piece of crap did it again. (This time there was no external monitor in play, btw)

Symptoms: you open up your late 2014 macbook pro 15” and expect it to wake from sleep in 0.5 seconds. Instead the screen stays black.

  • Press shift-option-control-power and release them all at the same time. This resets the “system management controller”, amongst others it makes sure the “power” button actually reacts.
  • Now look at the back of your mac’s screen. Make sure the light of the apple logo is off. If necessary, keep the power button pressed for 5 seconds to force a shutdown. (The power button should work again after step 1).
  • Keep your fingers ready for pressing command-option-p-r at the same time. You need both hands for this weird combination. Press the power button and then hold these four keys. Your mac will now actually boot, give you the startup sound and lo and behold, you’ve got your screen back.

I do hope apple will fix this Real Soon. I suspect a software bug. I hope it isn’t a hardware one. It was a hardware bug (wrong solder being used for the video card connection) that brought down my previous macbook after four years... Apples are great machines for programming (python) and I’m in love with the retina display, but...

Actually doing work - proof by blogging


Tags: nelenschuurmans, python

I’m a programmer. Which means it is not always clear when I’m working. What I mean is that, if I’m sitting behind my screen, I might be reading an article about postgres performance improvements, but I might also be reading the news. And clicking on my ipad might mean I’m keeping my python knowledge up to date by reading weblogs, but I might also be reading some comics. Typing furiously on my keyboard might indicate great productivity, but it also might indicate a long personal email.

The other way around, sitting nicely in the living room at home, clicking away on the laptop, might mean I’m relaxing by writing an update on my model railway work on a forum, but it might also mean I’m finishing off a work project in my own time.

“Doing research” is a potential problem for me. I’ve spend a ton of time reading about Ansible and on how to use it. But it was only when I actually started doing something with it that I started to wrap my head around it. So... just reading and thinking is dangerous to my productivity.

I want to be productive.

One of the simplest tricks I can use is to blog about it. Thinking about something for two days is fine, but I should at least get a blog entry out of it. Writing forces me to think things through. It focuses my thinking.

If I don’t blog for a full week, I’m probably not hard at work programming: otherwise I’d have encountered some dreadful mistake I made or some horrible bug, all of which are blog-worthy. Or I’ve started thinking too much about something without taking a higher-level view after a few days (again, resulting in a blog entry).

So... I’ll prove I’m working by blogging about it :-)

New emacs setup


Tags: python

I edit all my code with emacs. Since 1996 or so. And all blog entries. And all...

So my emacs settings are “quite important” to me. When I installed the new laptop I’m typing this on, my settings stopped working: I just deleted my ~/.emacs.d/ directory because I wanted to upgrade my setup.

Since a couple of years I used the emacs starter kit version 2. The link goes to my slightly modified copy of the original version 2 starter kit.

The setup change I wanted to make was to update to the version 3 of the emacs starter kit. That link goes to the real starter kit, which in version 3 is... just a text document!

Which is a good thing. You don’t need to clone a git repo just to get a bunch of settings. You use emacs’ package tool to download the better-defaults package, which contains most/all of the settings changes that made the original emacs starter git such a joy to use.

And on top of that, it suggests a couple of handy packages to use also. I’m using all of the suggested ones (apart from ‘scpaste’, which I don’t use).

And it provides a template for an almost-empty init.el main config file. I’ve copy/pasted a couple of lines from the version 2 starter kit at the end:

;; Standard emacs starter kit (version 3) startup code.
(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "") t)

;; List of packages that I use.
(defvar my-packages '(better-defaults paredit idle-highlight-mode ido-ubiquitous
                                      find-file-in-project magit smex)
  "A list of packages to ensure are installed at launch.")

;; Commands suggested by the Emacs starter kit to keep its packages up
;; to date.
(when (not package-archive-contents)
(dolist (p my-packages)
  (when (not (package-installed-p p))
    (package-install p)))

;; Load system specific init (read: "run only on osx").
;; Also load my username's config (read: my customizations are in 'reinout.el')
(setq dotfiles-dir (file-name-directory
                    (or (buffer-file-name) load-file-name)))
(setq system-specific-config (concat dotfiles-dir system-name ".el")
      user-specific-config (concat dotfiles-dir user-login-name ".el"))
(if (file-exists-p system-specific-config) (load system-specific-config))
(if (file-exists-p user-specific-config) (load user-specific-config))

Those last few of lines make sure that on my laptop, it finds the hostname-named nens-lt-148.local.el which starts emacs as a server. And it loads my username-named reinout.el with my personal customizations.

Anyway... Now my emacs config is just a few config files in my git-stored dotfiles directory (see for instance this article where I mention Dotfiles). Instead of a separate git clone. Nice and tidy!

VariableDoesNotExist: Failed lookup for key [text] in ‘None’


Tags: django

VariableDoesNotExist: Failed lookup for key [text] in 'None', that was the error that I got out of an internal django app that I wrote. The app is the most error-free one that I ever build, so I was quite surprised.

I was even more surprised when I started debugging it. The error was in the search functionality. Searching for “lizard” would give an “error 500” page. Searching for “lizar” gave 0 results (which is fine), for “lizardd”, too. “lidard” also gave 0 results. Only a search for “lizard” crashed the site.

Huh? Weird. I got the question whether I put a hidden message into this app (as “lizard” is one of our main products) :-)

Locally it all worked fine, of course. Only on the server did I see the error.

Here’s the relevant part of the traceback:

Stacktrace (most recent call last):
File "django/core/handlers/", line 112, in get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "haystack/", line 50, in __call__
  return self.create_response()
File "haystack/", line 144, in create_response
  return render_to_response(self.template, context, context_instance=self.context_class(self.request))
File "django/shortcuts/", line 29, in render_to_response
  return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
File "django/template/", line 169, in render_to_string
  return t.render(context_instance)
File "django/template/", line 140, in render
  return self._render(context)
File "django/template/", line 840, in render
  bit = self.render_node(node, context)
File "django/template/", line 78, in render_node
  return node.render(context)
File "django/template/", line 196, in render
File "haystack/templatetags/", line 30, in render
  text_block = self.text_block.resolve(context)
File "django/template/", line 735, in resolve
  value = self._resolve_lookup(context)
File "django/template/", line 781, in _resolve_lookup
  (bit, current))  # missing attribute

Eventually I found it. Via this django-haystack post. Haystack, which I use for searching, by default doesn’t deal with deleted objects. And, of course, yesterday I removed an object via the admin interface. A project with “lizard” in the title. Normally nothing is removed in this app, so the problem didn’t surface earlier.

Haystack’s results include the deleted now-non-existing object, which crashes the template. Which is a little bit weird, as normally django’s template layer is quite forgiving about None. It might be the {% highlight %} templatetag that I’m using.

Anyway, the solution is to run the update_index command with the --remove option. Or run rebuild_index. The second replaces the index, but is quicker. The first takes more resources, but leaves the index in place. Take whatever fits your usecase best.

Macbook pro black screen and restart problem


Tags: apple, python

Note: updated instructions (as he did it again).

This morning I took my macbook pro (a very nice late 2014 retina 15” one) out of my backpack, opened it and plugged in the external monitor. But it stayed black. The macbook’s own screen also stayed black. Ouch? It didn’t seem to wake up...

Some standard things I checked/tried:

  • The battery couldn’t be empty yet. The apple logo on the back of the screen was illuminated, right? Of course I had already attached it to the power cord. Which showed its LED, so power wasn’t the problem.
  • Screen brightness. Perhaps I dialed it down to zero by accident? No.
  • Close/open again. Take out external display. Etc. Nope.

Ok, time to press the power button. No reaction that I could see/hear. Press it 10 seconds. No. Problem with such a new macbook is that it doesn’t have a harddisk. The noises a harddisk makes tend to provide pretty good feedback on what’s happening normally :-)

I just couldn’t get the macbook to reset (or shut down)! If the power button doesn’t work... Googling turned up two additional tricks:

  • Resetting the “SMC”, the system management controller. One of the things it does is monitor the power button... Press shift-option-control-power and release them all at the same time.

    Sure enough the LED indicator on the power cord switched from red to green and back again, so the SMC reset worked.

    Only... the computer still didn’t respond to the power button as far as I could notice.

  • Resetting the PRAM/NVRAM, some sort of special memory with, amongst others, video settings. Sounds good! Press command-option-p-r when your macbook boots. AAARGH! That’s the whole problem! The macbook refuses to listen to the power button, so it won’t restart. So I can’t get at a potential fix for the restart-problem because the computer won’t, well, restart.

I asked my mac-using colleague to help me. Different google searches sometimes help. And a double-check whether I had not missed anything obvious is always handy. He also tried pinging/ssh’ing my machine. Weird thing that we started to notice that there were brief periods where the mac was pingable! Did some of the power button reset attempts work, perhaps, despite not seeing/hearing any feedback?

Because I had removed the macbook from its regular stand and placed it directly on my desk I suddenly noticed that the logo illumination was off. Hey! Perhaps one of the resets did work. Despite not resulting in the regular “peep” from the speakers. Of course the problems I was having might have disrupted the speakers also.

Ok, if it is off, time to do the PRAM reset mentioned above. Power button, press command-option-p-r, PEEEEEP! Bingo! The startup sound was suddenly audible and lo and behold, I saw the login screen.

Back in business!

Now... my only guess as to the original cause is that I plugged in the external monitor too quickly after waking my macbook. Of course this isn’t something that should cause a mac any problems, but apparently it can. I’ll just have to wait a couple of seconds next time.

VERY glad that it works again...

House number (“naambordje”)


Tags: house

We wanted to have a plate or something with our name next to the front door. It is a handy service for people visiting us without being 100% sure about the house number. Same for the postal service.

My wife had the excellent idea to use a group photograph of a holiday for it. Every holiday we ask someone to take a photo of the four of us next to our fully laden bicycles.

So I took one of those photos last year and added our name and house number to it and stuck it in a plastic file folder and taped it shut. This year I used the plastification tool (which I bought for a different project half a year ago).

Now... how to stick it to the wall without too much effort? I thought about using a couple of screws and thick see-through plastic (plexiglass, perspex, lexan, whatever it is called). But a visit to the DIY shop turned up a much simpler solution: double-sided thick outdour adhesive tape! One package costs about 10 Euro, but it’ll last me 10 years, too.

Just make a new photo every year and stick it to the wall with the double sided tape. You’ve got something new once a year and the photo stays reasonably OK. It’ll last a year without too much dimming of the colors.

A cheap and not too elaborate solution to have something original next to your front door!

Omnifocus folders: grouping projects


Tags: gtd

I’m currently updating my GTD setup. I use omnifocus for that.

One of the handy features is that you can group your projects in folders (and subfolders, should you be so inclined, which I’m not). I use it to bring a bit of order in my huge project list, as I have 50-100 projects, which is quite a normal number if you don’t make your projects too big.

I use a couple of targeted folders:

  • Big long-term projects, like building my model railway. I don’t want to turn that into one mega-project. Instead I have smaller specific projects for digitalizing a certain locomotive, figuring out the track plan for one of the stations, building a station building.

    Another example folder from a few years ago was “preparing the new house” with projects like “new window in the roof”, “install the kitchen”, “paint the staircase”.

  • Longer-term roles. Half a year ago, my four year period as my church’s secretary ended. So... that was a folder, too. A project per meeting, as 50% of my work was meeting-centric. Preparing the calender beforehand, gathering info, mailing people. Handling the tasks resulting from the meeting afterwards.

    Omnifocus made it easy for me to put something on my todo list for a meeting half a year in the future. I don’t think I ever forgot anything. GTD realy shone for me during this time! I’m not that organized and neat by nature. The fact that everybody was happy with how I did it is a testimony to GTD :-)

    The other projects in there were for specific projects like “clean out the archive”.

    Next year a big part of my work will be devops/sysadmin/cleanup/quality work. That’s a nice big role that deserves its own projects to drive it forwards.

The rest of the projects is grouped into:

  • Generic work.
  • Personal.

Splitting projects between the two, for me, is practical. As with any ordering, there are projects that are bound to fall in two categories at the same time. You’ll just have to make a judgment call for them. In my case, some open source projects are both a personal hobby and they’re use a lot at work. So sometimes I work on them in the evenings, sometimes during office hours. Ah well.

An additional comment about the targeted groups: they are great for focusing attention and ensuring progress. So I use them sparingly. Every year I try to pick three goals/areas upon which I want to focus. One or two of them invariably ends up as a folder. This just seems to demand projects and progress :-) logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):