Reinout van Rees’ weblog

Django debugtoolbar for site running inside virtualbox/vagrant

2014-04-10

Tags: django, apple

I had a hard time getting the Django debug toolbar to work this morning. Complicating factor: the site runs inside a vagrant virtualbox and my browser is simply running inside OSX.

(I use vagrant and virtualbox, see why I use vagrant and my vagrant setup on OSX)

Ok, I thought I had installed everything correctly. What can be the problem? I checked and double checked. The best way to verify your settings is to use django’s own “diffsettings” management command, that way you’re sure you’re looking at the right settings:

$ bin/django diffsettings
...
DEBUG = True
INSTALLED_APPS = ['debug_toolbar', 'lizard5_site', ... ]
INTERNAL_IPS = ('33.33.33.20', '127.0.0.1', '0.0.0.0')
...
MIDDLEWARE_CLASSES = ('debug_toolbar.middleware.DebugToolbarMiddleware', ...)
...

Debug mode is on, it is in the INSTALLED_APPS and I’ve correctly enabled the middleware. Oh, and I’ve adjusted the INTERNAL_IPS setting.

  • 127.0.0.1 even though I don’t really talk to the site from within my vagrant box.
  • 0.0.0.0 as that’s what I start my runserver with inside vagrant (bin/django runserver 0.0.0.0:8000): listen on everything on port 8000.
  • 33.33.33.20 as that’s the default IP address that vagrant/virtualbox gives me to talk to the virtualbox from within OSX. So on my laptop you’ll normally see http://33.33.33.20:8000/ instead of http://localhost:8000/ :-)

The 0.0.0.0 shouldn’t be needed, but at that time I was just trying things out to no avail.

... Time to call in the artillery. We have the source code, so I looked up the debug toolbar version I was using and put an import pdb;pdb.set_trace() into the def show_toolbar(request) method inside debug_toolbar/middleware.py.

The first lines of that short function are:

if request.META.get('REMOTE_ADDR', None) not in settings.INTERNAL_IPS:
    return False

In went the pdb and I reloaded the homepage:

> /.../django_debug_toolbar-1.0.1-py2.7.egg/debug_toolbar/middleware.py(26)show_toolbar()
-> if request.META.get('REMOTE_ADDR', None) not in settings.INTERNAL_IPS:
(Pdb) request.META['REMOTE_ADDR']
'33.33.33.1'

Sure enough, the error was in my INTERNAL_IPS setting after all. REMOTE_ADDR on the request turned out to be 33.33.33.1! Even though I talk to it from OSX with 33.33.33.20. So there must be some internal virtualbox/vagrant trickery that does some mapping here.

So: if you use the django debug toolbar inside vagrant: make sure you’ve got the correct port in the INTERNAL_IPS setting!

Print(reinout) for debugging

2014-03-31

Tags: python

Quite some colleagues use my name to raise an error. It is a trick I learned them :-)

There are a large number of debugging tips and tricks. import pdb; print statements; a full-blown IDE with an interactive debugger; logging; etc.

Often the simplest and most useful thing to check is to first verify that you’re looking in the right place at all. Why? Well, you might be working on the right python file, but are you really sure that your web app isn’t using a previously installed package instead of your checked out developer version, for instance?

Or you work on some method, but another class overrides the method you’re debugging.

In cases like this I often add a print(reinout) statement somewhere in the function. That’s not a typo. It definitively should not be print("reinout"). The latter only prints reinout, the first prints:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'reinout' is not defined

Much easier to spot. Makes it obvious that at least you’re looking in the right spot!

Why build it at all (TRS series part 1)

2014-03-19

Tags: nelenschuurmans, django, trs

(“TRS”? “Time registration system”! An internal company website for managing our projects and for booking hours. I’ll use this blog series to simply tell you how I build it. With topics such as “bundling javascript”, “python 3” and “letting test coverage slip”.)

Ok, a time registration system. A bunch of Person objects in a database; projects you can book hours on; probably some assignment of persons to projects; a bit of reporting. Why build it yourself?!? There are 2538 existing ones! On the other hand, it is relatively simple, so it can’t hurt too much to build it yourself.

In the end, the assumption was that there are enough special cases to make it worthwhile.

  • We’re a project organisation. Everything revolves around projects. All the work we do is within projects, most of them external (which means: paid by a customer). So we want to distinguish between internal and external. And which ones are subsidized projects, as they have some special rules. And some, like the internal “vacation” project, shouldn’t count towards some of the totals. Etc.
  • As projects are the core, it makes sense to craft a beautiful, well-working customized app instead of taking some works-reasonable-but-only-that existing open source project. Something that costs money is out of the picture (even if building it internally costs 10x the money, but that’s one of our weirdnesses).
  • We had an existing system: we needed to get the data out and into the new system. Getting it out of the old one is relatively straightforward (“requests” library, “beautifulsoup” and “import csv”). Getting it into the new system is way easier if that’s your own system, obviously :-)
  • Coupling with our own user database (active directory).
  • Custom metrics. Hours booked external versus internal (but not counting the vacation hours). Percentage of hours booked outside of the allotted time budget. Your turnover (hours per project times your tariff on those projects) this year. With a couple of special cases/rules in there, of course.

In the end, I was the one making the new system almost single-handedly. Fun! I had a personal reason to particulary enjoy building it: it was a relatively straightforward Django app. Lots of Python coding, quite some thinkwork to set up the data model, designing the user interface. All things I like.

Now, why was this particularly enjoyable? Well, as I was quite stressed at the time. I recognized it and took a little bit of time off, spending some afternoons cycling. And, to quote from a previous blog post:

And at about the same time I started an all-new internal project: a relatively straightforward internal Django timekeeping app. Lots of work and lots of tweaking, but at least a stressless productive programming assignment. Just coming up with a good architecture, a good css layout, some nice and proper code and yeah, I was happy again.

So... “therapeutic programming”? :-)

Ontraxs 2014 model railway exhibition

2014-03-09

Tags: trains

On friday I took a day off from work to go to the Dutch railway museum, just a short 30 minutes cycling from my home. The reason: the 2014 edition (the 6th) of the “Ontraxs” model railway exhibition.

They only invite beautiful layouts with a good landscape, which sets them apart from the other big exhibitions that have lower standards. It also means there are fewer big layouts with lots of trains, so if you like that more: go to the other exhibitions. Or just visit them all :-)

Anyway, I liked the exhibition. Everything was well made, but of course several were more to my taste than others. I’ve made a video (15 minutes) showing the nicest layouts.

Additionaly, I made a 3 minute one of the layout I personally liked best. The landscape was nice, but I especially liked the realistic operation they showed. Real train traffic that really seemed to serve a purpose:

Recovering from stress

2014-02-18

Tags: nelenschuurmans, python

Last year I laid all sorts of nice plans (in my yearly personal develoment plan) but it all got laid to waste by a full change of (company) direction. Poof went my plans. Plans for my own direction, plans for teamwork, plans for lots of improvements. That’s life. And such a problematic year gives its own opportunities for learning and improvement.

As an example of such an opportunity: somewhere in october I was quite stressed.

I didn’t agree with the direction we were taking. It was starting to take its toll on my health (mostly my back muscles acting up). Now... how to get out of it? My recipe was to tell it to my direct boss and to get his permission to take a couple of afternoons off to cycle a bit. Of course I got the permission (that’s the good thing about where I work!) and spend some 6 afternoons cycling within two weeks time.

That really helped clear my mind (and my body). And at about the same time I started an all-new internal project: a relatively straightforward internal Django timekeeping app. Lots of work and lots of tweaking, but at least a stressless productive programming assignment. Just coming up with a good architecture, a good css layout, some nice and proper code and yeah, I was happy again.

So... I think the lesson for me is that giving early feedback is important. And the company earned some bonus points for actually being happy with me for telling about my problems and coming up myself with a simple way to fix/reduce it.

Yearly development moment

2014-02-18

Tags: nelenschuurmans, python

The company where I work, Nelen & Schuurmans asks you to fill in a “personal development plan” (in Dutch persoonlijk ontwikkelings plan, POP) every year at the start of the year.

I like filling it in every year. Like every company-wide form, there are problems with it, but I like the idea and what it gives me.

Others don’t like it. A colleague wrote a blog post about it which is interesting if you’re into personal development. His core argument, if I summarize it correctly, is that personal development should be a continuous process. And anything that hints at time limits, like a yearly form does, is inherintly bad as it restricts you in your development.

The second problem he mentions is that it is a personal development plan. Where is the team in that? Where is the company in that?

I myself am, in contrast, quite positive about the yearly form. Thinking about it, I guess the main reason is that I’ve got a different attitude towards the yearly development plan.

  • I agree that personal development should be continuous. I always develop myself continuously, a form won’t change that positively or negatively.

  • It is essential for me to step back a bit from the continuous development once in a while. What am I learning? What is my direction? What am I ignoring? I’ve got limited time, so a bit of focus is needed. A yearly development plan is a nice trigger, a nice moment, to take a higher-level view. I do it more often than once a year, but this way I’ve at least got a yearly trigger.

  • It is a form I fill in for the company. So I naturally think about the company and the team. What is the direction we should take? What is my opinion? What should we focus on? I then translate that to what I could do.

  • Something I really need to put in more effort and attention: REST APIs. Consuming them and building them. So I’ve put that forth as a focus for me this year. (At the same time this sneaks in the suggestion that it’d be a good focus for the company.)

  • Hey, a form where the company asks me how it can help me to develop. In other words, I can give them a shopping list. I’ll develop myself just fine, but, yes, there are things the company can help me with:

    • I’d love to attend djangocon.eu again this year. I’ll learn a lot there and meet people and sharpen my understanding and opinions. And as I make blog summaries of all the talks, the rest of the team profits. (So: allow me to go and pay for it).

    • This year I want to improve my generic geographical IT knowledge by attending the German-spoken fossgis conference in Berlin (a cheap one: 140 Euro for three days. Add a hotel and a train ticket and you’re done). I attended once before and as a team we got for instance the useful mapproxy out of that one. That one sure saved us a lot of money in the Deltaportaal project!

    • There are a bunch of backend systems and unglamorous necessary behind-the-scenes technology at our company. Our internal sentry instance hasn’t been updated in a year. Who is going to get it going again? Last year, buildout moved to version 2 and at the same time the distribute/setuptools unfork happened. I spend a bunch of time making sure it all kept working for us. Not many have the interest or inclination to do this kind of work and I like it.

      So... in the interest of the company and the team and in the interest of sharpening my own skills, I’d like to regularly spend time on it. So I’ve requested a weekly time budget! Some of that time goes to regular maintentance work, but, and that’s my motivation: some of the hours I spend are going to save the team even more hours of work! Some of the work I do makes others’ work easier. A bit extra attention to a README.rst can mean the difference between “blech, how does this work” and “hey, I can easily fix up this debian package myself”.

  • Hey, the “top brass” is going to read this document. So I take the opportunity to state some preferences and my opinion on the direction and make a comment on one of the company policies. A form like this is the perfect opportunity :-)

Now does it work? Such a form? Well, it works in that it got me standing in front of my whiteboard at home from 21:00 till 3:00 in the night... Thinking deeply about my job :-) And that’s a good thing: doing some explicit thinking!

Chrome takes up a shocking amount of space on OSX (updated)

2014-02-13

Tags: django

I normally use Firefox as my webbrowser, but sometimes it is handy to have a second browser. Especially when I’m doing permission work in a Django website: certain users should see more than others. Having two browsers open at the same time, both with different logged-in (or anonymous) users is handy then.

So I’ve got chrome. Which is set to auto-update itself with newer versions.

As I was running out of disk space, I investigated my program folder a bit for stuff I could throw out. Suddenly I spotted that chrome took up more than 16GB of space. What? I right-clicked the icon and selected “show package contents” and started screaming:

http://reinout.vanrees.org/images/2014/chrome_16gb.png

Turns out the bloody idiotic program kept every version of itself for the last 1.5 years... 16GB! And I cannot find a setting anywhere that prohibits this behaviour. In the end I just deleted the olderer versions, which worked fine.

So...

  • Check your /Applications/ folder on OSX when you’re using chrome.
  • Does someone know how to switch this brainless behaviour off?

Update: yes, someone knows it. Chris Adams has identified the problem in the comments below. The DivX plugin has mucked about with permissions, preventing the chrome updater from working correctly...

Social media course - Theo Zijderveld

2014-01-20

Tags: nelenschuurmans

The company I work for (Nelen & Schuurmans) organized a course on social media.

People don’t trust commercials anymore. Social media and people are trusted more. For a company, having people active on social media is very important: they’re the external voice of the company that actually gets listened to.

We did a three-part exercise:

  • Part 1: find your story A story consists of a number of events that get sewn together. It results in some sort of meaning. You’ve got patterns that come back. Heroes or villains.

    So we got several questions to help us write several things down about ourselves and our history and our dreams.

  • Part 2: tell a story We got to sit together with another colleague and got 5 minutes each to tell eachother about our past, our now and our dreams for the future. This was the basis for part 3.

  • Part 3: translate it to social media Watch out: it is no one way street, you need interaction. And it is not lineair. And it is spread out over many different kinds of media.

    He showed some examples of Maersk, a shipping and oil digging company. Shipping containers from A to B might seem dull, but they make it exciting and personal and fun to follow.

    Another example: Rijkswaterstaat makes the work of their employees visible through twitter. Or they at least encourage it and retweet the individual employees’ tweets.

    External work-related social media visibility at the very least means you need to have a proper linkedin account. So this was the exercise: creating a linkedin summary for the colleague you just interviewed :-)

Some generic tips on social media:

  • At least take care of a good photo. Make your face visible and recognizable. No weird unclear holiday pictures. This helps you when people google your name.

  • On social media it is fine to show what you’re doing outside of work. In fact, those combinations are great. Don’t show yourself drunk at a weird party, but dead tired after some sport event is fine. Show your interests like music or cycling. Showing what you’re interested in gives others openings to connect to you when they meet you (also professionally).

  • Facebook is generally much more private-oriented. You can have work-related contacts there, though, especially for international contacts. So... perhaps take the opportunity.

  • Why blog? Good content-rich specialized articles help a lot and attract readers. By blogging you also sharpen your own understanding: the teacher learns the most. You get visibility and slowly take on an expert’s aura.

    Theo also blogs: niet voor communicatie. (I also have a blog, which is what you’re reading here... :-) )

    Things that help: humor, a look behind the scenes, positivism, nostalgia, authenticity.

Your own story is worth gold!

  • Don’t give up.
  • Ask for feedback and don’t be afraid of it.
  • At least create a proper linkedin account.

UI/UX course with all our software developers

2013-11-18

Tags: django, nelenschuurmans

Last week the company I work at (Nelen & Schuurmans) organized a one-day course for all the software developers. Nice initiative! Tom Stevens of Namahn.com lead the day.

By law of nature, I make notes during such a day. I didn’t, however, make a full summary; so I can only give an impression with a couple of loose remarks and quotes and ideas.

Anyway, the day was bits of theory interspersed with group exercises, culminating in a design that we tested with paper prototypes. To start off I’ve got a video of those tests. (If you don’t understand Dutch: scroll through to the 1 minute mark. That way you at least get a nice impression on how a paper prototype can work.)

Mental model

A core idea is mental modeling. What is the mental model of the user? What mental model should he have in order to use your app or website? What mental model of reality does the user already have? Which terms or concepts should you use in your app or site?

Interviews are important to get this clear. And if you interview or if you observe someone: listen. Listen a lot. At least initially. You are the student, the user is the teacher. Take time to gain trust. And when you do ask questions, try to get to the bottom. The user might say something, but after asking “why” five times, digging ever deeper, you might get a different answer than originally. Alternatively, use the why/where/what/when/who kind of questions.

Why are mental models important? Well, you make them anyway when interacting. Enter a building and you see the door knob: you automatically make a mental model of that door knob and the way you expect it to behave when you interact with it. Should I push or pull or whatever?

So when your site or application has a good and consistent mental model behind it, it will provide you and your user with something to hold on to: it helps you explain what happens and it helps you understand what happened and what is about to happen when you interact with it.

A very good requirement you can force upon yourself when you make a mental model: you should be able to explain it to your grandma within two minutes.

Gaining clarity: scenarios and terminology

Tell stories. Human beings like and relate to stories. So write down textual usage scenarios (or visual: simple comic strip). Use specific thought-up “personas” like “Harry from accounting who positively hates computers”. And try to get together a representative number of usage scenarios.

Those scenarios, when written down, help get discussion underway. If you only tell what you’re going to do, it is easy to nod your head and to agree. If you see something written down that’s obviously wrong or obviously missing key elements... Discussion! This way you gain clarity quickly. And... you gain clarity before spending four weeks programming something!

After gaining clarity in this way, the next step is classification. Basically: find the right words. Nouns and verbs. The terminology you use for your application. Spend time getting this right.

Btw, make a difference between information structure and application structure. The information structure means the content. “Article”, “blog post”, “map layer”. So: the core data structure. The application structure, on the other hand, is much more about about the form/layout.

(Note: he mentioned the elements of user experience by Garrett (link to PDF with the main graph). The way I understand it now as I can actually read the graph is that the information structure/application structure difference is more of a “duality” that needs to be resolved in the resulting visual design.)

Conceptual design

Conceptual design is the next phase: design the main structure of your website. Items like:

  • Sketches.
  • Interface grid.
  • Wireframes.
  • Session management and login behaviour.
  • How will we handle search/filter/browsing?
  • Messages, feedback.
  • Etc.

So basically: converting the output of the classification/terminology phase to a navigation. Tip: make wireframes or mock-ups of the various screens and draw the navigation (the flow) between those wireframes. He called it a wireflow.

Apparently designing interfaces by Jenifer Tidwell is “the bible” for stuff like this.

When designing, use four Gestalt principles: proximity, similarity, continuity and closure. (The page linked has five items, btw).

And keep in mind that too many choices mean that a task takes too much time (Hick’s law), so don’t put in too many choices.

Closing

Nice day! I hope it helps us (and me) to realize and remember that some advance planning and designing helps save time and money and effort.

Note: a colleague also blogged about this day.

Why I work where I work

2013-10-15

Tags: nelenschuurmans, personal, python

FastCompany’s Jeff Chu recently visited Nelen & Schuurmans (where I work) for an article about Dutch water management. The article is now online: a nice one.

As scientists predict a wetter, stormier future for much of the planet, the Dutch have become a nationwide consulting company, fanning across the world to talk about water.

I’m reasonably well-known as a Python developer (and zope/plone/django). So what am I doing here in a water company? Nelen & Schuurmans is one of those water consultancy companies mentioned in the article. The reason for my presence is also right there in the article:

“The Netherlands is very conservative. We think we can solve everything with a bit of sand,” says Jan-Maarten Verbree, who heads IT at Nelen & Schuurmans, the water-focused consultancy behind 3Di. “But we say that with a bit of IT, you can increase safety even more.”

IT and consultancy go hand-in-hand for us. Which gives nice results as the company is really smart: almost everyone has a university degree (with a couple of PhDs thrown in). Trying novel approaches. Combining ideas. Combining fields of work. Combining fields of experience.

Personally, I’ve got this combination right here inside myself. I’m a Python developer. And by education I’m a civil engineer. Two side notes:

  • I did do every IT-related course inside the study that I could. And my graduation subject and PhD was a combination of IT and civil engineering.
  • The most directly applicable course related to my current work would be the standard second year water hydraulics course. Which was the absolutely dead last course I did, just a few weeks before graduating :-)

So by education and experience (and importantly: interest!) I fit right in. There’s a lot of freedom that I also enjoy a lot. And we’re profitable, which is quite a feat in the current marketplace (and quite good for not worrying). A funny quote in the article points at one of the reasons for the profitability: we don’t spend a lot of money on unnecessarily decorative offices.

In the Nelen & Schuurmans lab, a Spartan collection of desks located in a house in the medieval heart of Utrecht, ...

We are however dead smack right in the center of Utrecht, which is a lovely place to have an office!

Note: as I’m spending quite some time muttering right now (over a technical decision), truthfulness forces me to say that not everything is roses and sunshine. But that’s something for a separate entry (probably on how I try to deal with it, I do have to re-gain my positive energy).

Anyway, I’ve got nice work. So let me finish with a video I made for demo purposes. I’d also like to point at another quick video, a more 3d-like visualization. Nice!

 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):