Bitbucket, lessons learned - Jesper Noehr (djangocon.eu)¶

Tags: djangocon, django

Jesper Noehr talks about the well-known bitbucket. Bitbucket is written in Django! He started with django four years ago (the “magic removal branch”).

Bitbucket started in may 2008 and launched 1 month after that. That short turn-around time was thanks to django. For two years they worked on it with two people. It ran entirely on Amazon web services. Last year it got bought by Atlassian. After being acquired they got a couple of java programmers on the team that got into python and django real quick: that’s how good python and django is.

He used mercurial when writing his django blog app. He also wanted to integrate mercurial’s basic http viewer into his blog. That’s the basis for bitbucket!

74k lines of code, all python. 86% is django. Of course they use mercurial. And postgresql and virtualenv/pip. Some of the production tools: HAProxy, nginx, gunicorn, puppet. Deployments are mostly weekly. For testing: selenium for functional tests, kong for headless tests and bamboo for continuous integration.

Some “lessons learned”:

Stay idiomatic. “Idiomatic” means something like “using native things”. So stick with django as much as possible. Everything they have is just a regular model. Sometimes there’s a mixin class to get some extra features, but it is still a django model. And everthing is splitted up in django applications.

Benefits. You get free upgrades and there’s little backporting. Getting and finding help is often easy as you’re just using the standard tools. An awesome community. And an advantage is that it is easy for new people to pick up.

Disadvantages. It sometimes feels like you have to make what you want fit what django needs. They had to do some crazy request-mangling django middleware to get their custom-url-per-user working. And they’re unusual in that they’re quite transactional: things happen on disk and in the database and they have to be in one transaction, really.

But… sticking to idioms is usual better.

Things change. Even the most beautiful thing in the world that you made may have to be changed. It might have been a good idea at the time. But the requirements change or the amount of users goes way up or whatever. They had to change several database parts to redis, for instance.

Likewise running django on fastcgi was fine at the start. Then they switched to mod_wsgi, which helped a lot. Till even that wasn’t up to the task. Now they’re using gunicorn behind ngnix and they’re happy with it. Gunicorn’s author is very responsive.

So: things are going to change. Embrace it! Change is a good thing. It means you’re evolving and growing. But watch out for vendor lock-in. They had this with amazon. They figured out something that would run fine on amazon, but it would mean vendor lock-in when they would ever move away from amazon. So they stuck to simple filesystem stuff. Which helped a lot as they actually moved away from amazon. So assuming you’ll change things is valuable.

Embrace technologies that let you replace and change.

Open source. There’s a lot of useful code out there. South and celery come to mind. They used their own home-brewn user registration system, but now they just use django registration and they’re happier.

They also gave back to the community, like django-piston (“yes, I’ll make a release soon”), django-dogslow and tipper.

Be transparent and humble. Their customers are super smart and need to be treated that way. They really had to learn that. When there’s an outage now, they give out full details and people like it, for instance.