They grew from a two-person company to a 70-person one in two years. Central to that growth were Django and google app engine.
Scalability means both load scalability and functional scalability. You also have to deal with organizational scalability and geographical scalability if you want to grow your organization.
It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it?
Their reasons to use it:
Auto-scaling. They build high-profile stuff and it needs to scale. They had a valentine day site that got a lot of attention on that day and it automatically scaled up without a change in the app. The day after it scaled down automatically, too.
Services and APIs.
No sysadmin needed.
Some caveats with app engine: it is a sandbox. You you cannot do “pip install”. The filesystem isn’t there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue.
They could work arounds these issues and ended up with a better application as a result.
There are three ways (that they use) of running Django on app engine:
Django non-rel. A ported version of Django, modified for NoSQL. Github, open source. It has a familiar API to Django, so you’ll feel at home. It works in production.
A drawback is that the familiarity can be misleading. So you might do things that won’t work like M2M relations. And it can feel heavy. Because of the fork/port, it might feel hacky.
Djappengine. A lightweight skeleton around app engine. You don’t use django’s models. It aims to be the best of both worlds. It also supports NDB, which is app engine’s new fast data storage layer.
Drawback: you need to learn a new database API, so you have a higher learning curve.
Django appengine + cloudSQL. You get a fully supported django.
Drawback: there’s more setup and it is probably not as scalable as a datastore.
Now to scalability. App engine will already do a lot for you. Some things you yourself must do:
Plan.
Cache the hell out of it.
Offline tasks out of the request loop.
Prepare load tests and do profiling.
Functional scaling provided by app engine (apart from what django provides):
Memcache
Taskqueue
Mapreduce
Search
Images
And you get up to 10 testable versions per app. http://0.yourapp,
http://1.yourapp (the previous version) and so on. You can do A/B testing and
traffic splitting. It blew his mind when he first discovered it.
Part of it is organizational culture:
Be a minimalist.
Removed bottlenecks and overhead. Don’t get in the way.
Just make good things. You can try (new) things out. You have freedom.
Internal apps. From a pool score app to steering deployments. They also have a big wiki with lots of info in it. It works well for them.
They also build a small Django app to handle all the incoming emailed job applications. One small app build in an afternoon on a beach in Thailand now helps them to hire better people more quickly :-)
You can work from everywhere. Plane, pub, train, at home, in an office, at a beach, whatever. The minimalism helps in scaling.
Important question: what if google shuts it down? Answer: for them, the advantages outweigh the risks. (Note: ouch, this shows what closing Reader did for google’s perceived reliability… Everyone in the room was applauding the question…)
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):
