Does your stuff scale? - Steven Holmes¶

1: Running Django on app engine¶

It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it?

Their reasons to use it:

• Auto-scaling. They build high-profile stuff and it needs to scale. They had a valentine day site that got a lot of attention on that day and it automatically scaled up without a change in the app. The day after it scaled down automatically, too.

• Services and APIs.

• No sysadmin needed.

Some caveats with app engine: it is a sandbox. You you cannot do “pip install”. The filesystem isn’t there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue.

They could work arounds these issues and ended up with a better application as a result.

There are three ways (that they use) of running Django on app engine:

• Django non-rel. A ported version of Django, modified for NoSQL. Github, open source. It has a familiar API to Django, so you’ll feel at home. It works in production.

  A drawback is that the familiarity can be misleading. So you might do things that won’t work like M2M relations. And it can feel heavy. Because of the fork/port, it might feel hacky.

• Djappengine. A lightweight skeleton around app engine. You don’t use django’s models. It aims to be the best of both worlds. It also supports NDB, which is app engine’s new fast data storage layer.

  Drawback: you need to learn a new database API, so you have a higher learning curve.

• Django appengine + cloudSQL. You get a fully supported django.

  Drawback: there’s more setup and it is probably not as scalable as a datastore.

Now to scalability. App engine will already do a lot for you. Some things you yourself must do:

• Plan.

• Cache the hell out of it.

• Offline tasks out of the request loop.

• Prepare load tests and do profiling.

Functional scaling provided by app engine (apart from what django provides):

• Memcache

• Taskqueue

• Mapreduce

• Search

• Email

• Images

And you get up to 10 testable versions per app. http://0.yourapp, http://1.yourapp (the previous version) and so on. You can do A/B testing and traffic splitting. It blew his mind when he first discovered it.

2: Scaling an organization + culture¶

Part of it is organizational culture:

• Be a minimalist.

• Removed bottlenecks and overhead. Don’t get in the way.

• Just make good things. You can try (new) things out. You have freedom.

• Internal apps. From a pool score app to steering deployments. They also have a big wiki with lots of info in it. It works well for them.

  They also build a small Django app to handle all the incoming emailed job applications. One small app build in an afternoon on a beach in Thailand now helps them to hire better people more quickly :-)

You can work from everywhere. Plane, pub, train, at home, in an office, at a beach, whatever. The minimalism helps in scaling.

Important question: what if google shuts it down? Answer: for them, the advantages outweigh the risks. (Note: ouch, this shows what closing Reader did for google’s perceived reliability… Everyone in the room was applauding the question…)