Database

A definition: an SQL transaction is a sequence of SQL statements that is atomic with respect to recovery. In SQL 92, a transaction begins with a transaction-initiating statement (almost everything can start a transaction) and it ends with a commit, an explicit rollback (ROLLBACK) or an implicit rollback.

SQL 1999 changed this a bit. It has savepoints. After a savepoint, you can rollback to that savepoint, to a previous savepoint or you can set a new savepoint. Oh, and there is an explicit transaction start statement (START TRANSACTION).

Key findings:

• Statements always run in transactions.
• Transactions are opened automatically.
• Transactions are advanced technology.

Remember the dreaded "current transaction is aborted, commands ignored until end of transaction block" postgresql fault? What it actually means is "a previous statement failed, the application must perform a rollback". You cannot let postgres do any auto-recovery, that would break transactional integrity. It is your application that needs to do it (and it should always do it).

(I didn't hear what the actual solution is). Update: Diederik says in his comment that the solution is to just switch on autocommit for postgres in the database settings.

There's also AUTOCOMMIT. Most databases default to it. It commits every single statement automatically. Normally, you are either in auto-commit mode or inside transactions.

Interface: the python client libraries

Psycopg2 and sqlite3 are wrappers around C libraries. They use the DB API 2.0, PEP 249. It defines connections and cursors. Connections implement transactions, cursors do fetching and setting.

Note: the PEP wants the auto-commit to be off, initially!

Psycopg2 handles it by inserting a BEGIN before every statement, unless there's already a transaction in progress. Even for SELECTs.

Sqlite3 also inserts BEGIN, but not for a SELECT. All other statements get a COMMIT. Even a statement like SAVEPOINT: this is broken by design ("documentation issue").

Key findings:

• The DB API requires the same transactional behaviour as the SQL standard.
• Client libraries for databases that always autocommit have to emulate this behaviour.
• But you can turn it off and use autocommit

Framework (django)

Django 1.5 and earlier runs with an open transaction. For updates/deletes/saves, django does a commit. More or less auto-commit.

There's transaction middleware. One http request = one transaction. Commit on success, roll back on exception. It only works for the default database, though. And depending on the order of your middleware, it may or may not apply.

Django provides a couple of high-level APIs. with transaction.autocommit():, with transaction.commit_on_success():, with transaction.commit_manually():. There is also a low-level API for doing stuff manually.

Key findings:

• OK to forget it, it will change in 1.6.
• The middleware is a reasonable idea.
• The decorators/context managers don't work well, they often cannot be nested.

Django 1.6 uses database-level autocommit, which is what you'd normally expect. There are atomic transactions for requests: only for the view functions. Again. one transaction per http requests.

The high level API is now called atomic. Usable as a decorator and as a context manager. It can be safely nested.

Key learnings:

• django 1.6 will have sane transaction-related functionality.
• Read the documentation at https://docs.djangoproject.com/en/dev/topics/db/transactions/

html5lib

Browsers are terribly forgiving. Python's parsers don't deal with everything, even valid html5 docs. html5lib was a problem. Google code and so and not python 3 compatible.

The new html5lib supports python 3. Github, readthedocs, works fine!

See https://pypi.python.org/pypi/html5lib

Real time web - Aymeric Augustin

He looked at web sockets in django. He played with tulip, Guido's library for async python. He had 1000 processes calculating a 'game of life' screen and django connected with them just fine and pushed the result to the browser.

PyWaw

PyWaw is a python community in Warsaw. They have now had 24 meetings with about 55 attendees. At the last meeting they even had 100 people attending.

They are not alone in Poland, there are other user groups.

So... go back to your cities and start user groups!

Scrapy

Screen scraping is when you need to get structured information from the web, quickly and with no hassle.

Scrapy takes the hassle out of screen scaping. It takes away the pain of parsing horrible html.

It has perfect documentation and a helpful community.

You can even scrape from amazon, even including logging in.

What can you do? Convert SVG to VML. Stock checker for a market place. Testing your own website.

Motivating users - Aaron Bassett

How to motivate kids aged 7-17 to learn online. Don't give rewards. If you give rewards, that means that the task must be really shit. Rewards don't scale either. After initial success, do you increase the reward?

Everyone is addicted to dopamine, the stuff you get in your head when you like something. Don't give a reward always, because that turns play (which kids like) into work.

They did some tests: random rewards do seem to work. So that's something you can look at.

Better model inheritance - Craig de Stigter

There are three kinds of model inheritance now in Django:

• Proxy models.
• Abstract models.
• Multi-table.

None fit exactly with his usecase.

What he made was django-typed-models. A bit like proxy models, but it does store the type of the object in a type field, so you can figure out what you actually are.

They even use python magery for re-casting objects as a different type: self.__class__ = NewClass :-)

Django-fluent CMS - Diederik van der Boor

It is a CMS he build for his own CMS. Many CMSs are, in the end, monolythic.

He made a CMS that consists of separate parts. If you just want a tree of pages, use django-fluent-pages. If you just want an editable main part of a page, use another app. Etcetera.

See https://pypi.python.org/pypi/django-fluent-pages/, https://pypi.python.org/pypi/django-fluent-contents/

And you can also use django-fluent-dashboard, a more beautiful admin skin.

Update: he's got a website now: http://django-fluent.org/

Adventurer in the land of production environment - Maciej Pasternacki

Is your production enviroment up? Use a monitor like http://pingdom.com.

Django should not run as root. Run it with gunicorn and nginx, for instance.

Get immune to surprise upgrades: pip freeze.

Amulet of life saving: re-spawn when death with supervisord.

Stun immunity: a crontab with @reboot, for instance.

Acquire skill: chef, puppet.

The final battle: the slashdot effect. Gear up: autoscaling, self-healing.

New core committer

Marc Tamlyn is the new core committer!

Invisible and intentional management - Darin Swanson

Your project is not code, your project is your people. Make them happy. Make them do the best they can, no more, no less. Keep them leveling up.

Reward teamwork. It is not about the individual. Don't have individual goals, have team roles instead. Talk about "we" and "us". Lead by example. Help the team. Help everyone do better.

If you're a manager, try to be invisible. You're behind the scenes. Multiplying your impact behind the scenes. Don't take credit, the credit is for the team.

Move people to autonomy. Stay away from command and control. Set degrees of freedom and let people grow.

There's an implicit contract between you and your teammember: I'll give you freedom, you'll share status and information with me.

Discard what doesn't work, double down on what does. Especially regarding teamwork.

Do try to become better yourself, too. Find a mentor, read books, talk with others.

Relationships - Daniele Procida

There are 7E9 people in the world. Is your relationship really the best choice? Mathemetically not. Don't worry. Instead, commit to what you have already chosen and make it the best relation for you.

Same with web frameworks. There are so many... Stop worrying about making the wrong choice, stick with the one you have already chosen and make it the best for you.

Another subject: he wrote https://github.com/evildmp/django-inspector to report on all sorts of pages that his uses have added to his system. Status codes and so.

Arduino.loal

He hacked his landlord's garage door opener. They only had one and there were multiple people that needed to use it. So hack the opener, add an arduino and a webserver to control the garage door. They also added django-social-auth.

An enterprise level garage door opener!

SPDY - Emanuele Palazetti

How to deploy Django over SPDY. How to get that to work? Run django inside jython and thus inside java and SPDY push actually works.

3 simple ways to make your side load faster - Filip Wasilewski

• Database connection pooling. Creating a connection can take quite some time. Connection pooling will come in django core 1.6.
• Cache templates. Especially if you use something lik django-crispy-forms that uses lots of small templates. You only need to enable a template cacher in TEMPLATE_LOADERS in your settings.
• pjax. Push state ajax. That helps a lot.

Salt stack - Chris Reeves

He used to use Puppet, but didn't like the DSL. It was quite slow and wanted something better, stronger, faster.

They came accros Salt. Written in Python. Very fast. It is explicit, you control everything from the master, the clients don't call home themselves.

In your configuration templates you can use jinja2 for loops and so.

See http://docs.saltstack.com/

His verdict: it is consise and clean.

Your webpages are too big

Why should you care?

• Less developed countries.
• Mobile users.
• Overloaded wifi at django conferences.

What can you do?

• gzip compression on the server.
• django-htmlmin for html minification. It is still young and quite buggy at the moment.
• css/js minification. Look at django-pipeline.
• Do you need the full jquery? jquip has 90% of the functionality and 10% of the size. If you need the full version, use a CDN.
• Bootstrap css: don't hit "download" go to "customize" and make yourself a smaller version.
• https://github.com/samastur/image-diet to optimize your images. Works out of the box with easy-thumbnails.

Being a community member - Mark Steadman

He sucks at people stuff. Small groups are OK, but bigger groups are a problem. So that's hard when trying to integrate in a community, also the django community.

He works now on bambu-tools, a huge collection of small useful tools and components. But it needs work and fixes to make it useful for everyone.

Which is, see the first paragraph, hard for him. He'll be at the sprints and he'll do his best!

Django and vagrant and PyCharm

Three kinds of magic:

• Django is beautiful. There's magic inside, but it is beautiful magic.
• Vagrant is non-understandable magic.
• PyCharm was already magic in 2010 and it is even better now.

He now has something even better than magic. He has a miracle. He showed vagrant workin inside PyCharm. Looks quite nice. The debugger even works when the code runs inside the virtual machine.

Jukebox - loci

What to do when different people have different music styles, for instance at a party? Time for democracy. A website running locally on your laptop allows you to log in and vote for numbers. The highest-voted songs will be played first :-)

See https://github.com/lociii/jukebox

Classy class based views

Very handy when working with Django's class based views: http://ccbv.co.uk

(Note: I already used a link here in my summary of Russell's class based views talk. See http://ccbv.co.uk/projects/Django/1.5/django.views.generic.edit/UpdateView/ )

Ideas don't solve problems - Lukasc Balcerzak

His first computer came with logo, you could move the cursor with it to draw lines. Infinite possibilities, so no goal.

There are a lot of open source projects. Does it reinvent the wheel? Does it solve a relative simple problem? Those are two ways to rate projects on.

Example one:

• Just try reading a URL with Python. Which built-in library to use? Hard.
• The "requests" library is a small library that solves one relatively simple problem.

Example two:

• Django-guardian extends Django's auth and has shortcuts for basic stuff. Much simpler.
• Django's auth itself is quite elaborate and hard.

Testing class based views - Benoît Chesneau

You can use django.test.client, but that is an integration test. All the middleware and so is used.

For unittests, you can use a request factory. You still test the system, the callable.

We can also do focused unittesting. We can mimic as_view():

view.request = request
view.args = args
views.kwargs = kwargs

With this, you can test your code much more focused. And you gain speed!

Further reading: http://tech.novapost.fr/django-unit-test-your-views-en.html

Django client certificates - Deni Bertovic

Why would you use client SSL certificates? Isn't user/passwd enough?

The advantage: nginx takes care of authentication.

See https://github.com/denibertovic/django-client-certificates

Arduino - Swift

Normally you have to code in C. But now you can also do it in Python.

See https://github.com/theycallmeswift/BreakfastSerial

He demo'ed it. Very nice! Looks useful and usable and simple. Perfect.

Next conference - Remco Wendt

This is now the fifth year. We have a tradition now! High quality conferences organized for programmers by programmers. Not for profit. Great! And now the fun factor is there, too.

The fun will stay! Next year it'll be France, on the beach, in the south! (They don't know the exact city yet).

Dicts

This works now:

>>> from collections import defaultdict
>>> data = defaultdict(list)
>>> data['key'].add(42)

It was added in python 2.5. Previously you'd do a manual check whether the key exists and create it if it misses.

Sets

Sets are very useful. Sets contain unique values. Lookups are fast. Before you'd use a dictionary:

>>> d = {}
>>> for each in list_of_things:
...     d[each] = None
>>> list_of_things = d.keys()

Now you'd use:

>>> list_of_things = set(list_of_things)

Sorting

You don't need to turn a set into a list before sorting it. This works:

>>> something = set(...)
>>> nicely_sorted = sorted(something)

Previously you'd do some_list.sort() and then turn it into a set.

Sorting with cmp

This one is old::

>>> def compare(x, y):
...     return cmp(x.something, y.something)
>>> sorted(xxxx, cmp=compare)

New is to use a key. That gets you one call per item. The comparison function takes two items, so you get a whole lot of calls. Here's the new:

>>> def get_key(x):
...     return x.something
>>> sorted(xxxx, key=get_key)

Conditional expressions

This one is very common!

This old one is hard to debug if blank_choice also evaluates to None:

>>> first_choice = include_blank and blank_choice or []

There's a new syntax for conditional expressions:

>>> first_choice = blank_choice if include_blank else []

Constants and loops

Put constant calculations outside of the loop:

>>> const = 5 * a_var
>>> result = 0
>>> for each in some_iterable:
...     result += each * const

Someone suggested this as an old-dated pattern. You can put it inside the loop, python will detect that and work just as fast. He tried it out and it turns out to depend a lot on the kind of calculation, so just stick with the above example.

String concatenation

Which of these is faster:

>>> ''.join(['some', 'string'])
>>> 'some' + 'string'

It turns out that the first one, that most of us use because it is apparently faster, is actually slower! So just use +.

Where does that join come from then? Here. This is slow:

>>> result = ''
>>> for text in make_lots_of_tests():
...     result += text

And this is fast:

>>> result = ''.join(make_lots_of_tests())

The reason is that in the first example, the result text is copied in memory over and over again.

So: use .join() only for joining lists. This also means that you effectively do what looks good. Nobody will concatenate lots of separate strings over several lines in their source code. You'd just use a list there. For just a few strings, just concatenate them.

That's the nice thing of Python: if you do what looks good, you're mostly ok.

1: Running Django on app engine

It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it?

Their reasons to use it:

• Auto-scaling. They build high-profile stuff and it needs to scale. They had a valentine day site that got a lot of attention on that day and it automatically scaled up without a change in the app. The day after it scaled down automatically, too.
• Services and APIs.
• No sysadmin needed.

Some caveats with app engine: it is a sandbox. You you cannot do "pip install". The filesystem isn't there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue.

They could work arounds these issues and ended up with a better application as a result.

There are three ways (that they use) of running Django on app engine:

• Django non-rel. A ported version of Django, modified for NoSQL. Github, open source. It has a familiar API to Django, so you'll feel at home. It works in production.

  A drawback is that the familiarity can be misleading. So you might do things that won't work like M2M relations. And it can feel heavy. Because of the fork/port, it might feel hacky.

• Djappengine. A lightweight skeleton around app engine. You don't use django's models. It aims to be the best of both worlds. It also supports NDB, which is app engine's new fast data storage layer.

  Drawback: you need to learn a new database API, so you have a higher learning curve.

• Django appengine + cloudSQL. You get a fully supported django.

  Drawback: there's more setup and it is probably not as scalable as a datastore.

Now to scalability. App engine will already do a lot for you. Some things you yourself must do:

• Plan.
• Cache the hell out of it.
• Offline tasks out of the request loop.
• Prepare load tests and do profiling.

Functional scaling provided by app engine (apart from what django provides):

• Memcache
• Taskqueue
• Mapreduce
• Search
• Email
• Images

And you get up to 10 testable versions per app. http://0.yourapp, http://1.yourapp (the previous version) and so on. You can do A/B testing and traffic splitting. It blew his mind when he first discovered it.

2: Scaling an organization + culture

Part of it is organizational culture:

• Be a minimalist.

• Removed bottlenecks and overhead. Don't get in the way.

• Just make good things. You can try (new) things out. You have freedom.

• Internal apps. From a pool score app to steering deployments. They also have a big wiki with lots of info in it. It works well for them.

  They also build a small Django app to handle all the incoming emailed job applications. One small app build in an afternoon on a beach in Thailand now helps them to hire better people more quickly :-)

You can work from everywhere. Plane, pub, train, at home, in an office, at a beach, whatever. The minimalism helps in scaling.

Important question: what if google shuts it down? Answer: for them, the advantages outweigh the risks. (Note: ouch, this shows what closing Reader did for google's perceived reliability... Everyone in the room was applauding the question...)