2013-05-17
Browsers are terribly forgiving. Python’s parsers don’t deal with everything, even valid html5 docs. html5lib was a problem. Google code and so and not python 3 compatible.
The new html5lib supports python 3. Github, readthedocs, works fine!
He looked at web sockets in django. He played with tulip, Guido’s library for async python. He had 1000 processes calculating a ‘game of life’ screen and django connected with them just fine and pushed the result to the browser.
PyWaw is a python community in Warsaw. They have now had 24 meetings with about 55 attendees. At the last meeting they even had 100 people attending.
They are not alone in Poland, there are other user groups.
So... go back to your cities and start user groups!
Screen scraping is when you need to get structured information from the web, quickly and with no hassle.
Scrapy takes the hassle out of screen scaping. It takes away the pain of parsing horrible html.
It has perfect documentation and a helpful community.
You can even scrape from amazon, even including logging in.
What can you do? Convert SVG to VML. Stock checker for a market place. Testing your own website.
How to motivate kids aged 7-17 to learn online. Don’t give rewards. If you give rewards, that means that the task must be really shit. Rewards don’t scale either. After initial success, do you increase the reward?
Everyone is addicted to dopamine, the stuff you get in your head when you like something. Don’t give a reward always, because that turns play (which kids like) into work.
They did some tests: random rewards do seem to work. So that’s something you can look at.
There are three kinds of model inheritance now in Django:
None fit exactly with his usecase.
What he made was django-typed-models. A bit like proxy models, but it does store the type of the object in a type field, so you can figure out what you actually are.
They even use python magery for re-casting objects as a different type: self.__class__ = NewClass :-)
It is a CMS he build for his own CMS. Many CMSs are, in the end, monolythic.
He made a CMS that consists of separate parts. If you just want a tree of pages, use django-fluent-pages. If you just want an editable main part of a page, use another app. Etcetera.
See https://pypi.python.org/pypi/django-fluent-pages/, https://pypi.python.org/pypi/django-fluent-contents/
And you can also use django-fluent-dashboard, a more beautiful admin skin.
Update: he’s got a website now: http://django-fluent.org/
Is your production enviroment up? Use a monitor like http://pingdom.com.
Django should not run as root. Run it with gunicorn and nginx, for instance.
Get immune to surprise upgrades: pip freeze.
Amulet of life saving: re-spawn when death with supervisord.
Stun immunity: a crontab with @reboot, for instance.
Acquire skill: chef, puppet.
The final battle: the slashdot effect. Gear up: autoscaling, self-healing.
Marc Tamlyn is the new core committer!
Your project is not code, your project is your people. Make them happy. Make them do the best they can, no more, no less. Keep them leveling up.
Reward teamwork. It is not about the individual. Don’t have individual goals, have team roles instead. Talk about “we” and “us”. Lead by example. Help the team. Help everyone do better.
If you’re a manager, try to be invisible. You’re behind the scenes. Multiplying your impact behind the scenes. Don’t take credit, the credit is for the team.
Move people to autonomy. Stay away from command and control. Set degrees of freedom and let people grow.
There’s an implicit contract between you and your teammember: I’ll give you freedom, you’ll share status and information with me.
Discard what doesn’t work, double down on what does. Especially regarding teamwork.
Do try to become better yourself, too. Find a mentor, read books, talk with others.
There are 7E9 people in the world. Is your relationship really the best choice? Mathemetically not. Don’t worry. Instead, commit to what you have already chosen and make it the best relation for you.
Same with web frameworks. There are so many... Stop worrying about making the wrong choice, stick with the one you have already chosen and make it the best for you.
Another subject: he wrote https://github.com/evildmp/django-inspector to report on all sorts of pages that his uses have added to his system. Status codes and so.
He hacked his landlord’s garage door opener. They only had one and there were multiple people that needed to use it. So hack the opener, add an arduino and a webserver to control the garage door. They also added django-social-auth.
An enterprise level garage door opener!
How to deploy Django over SPDY. How to get that to work? Run django inside jython and thus inside java and SPDY push actually works.
He used to use Puppet, but didn’t like the DSL. It was quite slow and wanted something better, stronger, faster.
They came accros Salt. Written in Python. Very fast. It is explicit, you control everything from the master, the clients don’t call home themselves.
In your configuration templates you can use jinja2 for loops and so.
See http://docs.saltstack.com/
His verdict: it is consise and clean.
Why should you care?
What can you do?
He sucks at people stuff. Small groups are OK, but bigger groups are a problem. So that’s hard when trying to integrate in a community, also the django community.
He works now on bambu-tools, a huge collection of small useful tools and components. But it needs work and fixes to make it useful for everyone.
Which is, see the first paragraph, hard for him. He’ll be at the sprints and he’ll do his best!
Three kinds of magic:
He now has something even better than magic. He has a miracle. He showed vagrant workin inside PyCharm. Looks quite nice. The debugger even works when the code runs inside the virtual machine.
What to do when different people have different music styles, for instance at a party? Time for democracy. A website running locally on your laptop allows you to log in and vote for numbers. The highest-voted songs will be played first :-)
Very handy when working with Django’s class based views: http://ccbv.co.uk
(Note: I already used a link here in my summary of Russell’s class based views talk. See http://ccbv.co.uk/projects/Django/1.5/django.views.generic.edit/UpdateView/ )
His first computer came with logo, you could move the cursor with it to draw lines. Infinite possibilities, so no goal.
There are a lot of open source projects. Does it reinvent the wheel? Does it solve a relative simple problem? Those are two ways to rate projects on.
Example one:
Example two:
You can use django.test.client, but that is an integration test. All the middleware and so is used.
For unittests, you can use a request factory. You still test the system, the callable.
We can also do focused unittesting. We can mimic as_view():
view.request = request
view.args = args
views.kwargs = kwargs
With this, you can test your code much more focused. And you gain speed!
Further reading: http://tech.novapost.fr/django-unit-test-your-views-en.html
Why would you use client SSL certificates? Isn’t user/passwd enough?
The advantage: nginx takes care of authentication.
See https://github.com/denibertovic/django-client-certificates
Normally you have to code in C. But now you can also do it in Python.
See https://github.com/theycallmeswift/BreakfastSerial
He demo’ed it. Very nice! Looks useful and usable and simple. Perfect.
This is now the fifth year. We have a tradition now! High quality conferences organized for programmers by programmers. Not for profit. Great! And now the fun factor is there, too.
The fun will stay! Next year it’ll be France, on the beach, in the south! (They don’t know the exact city yet).
2013-05-17
This works now:
>>> from collections import defaultdict
>>> data = defaultdict(list)
>>> data['key'].add(42)
It was added in python 2.5. Previously you’d do a manual check whether the key exists and create it if it misses.
Sets are very useful. Sets contain unique values. Lookups are fast. Before you’d use a dictionary:
>>> d = {}
>>> for each in list_of_things:
... d[each] = None
>>> list_of_things = d.keys()
Now you’d use:
>>> list_of_things = set(list_of_things)
You don’t need to turn a set into a list before sorting it. This works:
>>> something = set(...)
>>> nicely_sorted = sorted(something)
Previously you’d do some_list.sort() and then turn it into a set.
This one is old::
>>> def compare(x, y):
... return cmp(x.something, y.something)
>>> sorted(xxxx, cmp=compare)
New is to use a key. That gets you one call per item. The comparison function takes two items, so you get a whole lot of calls. Here’s the new:
>>> def get_key(x):
... return x.something
>>> sorted(xxxx, key=get_key)
This one is very common!
This old one is hard to debug if blank_choice also evaluates to None:
>>> first_choice = include_blank and blank_choice or []
There’s a new syntax for conditional expressions:
>>> first_choice = blank_choice if include_blank else []
Put constant calculations outside of the loop:
>>> const = 5 * a_var
>>> result = 0
>>> for each in some_iterable:
... result += each * const
Someone suggested this as an old-dated pattern. You can put it inside the loop, python will detect that and work just as fast. He tried it out and it turns out to depend a lot on the kind of calculation, so just stick with the above example.
Which of these is faster:
>>> ''.join(['some', 'string'])
>>> 'some' + 'string'
It turns out that the first one, that most of us use because it is apparently faster, is actually slower! So just use +.
Where does that join come from then? Here. This is slow:
>>> result = ''
>>> for text in make_lots_of_tests():
... result += text
And this is fast:
>>> result = ''.join(make_lots_of_tests())
The reason is that in the first example, the result text is copied in memory over and over again.
So: use .join() only for joining lists. This also means that you effectively do what looks good. Nobody will concatenate lots of separate strings over several lines in their source code. You’d just use a list there. For just a few strings, just concatenate them.
That’s the nice thing of Python: if you do what looks good, you’re mostly ok.
2013-05-17
The classical approach in django is:
This means that models are pretty much static. There is no way to modify them at runtime based on user interactions. You can get something working with for instance hstore in postgresql (see the postgresql talk).
His usecase is for medical forms. The contents of those forms should be able to be defined inside the system. There are strict processes for installing medical software, so you cannot just release a new version with a new field. So you must get it to work at runtime.
The solution could be to use dynamic models, models created at runtime. Sometimes configuration by subject matter experts is better than code customization by developers. Also, dynamic models reduce the number of deployment cycles.
He has some criteria:
There are a couple of possible solutions:
In the end, the runtime schema updates approach looks like the best one.
For more reading: https://code.djangoproject.com/wiki/DynamicModels
2013-05-17
They grew from a two-person company to a 70-person one in two years. Central to that growth were Django and google app engine.
Scalability means both load scalability and functional scalability. You also have to deal with organizational scalability and geographical scalability if you want to grow your organization.
It is easy to get confused. Is app engine real? Is it a joke? How to run your django stuff on it?
Their reasons to use it:
Some caveats with app engine: it is a sandbox. You you cannot do “pip install”. The filesystem isn’t there in the traditional sense; there is a blob storage instead. And it is lock-in, mostly; portability is an issue.
They could work arounds these issues and ended up with a better application as a result.
There are three ways (that they use) of running Django on app engine:
Django non-rel. A ported version of Django, modified for NoSQL. Github, open source. It has a familiar API to Django, so you’ll feel at home. It works in production.
A drawback is that the familiarity can be misleading. So you might do things that won’t work like M2M relations. And it can feel heavy. Because of the fork/port, it might feel hacky.
Djappengine. A lightweight skeleton around app engine. You don’t use django’s models. It aims to be the best of both worlds. It also supports NDB, which is app engine’s new fast data storage layer.
Drawback: you need to learn a new database API, so you have a higher learning curve.
Django appengine + cloudSQL. You get a fully supported django.
Drawback: there’s more setup and it is probably not as scalable as a datastore.
Now to scalability. App engine will already do a lot for you. Some things you yourself must do:
Functional scaling provided by app engine (apart from what django provides):
And you get up to 10 testable versions per app. http://0.yourapp, http://1.yourapp (the previous version) and so on. You can do A/B testing and traffic splitting. It blew his mind when he first discovered it.
Part of it is organizational culture:
Be a minimalist.
Removed bottlenecks and overhead. Don’t get in the way.
Just make good things. You can try (new) things out. You have freedom.
Internal apps. From a pool score app to steering deployments. They also have a big wiki with lots of info in it. It works well for them.
They also build a small Django app to handle all the incoming emailed job applications. One small app build in an afternoon on a beach in Thailand now helps them to hire better people more quickly :-)
You can work from everywhere. Plane, pub, train, at home, in an office, at a beach, whatever. The minimalism helps in scaling.
Important question: what if google shuts it down? Answer: for them, the advantages outweigh the risks. (Note: ouch, this shows what closing Reader did for google’s perceived reliability... Everyone in the room was applauding the question...)
2013-05-17
If you’ve got continuous deployment, you’ve got stable servers. You make big changes in small increments.
Continuous deployment forces you to do many good things:
Lots of good things. But let’s compare it with lion taming.
Originally, lions were beaten into submission, confused and kept in line with whips. Likewise you’ll be beaten if you dare to touch the production machine as it might break.
Now lions are understood better. Conditioning, behavior/signal mapping, reward and trust are the methods now. We understand that deployment is hard. We have behaviour/signal mapping with code/test/green/deploy. Etc.
Continuous deployment: everyone is responsible. Everyone deploys. You automatically learn. Everybody uses the same environment locally for test deployments. The same as on the server.
Testing is core. Slow tests are killing. Fast tests. And all types of tests: unit, functional and acceptance tests. Also automatic code checkers. The light must stay green. Quality must stay high, also test quality.
You need a repeatable build. And it should include not just code, but also configuration and infrastructure. And... always follow the pipeline.
Even in emergencies, follow the pipeline. Peer review, tests, and then the deployment. Don’t do manual steps.
Rollback. You must be able to switch back to the previous version.
You can take a canary approach. Canary in the coalmine. Show the new version to a few users. “User testing” in a sense.
Rollbacks in databases; keep it backwards compatible. Never drop columns, for instance. (After a long time, you can remove them safely, of course).
Small changes. Frequent releases mean less risk: if something breaks, you know where to search.
Some tips:
Split your stuff in components. A component is something that has a good API and that can be switched out for a different component. It can also be separately deployed.
This helps with testing, too.
Rehearse releases. Get very good at them!
You need good infrastructure. You must manage it and test it good.
Keep all environments equal. Use vagrant.
Automate everything! And if it is not possible: document it. But know that that’s something that’s not quite correct yet!
2013-05-17
Initially he didn’t know a lot about transactions, so he researched them in depth. A quote by Christophe Pettus: “transaction management tools are often made to seem like a black art”.
He moves from the database (postgres and sqlite) to the interface (psycopg2 and sqlite3) to the framework (django).
A definition: an SQL transaction is a sequence of SQL statements that is atomic with respect to recovery. In SQL 92, a transaction begins with a transaction-initiating statement (almost everything can start a transaction) and it ends with a commit, an explicit rollback (ROLLBACK) or an implicit rollback.
SQL 1999 changed this a bit. It has savepoints. After a savepoint, you can rollback to that savepoint, to a previous savepoint or you can set a new savepoint. Oh, and there is an explicit transaction start statement (START TRANSACTION).
Key findings:
Remember the dreaded “current transaction is aborted, commands ignored until end of transaction block” postgresql fault? What it actually means is “a previous statement failed, the application must perform a rollback”. You cannot let postgres do any auto-recovery, that would break transactional integrity. It is your application that needs to do it (and it should always do it).
(I didn’t hear what the actual solution is).
There’s also AUTOCOMMIT. Most databases default to it. It commits every single statement automatically. Normally, you are either in auto-commit mode or inside transactions.
Psycopg2 and sqlite3 are wrappers around C libraries. They use the DB API 2.0, PEP 249. It defines connections and cursors. Connections implement transactions, cursors do fetching and setting.
Note: the PEP wants the auto-commit to be off, initially!
Psycopg2 handles it by inserting a BEGIN before every statement, unless there’s already a transaction in progress. Even for SELECTs.
Sqlite3 also inserts BEGIN, but not for a SELECT. All other statements get a COMMIT. Even a statement like SAVEPOINT: this is broken by design (“documentation issue”).
Key findings:
Django 1.5 and earlier runs with an open transaction. For updates/deletes/saves, django does a commit. More or less auto-commit.
There’s transaction middleware. One http request = one transaction. Commit on success, roll back on exception. It only works for the default database, though. And depending on the order of your middleware, it may or may not apply.
Django provides a couple of high-level APIs. with transaction.autocommit():, with transaction.commit_on_success():, with transaction.commit_manually():. There is also a low-level API for doing stuff manually.
Key findings:
Django 1.6 uses database-level autocommit, which is what you’d normally expect. There are atomic transactions for requests: only for the view functions. Again. one transaction per http requests.
The high level API is now called atomic. Usable as a decorator and as a context manager. It can be safely nested.
Key learnings:
2013-05-17
Principle philosophy: a way to discuss our rules and beliefs that govern our actions. He tells it from his personal experience.
His parents wanted to raise him as a good person. So they thought him good principles (like don’t be a quitter, don’t steal, etc). This is quite black/white though. We are all more gray/gray.
What about the question “how can I be a good programmer”? Programmers use logic, which sounds black/white again: write tests, don’t repeat yourself. Sigh.
Talking about things like this is impossible without Immanuel Kant. He differentiates between reason and instinct. If “be happy” were our life goal, we’d just follow our instincts. So what is reason for, then, apart for doing good? Reason has to do with moral. There are three ways of looking at “doing good”:
Does this help with a question like “is testing good”?
Gandhi said that a man is the sum of his actions.
In a sense we are the sum of our experiences. So increase the amount of experience that you have. Either have the experiences yourself, or share them like on this conference. Everything looks different from the trenches: learn from eachother.
Some lessons he learned from a little baseball league experience (where he sucked) as a kid:
Some take-aways:
2013-05-17
Russell is a Django core dev.
Class based views were introduced two years ago, but they weren’t greeted with universal acclaim. So he’s here to clear up the mess and hopefully make it all more clear for everyone.
In the beginning of Django, there were only views. Function-based views. No generic views.
Next, because of DRY, don’t repeat yourself, several generic views were added. Listing objects, editing an object, for instance. Editing something happens so often that a generic view inside Django seemed like a good idea.
There are some problems here, though. The configuration you can do is limited by the arguments you can give in your URL configuration. No control over the logic view. You can’t pass in an alternative view. There’s no re-use between views.
You could “fix” this by adding more and more arguments and allow passing in callables and so, but in the end you’re almost building what you’d already get with object oriented class inheritance... So...
Next: class based views. It landed in Django 1.3 after it didn’t work out to get it in 1.1 or 1.2.
Then the wheels fell off. What went wrong?
Fundamental confusion over purpose. There were two problems being solved at the same time. The two: class based views and class based generic views.
Class based views are only a class-based variant on function-based views that handles get/post/put/delete. Classed based views will give you a lot for free. Automatic OPTIONS requests handling. And naive HEAD handling. You wouldn’t have that with a function based view. And you can modify it.
Class based generic views use class based views as a base. They’re re-writes of the existing function-based generic views. But a bit better and especially much more extensible.
Confusion over implementation choices. The reasons were good, but the reasons weren’t clear.
The whole discussion and the choices behind it can be found in the django wiki.
The biggest question is about instantiation. What is being instantiated? How? When? At the start, once, or for every single request? How do you pass in configuration? What’s the lifespan of an instance? Can you safely assign something onto self? What are the expectations?
Note: the admin was already always class based. And it had state problems (assigning to self would leak state to other requests).
In the end, all this was what resulted in the MyView.as_view(). as_view() results in a class factory. Otherwise they’d have to change the urls.py contract. A view is currently a callable. It would have to be changed to “a callable or a class”. It was a value judgment in the end.
Ravioli code. It wasn’t spagghetti code, but ravioli. A package with unknown contents.
The generic class based views are made with a whole bunch of mixin classes. The edit view (UpdateView) consists of 9 (mixin) classes. See ccbv.co.uk.
Why would you go through this 9-level madness? Yes, we have a complex class hierarchy. But the reason is that you can easily customize it.
Ravioli tastes good! Maximum reuse of core logic. Extremely flexible. Easy to add your own functionality. But you need to learn it, that is the price you pay for the power you get. Learning means documentation, so...
Bad documentation. The initial documentation was bad. It is now better, but it needs to be made better still.
The biggest thing that needs fixing after the documentation is how to handle decorators like @login_required.
But... did we solve the right problems with the generic views? Modern websites have different problems. Multiple forms. Conditional forms. Continuous scrolling instead of pagination. AJAX support. PJAX (see yesterday’s ajax+django talk). Multiple “actions” per page.
In discusions, always make sure you whether you mean CBV or CBGV (class based views or class based generic views).
Suggestion made later during the questions: call the latter just “generic views”. The old function based generic views are gone, so...
Docs still can be improved.
Experiment with APIs. Django’s admin is a useful case study. Why not do that with an API and make it easier to create your custom admin?
2013-05-17
She works for Red Hat on http://freeipa.org, on identity stuff for Linux.
Note: see her website for instructions and code examples.
Say that your pointy haired boss (or customer) asks you to make an internal web app with all the buzzwords.
So you can’t use regular django auth, you’ll need single sign on. Luckily since Django 1.5 you can have custom user models, so it’ll fit with all your external requirements. One or two pieces of MIDDLEWARE_CLASSES and AUTHENTICATION_BACKENDS later and you play nice with the external single sign on. Django can be a team player.
Webserver? You’ll probably have to use apache. So the environment can be kerberos+apache. Add mod_auth_kerb for kerberos support. Add a “keytab” (making sure it is chown’ed to apache).
There’s a difference between authentication and authorization. Authentication is “just” logging in, authorization is what you’re allowed to do. You’ll have to connect to LDAP for that to ask which group(s) the user is a member of.
Setting up your own kerberos environment (for testing) is a pain. Unless you use a ready made vagrant box for it. Instructions are on her website.
2013-05-17
Django conferences have a tradition: there’s an external luminary that gets to give a critical talk on Django. His talk won’t be that. He’s not external either: he wrote two scoops of Django together with Audrey (see also my review).
Being critical is sometimes easy. Just bash class based views, for instance. Bashing is easy. A rant like Zed Shaw’s is fun, but he’s not asked because of his rants, but because of his contributions (like books).
Similarly, Django delivers working stuff and that working stuff makes a lot of our work possible. So here are some good points about Django:
Django is everywhere. So many people and companies use it.
Django is powered by Python. Pep8, python is beautiful. And there’s the import this zen of python that we use all the time to steer others and ourselves in the right direction.
Django’s API wins. It is understandable. No weird names: templates, views, logging, sessions. Django projects also have understandable structures. If there’s no views.py or models.py or templates/ directory, you know someone messed something up.
Fat models are great. Just put your business logic all on your models. They do get big this way, however. You can make a separate module with helper functions you call in your model. Just call it with a model attribute. This way you get a reasonably small models.py and another file with very testable small functions. Win!
The API is clear and logical. We’re not fighting about architecture, we are getting things done.
Django views are simple callables. (Even class based views).
Django is awesome at deprecation. Code often just keeps working fine for multiple Django versions.
Django has lots of features. For instance, Django’s admin is awesome. This is what you use to sell Django to others.
Tip: don’t try to use the admin with a nosql database. Just build something from scratch, that’s easier than trying to get it working and especially keeping it working.
Django’s full stack is awesome. Real projects are being done in unextended python+django. No third party packages. Not everyone goes to conferences and not everyone knows everything on pypi. You can get a lot done with just Django (though external packages help a lot!).
Django is also python. There are over 30000 packages! Even if only 20%
Documentation. Django set the bar for others.
Truth be told, some other projects have better documentation now. Django set the bar and others followed. They’re playing by our rules :-)
Django is humble. We have a tradition of invited critical talks. They shape the community, they shape the core committers.
Here are two critical talk summaries I have:
Good criticism is good. They got a lot on their book. That was hard, but the book is much better for it.
The django community is generous. There’s almost an unwritten rule: “the more you help people in the Django/Python community, the more the community helps you”.
They recieved a lot of help with their book. They also helped others with free books in case they couldn’t pay for it. But that required an email to ask for it. Note that Daniel and Audrey did ask for something in return: either buy the book later, donate to charity or help someone.
It worked! Somone gave a free guitar course to someone else. Someone bought a homeless person a dinner. People did projects for schools/churches/whatever. Contributions to open source software.
It worked! People did good work! Lots of small local positive actions. All over the world.
Call to action: be awesome. Make the world a better place.
Statistics: charts of posts per year and per month.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):