![[Valid Atom]](http://www.atomenabled.org/feedvalidator/images/valid-atom.png "Validate my Atom feed"){kind=small}
This is an Atom formatted XML site feed. It is intended to be viewed in a Newsreader or syndicated to another site. Please visit Atom Enabled for more info.
This is an Atom formatted XML site feed. It is intended to be viewed in a Newsreader or syndicated to another site. Please visit Atom Enabled for more info.
In rare cases, you can import a file that imports the file you're importing from. This might sound a bit recursive, and it is. In Python it is called a circular import error.
The best way to recognize a circular import that goes wrong is to look at your results. You will get an ImportError message from Django, like cannot import some_view from your_project.views. When you look in your_project/views.py you see that some_view really exists in that very same file. Huh? That wasn't expected.
This huh??? is exactly what a circular import looks like. Something that exists (you checked it five times at least) doesn't seem to exist. A computer doesn't lie, but you start to wonder.
The problem is, Python cannot complete the import, so it complains about the point where it goes wrong, even though the actual error is the circular import loop as a whole.
The solution is to break the loop somewhere. Perhaps the thing you want to import is best placed somewhere else? Most often, a circular import error indicates an organization problem in your code.
AOP is Aspect Oriented Programming. It is not often used in Python, you see it more often in for instance Java.
His program does calculations on biogas installations. And the calculation needs to be exposed through a web API (using Django).
He thought: what about aspects? For instance the security aspect? Statistics/logging aspect? The serialization aspect? All several aspects of the API. He showed a bit of sample code where all these aspects were all inside the same single Django view method. Looked like quite a mixed-up mess (as was his intention).
The resulting code is both scattered and tangled. Scattered because one aspect is scattered all over the place in your code. Tangled because different aspects are often interweaved/mixed.
You can use AOP to fix this. Now, how do we implement this in pure Python?. The number one candidate is decorators.
AOP deals with pointcuts, join points and advices. Advices can be pre, post and around ("a wrapper"). You can use three decorators for these three cases. In the end his example looked a bit like this:
@secure
@serialize
@statistics
@dispatch
def api_call(...):
...
AOP offers some brilliant concepts for software engineering. Separate your concerns and aspects; avoid tangling and scattering!
In response to a question: no, this is not a framework, these are just regular decorators. But he used the ideas of AOP to design his decorators.
I also gave a talk at http://pygrunn.nl on development laptop setup: explicit automation. I don't have a summary of my own talk, but I have two other things for you:
Later on I'll try to document my dotfiles, but you can already find a lot of ideas and tips online for bash settings, for instance.
Sisyphus, which they made at paylogic, is a semi-distributed job scheduling system.
Originally they used cronjobs. It is very robust and reliable. But it has limitations:
So... they wanted something new! Sisyphus. The features:
The system depends on a central SQL database, so that part is not redundant. But that central database is the most critical part of Paylogic, so if the main database goes down, jobs that don't run is hardly a priority :-) So this dependency is fine.
Sisyphus is divided into separate parts, like scheduler, dispatcher, configurator, etc.
Nice detail: sisyphus is actually run as a cronjob on all the active hosts (polling for jobs). If a job is available, a "guardian" claimes a process using a database lock. The job is run in a subprocess. The guardian keeps track of the job and takes care of the bookkeeping.
The current state: they are using it in production at paylogic. There are still some rough edges that need attention, however. It is not open source yet, but it will be.
django-crispy-forms is a Django application, but Miguel thinks the talk will help you also with designing other systems and applications.
Django has three ways to render forms: as_ul, as_p, as_table. They do the same, but render themselves in a different way. Common questions by people new to Django is "what about divs?" and "how to reorder fields?". For the last one, you need to switch the order of the form fields in your python form code. There are some other tricks like overriding the self.fields.keyOrder attribute. If you have many fields, regular list methods like delete() and pop() and insert() might help you.
But... ModelForms for the admin interface are different again: the abovementioned form tricks don't work there. And those tricks sound a bit dirty anyway. So: how to customize the form output?
You can do a lot in by customizing the Django form in the template, but most of it will be hardcoded and hand-tweaked that way. And if you customize a form, you'll often forget form.errors and form.non_field_errors, for instance.
Django-crispy-forms was formerly known as django-uni-forms. It was created by pydanny in 2008, Miguel is now one of the main committers.
Crispy forms work on forms, modelforms and on formsets. A |crispy filter in the template renders your form as handy divs with better classes and IDs which helps a lot if you want to customize your form with css. Neat!
Crispy also has a {% crispy %} template tag. You can pass it a "form helper". A form helper is a global helper: it is decoupled from forms. So it normally works with any form. There are some attributes on the form helper that you can set, like method (get/post), form_id, things like that.
You can customize such a helper specifically for one form and set the order of the fields, for instance. There are so many things you might want to customize; crispy supports/allows most/all of them with a Layout class and other layout classes like Div. You can get really deep into the machinery by letting crispy inject Django template code directly into the template...
For the ultimate in customizability, you can write your own layout class that renders itself in whatever way you want. Layouts can be nested, so there is a lot of flexibility here. You can also customize crispy's own templates that it uses for fields and forms using the regular Django overwrite-a-template mechanism.
Handy: crispy forms has specific support for twitter bootstrap. This helps you get a nice looking form.
In the GIS world, everything used to be proprietary (ESRI, Oracle), but there is a lot of commoditization in the last years. Lots of open source. One of those open source pieces is geodjango.
Geodjango is bundled with Django, but out of the box you miss a couple of pieces. You need to install a couple of extra libraries (gdal, geos, proj) and you need a geospatial database (postgis, oracle, mysql, spatialite).
Ivor guided us through a sample application. Things like setting a gis database instead of a standard one. Adding django.contrib.gis to the INSTALLED_APPS setting. And special geometry fields for points, lines, polygons. Using a specific OSMGeoAdmin for showing a map in the admin interface for those geo fields.
A limitation in geodjango is that it doesn't give you regular form fields for the geo fields. They work in the admin, but not in regular forms. Luckily django-floppyforms does provide them, so he used floppyforms to get nice forms including a map in his regular web interface. Creating geojson from database content and show that in the map.
(Note to self: look at proj4js).
Geodjango is well-integrated into Django, but you do need to use the geodjango variants of fields, databases, admins. "You need to prefix the stuff". You get a lot out of the box, but there's quite a learning curve. You also need to learn quite some javascript for the user interface.
According to Armin Ronacher most (Python) web frameworks use a request/response style of handling HTTP. At his company, they're treating HTTP a litle bit different. (So the talk is first about some HTTP-usage-in-Python observations and second a look at the alternative way they're treating HTTP).
Note: my brother has a clearer summary, btw.
The most low-level way is to write directly to the response. Write the response headers, write the actual response content. In Python, you often have some response object; often some sort of middleware gets the chance to do something to the response on the way out.
The nice things we like about HTTP:
A basic question you should ask yourself is why does my application look like HTTP? A common Django application gets a request, does something and returns a response. Works well. But why is it set up that way? Why is it so focused on HTTP? (It is logical that it focuses on this use case, but you can still ask the question).
HTTP can be a stream or buffered. Sending stuff from the server to your browser is a stream. But often an incoming request in a Python web framework is first buffered internally (memory or disk). In the same way a request is a bit of a strange mix:
On the client (like your webbrowser) you cannot do anything to an incoming request, once it started, is to close the connection. You cannot interact anymore once you received your first incoming byte.
A consequence of the buffering and the way HTTP is handled is that you can have problems accepting data. How big a file should you accept? How big an incoming form? Buffer it in memory? Or on disk? And how do you handle streaming? You might be streaming in one part of your code, but how do the other layers handle it?
Internally in his company, he's trying to handle HTTP differently. There's no direct HTTP contact in most of the code base. Everything that eventually ends up in the HTTP layer is implemented as some sort of "type object". This allowed them to really flexible in the HTTP layer. Support for different input/output format. Easier to test. Documentation can be auto-generated. Lots of common errors can be catched early.
A basic rule is to be strict in what you send, but generous in what you receive. But web Python code is often generous by "just" accepting a lot without much checking. That might be a security risk. In Armin's system, you know what type should be coming in, so you can do proper checking.
How does this deal with the big-upload problem? Incoming streaming data? Well, because of the type system, you actually know which types need a streaming API. This makes it easy to set up your API correctly. You can even selectively use a different protocol than HTTP.
He had to show lots of markers on a google map, which often led to bad performance. And he wanted real-time updates. And... the client needs to be responsive!
A marker is a DOM element, but also a shadow, a click area, an icon image and so on. Which takes a lot of memory and rendering it takes lots of CPU.
How to optimize? One option is to reuse an existing marker image if they are the same. If they are different, using an image sprite helps a lot. You have to set offsets and sizes on the markers to get the sprite to work.
The lazy way to draw lots of markers is to just draw all of them, regardless of the viewpoint. Most of them will probably not even be visible... Looping over so many DOM elements is slow.
There are some weapons you can use to speed it up: the idle (zooming/panning is done) and zoom_changed google map events. For this, underscore.js helps with functions like defer and with handy collection types. Similarly backbone.js helps maintaining collections and binding them to events and to an API on the server.
Backbone trick. Backbone has no real handy way to remove items from a collection that are old, you can only add or reset a collection completely. He did it by adding a version number to the data he fetched from the server. If, after an update, items have a lower version number, they're old and you can delete them. This way he could refresh a backbone collection without needing to zap it first and rebuilding it from scratch.
Now, what about the webserver? It should be simple, fast and scalable for his usecase. He went for tornado. Tornado is quick. And it simplifies asynchonous code.
The database should be JSON friendly, support geo queries and it should have an async Python binding. MongoDB has it. You can do queries like {'loc': {'$within': {'$box': box}, }}. For the Python binding he used pymongo.
He used google's fusion tables for zoomed out levels: they group markers together so that you only see an indication of the number of items instead of showing all the markers. (Reinout: but somehow he didn't do this, at least I didn't see it in his demo. I probably misunderstood something.)
His talk is titled "space invaders, relational modeling and domain models, a mashup" instead of just the plain sqlalchemy that I expected :-)
He started his computer career with a TRS-80 in 1980. The killer app at that time (5th grade school) was printing out a picture of Snoopy on the matrix printer.
Next up an Atari 800. There was a great 'vertical scroller' game and he wanted to build his own. But you needed to write assembly to pull it off which was way over his head. Machine level programming is depressing.
Somewhere in the 90 was, for him, the time of architecture astronauts and enterprise java beans. 1200 database tables for reading a big SGML document, that sort of nightmare. He was made to use the Pure waterfall method. Spending a month only drawing Rational Rose UML diagrams. Also depressing.
But he did finally understand objects and liked them. A domain model is important.
He showed wordpress' comment edit code. PHP code; hacked often. Hardcoded ad-hoc SQL code to delete comments, embedded right in the PHP code. Comments are only an integer ID, not real comment objects, of course. In the end, he showed the entire PHP page in 3px font size: HTML, SQL and PHP all mixed up in one file. Horrid.
You basically give an integer ID through a couple of calls all the way through the application. Only valid for small scripts, not really for a system the size of wordpress. Can we call it the water slide model? One straight flow down from an ID to some action.
A domain model is "a conceptual model of a domain of interest", according to wikipedia. You model the problem, agnostic of implementation details. And we create explicit adaptions between the outside world and the model (adapters, serializers, views, etc: separation of concerns). And ideally we only use neat normalized data.
To him, dealing with such a domain model feels a bit like a horizontal pipeline. (Reinout: not sure what he means with this).
At the core of relational databases is the relational model. Information represented as collections of rows, consisting of columns. Mostly ACID.
Essential to relational databases are joins: intersecting or joining data. SQL, the language, is a declarative language (as opposed to imperative or functional, see stackoverflow).
How do we reconcile the SQL/relational model with the domain model? He things an ORM, object relational mapper, is a good idea. An ORM needs to map objects and collections to the tables, rows and columns of the database, including relations.
Keeping the "R is for relation" intact is a real challenge. There are several ORMs that don't expose enough of the available relations. Sqlalchemy does expose it, you can keep thinking in terms of your database and its tables and relations that you know.
Another challenge is to limit the amount of queries. For instance, when you iterate over some set of database objects (rows) in Python, there is a big risk that you do a follow-up query for every row. So you need to be able to give the ORM some instructions for loading that extra related data directly. Sqlalchemy helps you with this: this way you get the luxury of a nice ORM, but you still have well-optimized SQL.
Mashup? Yep, he wrote a space invaders clone that stores all the game data in an SQL database (sqlite, in-memory)! Idiotic example, but loads of fun.
The domain model includes missile, player, splat, enemy, army and saucer glyphs. Every glyph has a coordinate.
The code has a main loop with update_state() that moves the missiles, player, etcetera and draw() that draws everything. Both query the database! Are there coordination overlaps between missiles and a saucer? Boom! You can finally get an explosion out of a database query. And drawing means querying for all coordiates and rendering what's there. Wow.
Losing is also determined with an SQL query: an enemy coordinate matches the player coordinate. Or an enemy coordinate matches the bottom row coordinate :-)
Then he started playing the game in one screen with a log file showing all the SQL queries live in another screen. Fun!
Today, 4 May, is day against DRM.
The publisher I'm writing my Django book for, the pragmatic bookshelf, is fully DRM free. Lovely. You get your books in a number of formats (I use epub and pdf) without any restriction. Well, they generate the book with your name tucked away somewhere in the header so you cannot post it on some FTP site and get away with it :-)
A nice feature they added a couple of months ago: dropbox integration. Any book you buy is automatically added to your dropbox account! No need to keep things in sync yourself.
So... I don't need to worry that my eBook purchases disappear into some DRM-locked no-longer-supported device. Make sure your books are DRM free, too!