2012-02-09
I updated subversion yesterday (because I installed git-svn). Subversion version 1.7 has a new repository structure and requires you to upgrade all your existing checkouts:
$> svn up
svn: E155036: Please see the 'svn upgrade' command
svn: E155036: Working copy 'xyz' is too old (format 10, created by Subversion 1.6)
Calling svn upgrade by hand for all checkouts is boring. And I made checkoutmanager to make checkouts less boring. A simple checkoutmanager up in the morning does an svn up, git pull or hg pull -u in every one of my checkouts.
And my brother luckily added a hidden command upgrade to checkoutmanager. It is called hidden because it is only listed on the pypi page, not when you call checkoutmanager --help, because it is so rarely needed.
But anyway, a simple checkoutmanager upgrade call later and all my svn checkouts were upgraded! Nice.
2012-02-06
At work, connecting to one of our three windows network shares with OSX took a long time. 1.5 minute or so. Irritating. I tried several solutions, but none of them worked. Connecting from linux or windows wasn’t a problem.
With last week’s OSX 10.7.3 update, the problems are gone! A connection is now made in 6 seconds or so. Way better than 1.5 minute.
There are some notes on the update page like “address compatibility issues with Microsoft Windows file sharing” and “improve binding and login speed for Active Directory users in a domain whose name ends in .local”, so one of those probably solved it.
2012-02-05
My year-old macbook installation was showing its age. Or rather, there were some things wrong with it:
So I made sure my backups were OK, that my code was all committed, that my repositories were cleaned up, that all my dotfiles in my homedir were in version control and so on. Most of it was already OK, but of course there were some small things left. I’ll do a write-up later on of my backup strategy and how I handle my dotfiles and so.
Time for the actual lion reinstall. How does that work? I bought Lion from the app store, so it was downloaded and installed by my mac: I didn’t have an install DVD. Turns out to be easy: just restart and press command-r during bootup and you’ll get a “lion recovery” menu. Choose the reinstall option and it will download the latest full version and install it for you. Simple and works.
The big surprise came when the computer rebooted. I expected a dialog to set up a main user. Instead, I got the regular login screen. Ok... Logging in... Hey! All my stuff is still there! All the settings, all my documents, all my music... No need to restore backups.
So: an OSX lion restore wipes only the OS and reinstalls it. Including xcode, btw. The rest (your own data, applications, settings) is retained. Actually pretty handy.
This did mean I had to clean up the kyngchaos packages and homebrew by hand. Just a matter of deleting some directories, telling homebrew to erase itself and adjusting my paths.
2012-01-26
All those UFO sightings? Just an unholy mix between a flying saucer and a German tank. You know, lots of nazis escaped to the moon (the dark side of it) in 1945. Now they’re coming back.
The film is partially volunteer-made and partially volunteer-funded. There’s no big Hollywood studio behind it. Made in Finland. Shot in Germany and Australia. Lots of input and ideas and designs from all over the world (“crowd-sourced”).
And... they made it to the Berlinale (Berlin film festival) program and they’ll apparently show in Berlin’s biggest cinema. Wow.
Here’s a trailer where they announce it. And there’s a lot of gorgeous footage at the start. This will be an awesome movie!
2012-01-26
I just had a funny autocorrect happening on my iPhone. It was still set to English and I entered the Dutch word “slagroom”, (“whipped cream” in English).
Autocorrect suggested “slag room”. It reminded me of slag that room, now from an older Schlock Mercenary comic. That was not what I had in mind when preparing my shopping list :-)
2012-01-22
I’ve made a couple of changes to my website (and thus weblog):
I’m getting used to git at the office. And my website was in svn. The svn is on me and my brother‘s server. And we’re going to re-build that server in the near future. And not having to set up svn there saves quite some time.
So I moved part of the code to github and the actual content to a simple directory in our own server. Git works just fine with a repository URL like ssh://vanrees.org/~/git/websitecontent. Just init a repo in ~/git/websitecontent on the server and off you go.
My site is static html now for two years or so. Generated with http://sphinx.pocoo.org/. No Plone (which it was when I worked with Plone) and no Django (which I’d like, as I work with it, but my current text-only setup works real fine and Sphinx is an essential Django documentation ingredient, too).
I also make my weblog with Sphinx. I need a couple of custom restructuredtext tags for that (like ..tags:: for handling tags). I basically point a script at a sphinx directory that contains my entries and it generates index.txt files for all the directories with a list of entries. And it generates tags/TAGNAME.txt files for the tags. Afterwards, I just run Sphinx and everything gets generated OK.
I got some questions in the past about my code, but never followed up by publishing my terribly Reinout-specific code. It is still Reinout-specific, but I’ve published it now. It might provide some ideas. The full code is at https://github.com/reinout/reinout.vanrees.org . Again: this is not something you can pip install. Best you can do is borrow and copy at the moment. (Threat the Python code in there as public domain).
Most of interest are probably https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/rst.py which provides the custom restructuredtext (and thus sphinx) tags. And https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/weblog.py which creates all the index and tags files.
It is not well-documented and quite specific to my case. But there are some docstrings and the code is nicely PEP8 compliant :-) If you need any info, feel free to contact me.
I have to re-visit the layout of the websites we make at my work (Nelen & Schuurmans). One of the things we want is to have a more attractive layout. And after some browsing, twitter bootstrap struck me as a good basis.
And, as I do often, I first experiment on myself. So I set out to change the layout of my own website by using twitter bootstrap. I’m not even using half of bootstrap’s possibilities, but I’m quite happy with the result. The basic typography is fine. There are some handy helpers for common things like labels (which I used to liven up the tags below the weblog headers on my weblog). And I quite like the simple header at the top of the page: just a simple bar.
And there’s another good thing about twitter bootstrap: less. See below.
What coffeescript is for javascript, less is for css. A simpler syntax that helps you write better and less duplicated css.
I like it! Take for example the headings of this entry. Hover over one and you’ll see a paragraph token at the end. Sphinx adds this so you can link to individual sections in a bigger file. Sphinx also has some css that hides the paragraph token normally and only shows it when you hover over the heading:
a.headerlink {
visibility: hidden;
}
h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink,
dt:hover > a.headerlink {
visibility: visible;
}
I restricted that to:
.headerlink {
visibility: hidden;
}
h1, h2, h3, h4, h5, h6, dt {
&:hover {
.headerlink {
visibility: visible;
}
}
}
It looks just like css, as you see in that first .headerlink rule. The second one has the special less stuff. The ampersand means an extra condition, so the &:hover effectively adds :hover to each of the h1..``dt``. This really shines if you have a couple of other extra conditions on the same set of elements.
Anyway, head over to http://lesscss.org/ where they have a better explanation.
I’m happy with the change. One extra change that I didn’t mention yet: I’ve removed deliverance from the mix. Deliverance is a tool that I used to change the layout of a couple of inputs (sphinx, docbook2html, latex2html) into my vanrees.org layout. It did that on the fly. But the versions kept changing and the versions I had were getting old and hard to update. And I had to run a wsgi process to run it, even though the actual website was all plain html.
I’m back to plain html now. I’ve implemented the layout in a Sphinx template. Simple to do, as it is just jinja2, which has the same syntax as Django. And the docbook and latex html exports are just that. I’ll leave them that way. It isn’t that important to have my logo on those pages :-)
One extra advantage: the site ought to be faster!
2012-01-18
(Talk at the 2012-01-18 Dutch Django meeting)
Security is about three things:
A well-known bad case is DigiNotar. Malware on most of their computers, including the most secure once. Their digital certificates were compromised and the company went bankrupt in three weeks time.
One thing that we have to take care of regarding Django: cookies. Cookies are stored in your browser and are linked to a website. All requests to that website get the cookies in every request. In Django, your session ID is stored in your cookie. If you get hold of the session ID, you get hold of the user’s session. Django stores the actual session’s data in the database, so the actual data is safe.
In Django 1.4 there are cookie-based sessions, but they’re a bad idea: never use them. He doesn’t know why they even build it in. UPDATE: they’re not that bad, see the comments.
CSRF: cross site request forging. Tricking the browser into posting requests to other websites. http://your-router/set_password?pwd=hamster . Django protects you by adding a secred csrf_token to every POST form and checking for it. So you’re safe.
XSS injection. Cross site scripting. Happens when you don’t filter/escape user submitted content. So you could send a <script> tag with a script that for instance steals every cookie on your server for stealing session IDs. Common is tricking a single user into running the script. A more powerful, but rarer, way is to get a script on some website and trick everyone visiting the site...
Django is mostly safe as it escapes template tag output by default. There is a rare vulnerability if you don’t put double quotes around html attributes. Everything after a space in var in <p class={{ var }}> ends up as a separate attibute...
Cookie security is pretty good in Django. 1.4 sets SESSION_COOKIE_HTTPONLY to true. You ought to do this always. And if you always use https, set CSRF_COOKIE_SECURE and SESSION_COOKIE_SECURE to true.
And... don’t trust in a regex that filters dirty tags out of your user-submitted code. There are SO many tricks and ways to circumvent any regex. Use markdown or something like that.
SQL injection is common, but Django has no know exploits. It is rock solid. But there are other ways. For instance LDAP injection. Don’t put an unfiltered username into your LDAP query as you can hack the query. Or direct user input into filenames. And what about shell injection? some_filename.txt;rm -rf... Shell calls are always very risky, but Python’s subprocess() helps a lot.
There’s a difference between validation and escaping. Validation is done as early as possible. Escaping is done as late as possible. Both serve a different purpose. As always: whitelisting is better than blacklisting.
A new option is clickjacking that overlays a transparent website over another one, stealing your form fields and clicks. It works with iframes. Django 1.4 has a new X_FRAME_OPTIONS='DENY' that works with newer browsers.
And... Don’t trust browsers. If you disable something in javascript, a user can work around it, for instance. The speaker could always book rooms in school before anyone else could as the only-two-weeks-in-advance limitation was implemented in javascript, not in the backend...
Regarding tests: check login-protected pages as anonymous first. Check actions for the admin user first as anonymous, then as a regular user. Check it.
And if you run some test site that’s (temporarily) on an external IP address, you can be sure someone finds it. So watch out.
If you run backups, do you also run restores? Do you keep backups with a different provider? Once a provider was hacked and its servers were wiped. Including their backups. All customer data lost.
Django: don’t ever run it in DEBUG mode in production. It gives away way too much information. Another tip: name the admin url something different than /admin. it is a dead give-away that it is a Django website.
Watch out for other apps on the same server. Your Django server can be compromised by running some faulty-configured PHP code on the same server.
Always restrict database access per app.
Summary. Django helps out with CSRF, SQL injection and XSS; you need to take care of all other injection attacs (LDAP, filesystem, etc). Use the build-in Django security features. And never trust what the browser sends. And secure the web server itself as well. Make backups.
2012-01-18
I sometimes commit sacrilege.
I make notes in my books. I scrawl in the margins. I underline important sentences. Sacrilege! It horrifies some people that someone can do that to a book.
Why do I do it?
You know what’s real sacrilege? Making notes in your bible :-) One of my bible study group members shuddered at the thought of not keeping it as tidy as possible. Well, mine is missing part of the cover, has notes in it and has lots of marks from been carried across town in many different kinds of bags... Sacrilege or usage? Both attitudes are fine, I think.
The kinds of notes that I make:
Maybe it is time for you to start making notes in your books?
2012-01-18
(Talk at the 2012-01-18 Dutch Django meeting)
Jeroen actually has zero Django experience, but he does nice things :-) That website of his is actually made with SVG.
SVG is more than 10 years old, Adobe published the first version in 1999. IE6 supported it with an adobe plugin, most other browsers started adopting it in 2003. IE9 fully supports it, btw. Nice: since 2011 (corrected, I said 2001...) it is even part of the html5 specification.
Scalable vector images are, well, scalable. Despite the screen size, you won’t see the pixels as in an upscaled pixel image.
SVG as a format is just XML. You can draw lines, arcs, circles, boxes and so on on a canvas. The output can be styled with css, just like html. It are DOM elements. And you can manipulate the drawn SVG elements with javascript, too.
What you use often is pen movements. M or m for move, L or l to draw a line. The lower case is relative to the current location, the upper case for absolute location. M 10 10 l 10 20 l 20 20 l 20 10 l 10 10 would draw a rectangle. For more elaborate forms you can use inkscape, an open source SVG editor.
Nice: events on SVG objects can be processed by Javascript. You can also do animations with SVG. There are three ways:
As browser compatibility is still a bit of a problem, you’re probably best off by using libraries like raphaeljs if you want to use SVG. (Personal note: I used raphaeljs with success in a small Django project last week).
He showed a couple of demos. What was particularly good was the easy and on-the-fly integration between forms and the picture (like for a custom table that can hover over the couch). Put in a different height of the couch and the table would adjust. Good javascript integration.
He uses it for making designs for feeding them into CNC milling machines like they have at the fab lab in The Waag in Amsterdam.
2012-01-18
(Short 5 minute talks at the 2012-01-18 Dutch Django meeting)
Coffeescript is not really a language, but it is syntactic sugar for javascript. A nicer syntax. It compiles to javascript. It uses indentation (like Python) to indicate code blocks instead of using curly braces, for instance. And the way to declare functions is easier and less verbose.
Coffeescript also allows classes, including constructors, and inheritance.
Nice: on the coffeescript site you can type in some code and get it transformed in to javascript. Handy for trying it out.
The language isn’t difficult. You can learn it within the hour (he said ‘a bit more than 5 minutes’). You can compile it into javascript on the server with node.js. You can also include the coffeescript.js script in your page and let the browser handle the compiling automatically (at a performance penalty, but it is easier for development).
Django-coffeescript integrates the compiling into Django. And you can even put inline coffeescript in your template. But normally you’ll just point at external files.
Bonus tip: django-less. Less is css + variables + nesting + inheritance. Integrated in the same way as django-coffeescript.
One drawback of coffeescript: it makes debugging harder as you have to debug the javascript that comes out of the compiler instead of the coffeescript code you just wrote.
“Django-fluent-dashboard - a fresh dashboard for your admin interface” was the title.
Django is a bit technical. His thought: how to make the Django admin more usable as a CMS that’s friendly? Another core thing in his thinkwork was the idea of reusable apps.
If you look at the current admin, you directly see the huge number of apps. Not nice.
Somebody made Django admin tools. Nice. A good solution. It gives the admin a nicer layout and even a menu bar that helps you find things.
The core idea: django-fluent-dashboard only modifies the first page of the admin: instead of the big list of apps you get friendly, grouped, icons.
He made a modification (or plugin, or newer version, I don’t know) that looks even nicer. More dashboard-based. See the example on the documentation site.
In the end, he hopes to turn it into a full CMS. There’s already a django-fluent-contents and django-fluent-pages on github. But they’re separate projects. django-fluent-dashboard works just fine with Django Fiber, for instance.
A small teaser for a bigger talk at next month’s Dutch python meeting.
Logbook is nicer logging for python than the standard library’s default logging module.
And for Django: whatever logging module you use, use django-sentry to get a website that collects your various site’s logs. He made a version that works especially nice with logbook, as logbooks gives you all logs of a web request if something goes wrong: it doesn’t only give you just the last traceback. So use debug logging liberally, it will help you debug.
http://30loops.net/ is a Python and Django hosting platform, located in Amsterdam (and a bit in Berlin since he moved recently).
They have years of experience in sysadmin and hosting, so they’re building yet another django/python/wsgi hosting platform. What you normally get at such a “platform as a service:
They differ on two major points:
They actually abstracted away quite a lot, so you can run your apps on 30loops’ own hardware, but also on other cloud providers. Handy if you have to deploy to multiple physical locations.
You can access all logs and metrics with REST: you own all your data.
They’re also on github: https://github.com/30loops
You can sign up for private beta now, btw.
Asking around gave the usual list of “good tools”, “passion” and “emacs”. He mentioned:
Why all these questions? Reason: there is a big need for professional software developers. You see lots of mails on the mailinglist asking for personell.
Would it be a good idea to start a “django academy”? Certification? Learning about acquiring clients? Create a network for supply and demand of professional developers?
There was quite some discussion afterwards (and I didn’t agree with some of the points, for the record).
Django fiber is a simple, user friendly CMS for your Django projects. It is getting bigger and bigger. 11k downloads since April 7, 2011. That’s about 1250 downloads per month. That’s only counted on pypi, though. Django CMS has 60k downloads since the beginning of time, as a comparison. So it is turning out to be pretty big!
What did we do since early 2011?
And... they build a lot of sites and apps with it (he said to me they build some 60 sites!)
Near-future plans:
They are realizing that they need to build a community around it if they want to get it off the ground further:
Nice: a couple of days ago, Luke Plant mentioned some big db optimizations for Django. They will look at that tomorrow, too.
(I wrote about an earlier 30 minute talk he gave on Django Fiber for if you want more information.)
Statistics: charts of posts per year and per month.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):