Svn to git

I'm getting used to git at the office. And my website was in svn. The svn is on me and my brother's server. And we're going to re-build that server in the near future. And not having to set up svn there saves quite some time.

So I moved part of the code to github and the actual content to a simple directory in our own server. Git works just fine with a repository URL like ssh://vanrees.org/~/git/websitecontent. Just init a repo in ~/git/websitecontent on the server and off you go.

Sphinx as weblog: code is now available (somewhat)

My site is static html now for two years or so. Generated with http://sphinx.pocoo.org/. No Plone (which it was when I worked with Plone) and no Django (which I'd like, as I work with it, but my current text-only setup works real fine and Sphinx is an essential Django documentation ingredient, too).

I also make my weblog with Sphinx. I need a couple of custom restructuredtext tags for that (like ..tags:: for handling tags). I basically point a script at a sphinx directory that contains my entries and it generates index.txt files for all the directories with a list of entries. And it generates tags/TAGNAME.txt files for the tags. Afterwards, I just run Sphinx and everything gets generated OK.

I got some questions in the past about my code, but never followed up by publishing my terribly Reinout-specific code. It is still Reinout-specific, but I've published it now. It might provide some ideas. The full code is at https://github.com/reinout/reinout.vanrees.org . Again: this is not something you can pip install. Best you can do is borrow and copy at the moment. (Threat the Python code in there as public domain).

Most of interest are probably https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/rst.py which provides the custom restructuredtext (and thus sphinx) tags. And https://github.com/reinout/reinout.vanrees.org/blob/master/rvo/weblog.py which creates all the index and tags files.

It is not well-documented and quite specific to my case. But there are some docstrings and the code is nicely PEP8 compliant :-) If you need any info, feel free to contact me.

Twitter bootstrap css

I have to re-visit the layout of the websites we make at my work (Nelen & Schuurmans). One of the things we want is to have a more attractive layout. And after some browsing, twitter bootstrap struck me as a good basis.

And, as I do often, I first experiment on myself. So I set out to change the layout of my own website by using twitter bootstrap. I'm not even using half of bootstrap's possibilities, but I'm quite happy with the result. The basic typography is fine. There are some handy helpers for common things like labels (which I used to liven up the tags below the weblog headers on my weblog). And I quite like the simple header at the top of the page: just a simple bar.

And there's another good thing about twitter bootstrap: less. See below.

Less

What coffeescript is for javascript, less is for css. A simpler syntax that helps you write better and less duplicated css.

I like it! Take for example the headings of this entry. Hover over one and you'll see a paragraph token at the end. Sphinx adds this so you can link to individual sections in a bigger file. Sphinx also has some css that hides the paragraph token normally and only shows it when you hover over the heading:

a.headerlink {
    visibility: hidden;
}

h1:hover > a.headerlink,
h2:hover > a.headerlink,
h3:hover > a.headerlink,
h4:hover > a.headerlink,
h5:hover > a.headerlink,
h6:hover > a.headerlink,
dt:hover > a.headerlink {
    visibility: visible;
}

I restricted that to:

.headerlink {
  visibility: hidden;
  }

h1, h2, h3, h4, h5, h6, dt {
  &:hover {
    .headerlink {
      visibility: visible;
      }
    }
  }

It looks just like css, as you see in that first .headerlink rule. The second one has the special less stuff. The ampersand means an extra condition, so the &:hover effectively adds :hover to each of the h1..``dt``. This really shines if you have a couple of other extra conditions on the same set of elements.

Anyway, head over to http://lesscss.org/ where they have a better explanation.

Closing off

I'm happy with the change. One extra change that I didn't mention yet: I've removed deliverance from the mix. Deliverance is a tool that I used to change the layout of a couple of inputs (sphinx, docbook2html, latex2html) into my vanrees.org layout. It did that on the fly. But the versions kept changing and the versions I had were getting old and hard to update. And I had to run a wsgi process to run it, even though the actual website was all plain html.

I'm back to plain html now. I've implemented the layout in a Sphinx template. Simple to do, as it is just jinja2, which has the same syntax as Django. And the docbook and latex html exports are just that. I'll leave them that way. It isn't that important to have my logo on those pages :-)

One extra advantage: the site ought to be faster!

Initial one-time setup

First things first: create a ~/.svn2git/authors file. svn2git uses that to couple svn users with emails known to github. (Update 2011-10-13: I used to say "usernames on github" instead of "emails known to github", Adam Reeve corrected me in the comments below. He's right that the username doesn't matter, you can use a real name.)

Here's a sample from mine:

# svn_username = Real Name <email@you.use.in.github>
Reinout = Reinout van Rees <reinout@vanrees.org>
Gijs = Gijs Nijholt <gijs.nijholt@gmail.com>
reinout.vanrees = Reinout van Rees <reinout@vanrees.org>
...

If there are usernames missing, svn2git will warn you about it, btw.

Next, install svn2git. See https://github.com/nirvdrum/svn2git#readme . I've copied the instructions for debian/ubuntu here:

$> sudo apt-get install git-core git-svn ruby rubygems
$> sudo gem install svn2git --source http://gemcutter.org

Converting something to github

Create a temporary directory and chdir to it:

$> mkdir /tmp/something
$> cd /tmp/something

Call svn2git on the main directory, not on trunk:

$> svn2git https://office.nelen-schuurmans.nl/svn/Products/Library/timeseries
Using higher level of URL: ...
W: Ignoring error from SVN, path probably does not exist: ...
W: Do not be alarmed at the above message git-svn is just ...
This may take a while on large repositories
W: +empty_dir: Products/Library/timeseries/trunk/doc/source/_static
W: +empty_dir: Products/Library/timeseries/trunk/timeseries.egg-info
...
Lots of warnings and fatal errors, btw.
...
HEAD is now at a6be9f3... Tagging 0.7
Previous HEAD position was a6be9f3... Tagging 0.7
Switched to branch 'master'
Counting objects: 367, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (359/359), done.
Writing objects: 100% (367/367), done.
Total 367 (delta 220), reused 0 (delta 0)

Your temporary directory is now a full local git repository. Now we have to make the github repository ("the remote") and push our repository to it.

Create a repository on github and follow the instructions for importing an existing repository:

$> git remote add origin git@github.com:nens/timeseries.git
$> git push -u origin master

And also push the tags:

$> git push --tags

Then, run git branch to see if there are branches that you want to push over to git as they're NOT pushed by default:

$> git branch
* master
reinout-experimental
$> git push origin reinout-experimental

Corner case: moved svn directories

Several of our projects are old enough to have been moved inside our svn repository. Such a move is not something that svn2git likes. In such a case, find the revision number of the move in svn and run svn2git as follows:

$ svn2git https://office.nel...nl/timeseries --revision 11070

This starts the conversion at that revision number. You do lose the specific changesets from before that time, though.

Basic mental model of how Git/github works

The basic concept of git, paraphrased a bit:

• You've got multiple respositories in various places.
• Every repository is basically a big bucket of changesets and a handful of pointers.
• Git effectively starts out empty and and applies a string of changesets and ends up at a directory filled with your source code.
• Which string of changesets? That is determined by the pointer. You point at a certain changeset and you get that one and its parents. (They all have a pointer to their parent).
• A pointer can be a tag or a branch or even a pointer at something in another repository.

"Multiple repositories in various places"? Yep. A project on github is a repository. If you grab a local copy (=you clone it), your local copy is also a repository with all the contents. If you "fork" a project on github? Yep, a full copy. Getting stuff from one to the other means "push" and "pull".

But if your local thingy is a full repository, including all branches/tags/whatever, what ends up in your actual code directory? This is because git has two layers (and effectively three if you count a remote repository):

• Your actual code directory. This is your visible directory. The git name for it is your "working copy".
• The full contents of the repository including pointers and changesets. Hidden in your working copy in the .git/ directory. When you grab stuff from a remote repository, it ends up in here. And when you push, it is the contents of the .git directory that you push over.
• One or more remote repositories. (Technically those repositories' indexes: you can't muck around directly in someone else's working copy, of course).

Let me sneak in one extra layer:

• The so-called "index". These are the local changes you've collected for your next commit. So if you commit them, the index is empty again and what's in your index is now in your repository.

So, compared to subversion, you've got one extra layer to keep track of, which of course complicates the mental model of what's happening. With subversion, you only have a central repository and a local working copy. Git (and mercurial, bzr and the rest) add that full local repository (git's "index").

Most basic git commands

Here are the git commands I've personally used in the last few weeks:

62 git status
31 git diff
26 git push
20 git add
18 git commit
17 git show
13 git mv
13 git help
10 git pull

(Output of history|grep git|awk '{print $2 " " $3}'|sort|uniq -c|sort -nr|head).

I won't repeat git's own documentation here. You'll probably have to do quite some googling before you're comfortable with those commands. I'll have to do that googling myself, too.

Some comments on most of those commands that might help. Note that I'm using italics explicitly here to differentiate your repository from the remote repository (assuming you have a remote repository at github). Here you go:

• git status. Relatively helpful, as it often suggests what you need to do. If files are changed, but they haven't been "added to the index" yet, it suggests that you might want to git add them. The index is what ends up as your next commit, so this is the set of changes you want to stuff into your repository:

  $ git status
  # On branch master
  # Changes not staged for commit:
  #   (use "git add <file>..." to update what will be committed)
  #   (use "git checkout -- <file>..." to discard changes in working directory)
  #
  #   modified:   MANIFEST.in
  #
  no changes added to commit (use "git add" and/or "git commit -a")
  

• git diff. All by itself, without options, it is the difference between your local working copy and the index. So the changes that you haven't marked yet for inclusion in your next commit:

  $ git diff
  diff --git a/MANIFEST.in b/MANIFEST.in
  index c57b4ff..fe6332b 100644
  --- a/MANIFEST.in
  +++ b/MANIFEST.in
  @@ -1,2 +1,2 @@
   include *.rst
  -recursive-include lizard_map *.rst *.py *.html *.css *.js *.jpg *.png *.gif *.pdf *.shp *.json
  +recursive-include lizard_map *.rst *.py *.html *.css *.js *.jpg *.png *.gif *.pdf *.shp *.json *.po *.mo
  

  After adding a couple of things to the index, you probably want to review what you've "staged for commit in the index":

  $ git diff --staged
  diff --git a/MANIFEST.in b/MANIFEST.in
  index c57b4ff..fe6332b 100644
  --- a/MANIFEST.in
  +++ b/MANIFEST.in
  @@ -1,2 +1,2 @@
   include *.rst
  -recursive-include lizard_map *.rst *.py *.html *.css *.js *.jpg *.png *.gif *.pdf *.shp *.json
  +recursive-include lizard_map *.rst *.py *.html *.css *.js *.jpg *.png *.gif *.pdf *.shp *.json *.po *.mo
  

• git add. Just add filenames as suggested by git status. A helpful variant is git add -u, this automatically adds changes to all already-known files. (-a adds all changes, but at the risk of including unwanted files).

• git commit. This stuffs the changes you added to (=staged in) the index to your repository.

• git push. This pushes what's in your repository to the remote repository. (Assuming git knows what your remote repository is and which branch in the remote repository it has to talk to. And when the push doesn't break the remote repository. And when you've got access to that remote repository.)

• git pull. This is basically a combination of git fetch followed by git merge. Fetch fetches the current state of the remote repository (all changesets and pointers), merge merges those changes (=*your* repository) into your working copy. So just do a git pull to grab the changes of the remote repository and get them in your repository and your working copy.

• git help. git help some_command or git some_command --help both do the same. Quite decent documentation, actually! Use "git help" often when starting out with git!

Don't forget you can add -v to get more verbose output of what git is doing. This can be very helpful to see what's happening! Compare:

$ git pull
Already up-to-date.

with the more explanatory:

$ git pull -v
From github.com:lizardsystem/lizard-map
 = [up to date]      alexandr-workspace-changes -> origin/alexandr-workspace-changes
 = [up to date]      gijs-nepal -> origin/gijs-nepal
 = [up to date]      master     -> origin/master
 = [up to date]      reinout-api -> origin/reinout-api
 = [up to date]      reinout-tastypie -> origin/reinout-tastypie
Already up-to-date.

Github remote repository

When you create a new repository on github, you get a "clone url". Something like https://github.com/jcrocholl/pep8.git. To get a local copy, clone it:

$ git clone https://github.com/jcrocholl/pep8.git
Cloning into pep8...
remote: Counting objects: 861, done.
remote: Compressing objects: 100% (369/369), done.
remote: Total 861 (delta 424), reused 833 (delta 398)
Receiving objects: 100% (861/861), 160.20 KiB, done.
Resolving deltas: 100% (424/424), done.

This sets up some defaults in your repository. "Your repository" means that "pep8" directory that got set up (and more specifically the .git/ directory inside it).

Some of the defaults that are helpful to know and that partially explains why several of the commands I showed don't need that many options:

• git branch shows just a "master" branch. That's "trunk" in svn-speak. The star in front of it shows it as the default branch:

  $ git branch
  * master
  

• git branch -a shows also the remote branches:

  $ git branch -a
  * master
    remotes/origin/HEAD -> origin/master
    remotes/origin/master
  

  That remotes/origin/master remote is a pointer at the master (so: the trunk) of the remote github repository.

• Your respository's "master" branch has been configured with the remote

  repository's "master" as its default push/pull location. So git pull grabs new revisions from github into your repository and updates your working copy.

Integration with checkoutmanager

Many colleages (and others!) use my checkoutmanager tool. Based on a config file, it manages your various svn/git/hg/bzr checkouts. Normally you have a couple of directories with checkouts and doing "svn up" in all of them is a bit of a chore. That's where checkoutmanager comes in. checkoutmanager up does an svn up in all your svn directories.

And.... for git it does a git pull. Yes, it supports git.

checkoutmanager status does git status in your git directories that it knows about.

Very handy: checkoutmanager out, which tells you which of your respositories have commits that you haven't pushed to github yet!

As a quick example, here's a snippet from my .checkoutmanager.cfg:

[git]
vcs = git
basedir = ~/git/
checkouts =
    git@github.com:reinout/pep8.git
    git@github.com:reinout/gitignore.git

[nensgit]
vcs = git
basedir = ~/git/nens/
checkouts =
    git@github.com:lizardsystem/nensskel.git
    git@github.com:lizardsystem/lizard-ui.git
    git@github.com:lizardsystem/lizard-map.git

Integration with buildout

We use buildout for all our projects. Handier than pip/virtualenv as it provides some extra functionality like easy django setup, apache config files from templates, cronjob setup and so on. And... it has mr.developer. A buildout extension for managing a couple of git (or svn/bzr/hg) checkouts inside your project.

Because.... git and mercurial don't have something that works like svn:externals. Or at least that doesn't work as nice/reliable/integrated/whatever. But mr.developer takes care of that in buildout. Here's a relevant snippet from a buildout.cfg:

[buildout]
...
extensions =
    mr.developer
    buildout-versions
parts =
    mkdir
    django
    ...
develop = .
...

[sources]
lizard-ui = git git@github.com:lizardsystem/lizard-ui.git

...

This does three things:

• It adds a bin/develop command for managing your sources.
• It adds a src/ directory where it places any checkouts/clones it makes.
• bin/develop now knows that the "lizard-ui" package is a git checkout with a certain clone url.

mr.developer does a couple of things. Here's the help:

$ bin/develop -h
usage: develop [-h] [-v]  ...

optional arguments:
  -h, --help        show this help message and exit
  -v, --version     show program's version number and exit

commands:

    activate, a     Add packages to the list of development packages.
    checkout, co    Checkout packages
    deactivate, d   Remove packages from the list of development packages.
    help, h         Show help
    info            Lists informations about packages.
    list, ls        Lists tracked packages.
    rebuild, rb     Run buildout with the last used arguments.
    reset           Resets the packages develop status.
    status, stat, st
                    Shows the status of tracked packages.
    update, up      Updates all known packages currently checked out.

Once you want to work on the "trunk" of one of the packages managed by mr.developer, check it out (which also activates it):

$ bin/develop checkout lizard-ui
INFO: Queued 'lizard-ui' for checkout.
INFO: Cloned 'lizard-ui' with git.
INFO: Activated 'lizard-ui'.
WARNING: Don't forget to run buildout again, so the checked out packages are used as develop eggs.

As mentioned in that message, run buildout again and it will be installed as a development egg.

bin/develop status shows the current status of the checkouts, but the output isn't very clear. Run bin/develop status --help to see what it all means.

bin/develop update does a git pull for every package.

Creating a new project

Starting a new project and getting it in github is pretty simple:

• First create it in github (just clicky-clicky on the github website).
• Then set up your project locally (we use nensskel).
• Run git init in that directory to turn it into a git directory. Add/commit/adjust where needed.
• Do a git push to the clone url you see in your git webpage!

Upcoming: branches and workflow

What I haven't covered yet: branches and workflow. I'll do that in a later article.

class View

View is the base class for all views. It takes care of two things that you should know about:

• The .as_view() method you call on the view in your urls.py.
• It calls .get() for a GET request, .post() for a POST request and so on.

In your urls.py, you need to import the class and feed it to your urls with a .as_view() appended. You might ask why that .as_view() is needed. It is needed because you want your view objects to be thread-safe. That's an expensive word for "self.bla_bla = 3 should be safe to use and self.bla_bla should not end up as a value in another view instance". The way urls.py works (it expects callables) wouldn't work nicely with classes. Anyway, that's High Python magery. Nothing to worry about. Just accept that the .as_view() is a pretty fine and decent solution :-)

You can even pass keyword arguments to as_view() which will be set as attributes on your view objects. So YourView(template_name='my_view.html') will give your view a self.template_name with the value 'my_view.html'. Provided your view had some default value for that attribute, otherwise you'd get an error: nicely explicit.

Behind the scenes, what as_view() returns is the .dispatcher() method that basically just calls .get() for a GET request, .post() for a POST request and so on. Of course it passes along arguments and keyword arguments that your url config wishes to pass along.

Now, you don't need to worry about that dispatch method. You won't ever have to customize that one. Likewise the as_view method. Remember the two customization points:

• If your view class always has some attribute (self.template_name will be a common one), you can customize that right in your urls.py by passing it a long as a parameter (.as_view(template_name='something.html')).
• Args and keyword args you defined in your urls.py get passed just fine to the get()/post()/delete()/head() whatever.

class TemplateResponseMixin

Hey, our first mixin! Python support multiple inheritance. So your view can inherit from more than one class. This way, you can mix in one or two methods or attributes into your class by inheriting from an extra mixin class.

Warning: using mixins looks fine on the first sight, but experience has shown that debugging can quickly become an utter hell as you have to look in 10 different spots before you figure out which method does what to your hapless data... (8 years of zope/plone experience speaking here!)

Anyway... TemplateResponseMixin mixes in two really important things:

• A template_name attribute. The value of this gets used as the template. (Actually, it is a little bit more involved: there's a .get_template_names() that returns the value of template_name, but that's not something you'll commonly customize).
• A render_to_response() method that accepts a context dictionary and renders the template with that context. Behind the scenes, it wraps the context dictionary automatically in a RequestContext so that all the so-called context processors are active. Not wrapping the context in a RequestContext is a common error ("hey, my STATIC_URL variable seems to be empty in my templates!), so getting it handled automatically is great.

As it is a mixin, this render_to_response() isn't actually called anywhere within the class itself. So onwards to...

class TemplateView

... TemplateView inherits from both View and TemplateResponseMixin. It is indended just for rendering a template with a context. Technically, it means responding to a GET request and returning a response.

What you need to know about TemplateView itself:

• It has a .get_context_data() method that you ought to implement in your subclass. It should return a context dictionary. So implement this method and the dictionary you return will be given to your template. Note that this method gets passed any keyword arguments you defined in your url.
• It implements the .get() method that View's dispatcher will call when it gets a GET request. This one calls .get_context_data() to get the context and feeds it to TemplateResponseMixin's .render_to_response(). This happens behind the scenes and you shouldn't ordinarily have to customize it.

So. TemplateView is useful by itself. For many views, this is what you ought to be subclassing. Despite the many layers and methods-calling-other-methods that happens, there are just two items you need to implement in your subclass:

• The template_name attribute that points at your template.
• The .get_context_data() that ought to return a context dictionary.

Just for clarity, here's the example view again:

from django.views.generic.base import TemplateView


class SomeView(TemplateView):
    template_name = 'my_app/someview.html'

    def get_context_data(self, **kwargs):
        return {'name': 'reinout'}

Conclusion

In the end, the TemplateView class is pretty simple when you actually use it.

The underlying mechanism is relatively elaborate and depends quite a lot on every part doing exactly what it has to do. And it consists out of quite some parts. So I hope the complexity you can build into it doesn't grow out of hand.

Getting the RequestContext automatically is a big plus for TemplateView.

Have fun writing class based views! I'll probably write more about them in the weeks to come as I discover more things. There are handy views for redirecting, for handling forms and so on. Lots to discover.