Pygrunn: Data acquisition with the Vlermv database - Thomas Levine¶

Tags: pygrunn, python

(One of the summaries of the 2015 Pygrunn conference)

Thomas Levine wrote vlermv. A simple “kind of database” by using folders and files. Python is always a bit verbose when dealing with files, so that’s why he wrote vlermv.

Usage:

from vlermv import Vlermv
vlermv = Vlermv('/tmp/a-directory')

vlermv['filename'] = 'something'
# ^^^ This saves a python pickle with 'something' to /tmp/a-directory/filename

The advantage is that the results are always readable, even if you lose the original program.

You can choose a different serializer, for intance json instead of pickle.

You can also choose your own key_transformer. A key_transformer translates a key to a filename. Handy if you want to use a datetime or tuple as a key, for instance.

The two hard things in computer science are:

• Cache invalidation.

• Naming things.

Cache invalidation? Well, vlermv doesn’t do cache invalidation, so that’s easy. Naming things? Well, the name ‘vlermv’ comes from typing randomly on his (dvorak) keyboard… :-)

Testing an app that uses vlermv is easy: you can mock the entire database with a simple python dictionary.

What if vlermv is too new for you? You can use the standard library shelve module that does mostly the same, only it stores everything in one file.

A drawback of vlermv: it is quite slow.

Fancy full-featured databases are fast and nice, but do you really need all those features? If not, wouldn’t you be better served by a simple vlermv database? You might even use it as a replacement for mongodb! That one is used often only because it is so easy to start with and so easy to create a database. If you don’t have a lot of data, vlermv might be a much better fit.