Reinout van Rees’ weblog

PyGrunn: advanced pytest - Òscar Vilaplana

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Imagine being a developer being woken up at night because your latest commit broke the website. You fix the issue, run the tests of your part of the code (which passes) and push to github. That runs all the tests and it fails in a completely unrelated piece of the code. But what is happening? Is the test wrong? Is your code wrong? “3 is not 90”: what does that mean?

What does it mean that this fails? What is the test’s promise? If a test you wrote fails, it should fail beautifully. It should tell exactly what’s wrong:

assert num_something == 2, "The number should match the number of added items"

You can use pytest fixtures to at least make the data the test is working with clearer.

You can make fixtures that work as context managers:

@pytest.fixture
def test_with_teardown():
    thing = create_something()  # setup
    yield thing
    thing.destroy()  # teardown

A tip: have fixtures per subsystem. Assuming you have multiple test directories, one per subsystem. Give every subsystem its own conftest.py. Different subsystems might get different fixtures even though using the same name (“product” for instance). This way you can tweak your main fixture items per subsystem.

  • Disadvantage: it is implicit instead of explicit…

  • Advantage: the fixtures can stay minimal. Otherwise your fixture has to support all use cases.

You can parametrize fixtures:

@pytest.fixture(params=["no-user", "disabled-user", "read-only-user"])
def unauthorized_user():
    if param == ...
        return ...
    if param == ...
        ...

Tests using that fixture are run three times, once for every possible kind of unauthorized user!

You can do it even more elaborate. You can make a kind of a “build matrix” and use @pytest.mark.parametrize.

If every test needs a temporary database or a temporary something, you can pass auto_use=True to the fixture, that’ll apply it automatically.

Pytest can help you with mocking, but sometimes you’re better off setting up dependency injection. So adding a parameter to the method you’re testing to accept some mocked item instead of its regular default.

If you think regular code is more important than the tests: he pro-tests :-)

You need tests because they give you a feeling of safety. If you feel safe, you dare to try things. Tests are a bit of a shared goal inside a team: you and your code want to belong. You want interaction: make sure your tests are communicative and helpful.

PyGrunn: python as a scientist’s playground - Peter Kroon

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

He’s a scientist. Quite often, he searches for python packages.

  • If you’re writing python packages, you can learn how someone might search for your package.

  • If you don’t write python packages, you can learn how to investigate.

Scientists try to solve unsolved problems. When doing it with computers, you basically do three things.

  • Perform simulations.

  • Set up simulations.

  • Analyze results.

Newton said something about “standing on the shoulders of giants”. So basically he predicted the python package index! So much libraries to build upon!

A problem is that there is so much software. There are multiple libraries that can handle graphs (directed graphs, not diagrams). He’s going to use that as an example.

Rule one: PR is important. If you don’t know a package exists, it won’t come on the list. Google, github discovery, stackoverflow, scientific liberature, friends, pygrunn talks, etc.

A README is critical. Without a good readme: forget it.

The five he found: graph-tool, networkx, igraph, python-graph scipy.sparse.csgraph.

Rule two: documentation is very important. Docs should showcase the capabilities. This goes beyond explaining it, it should show it.

I must be able to learn how to use your package from the docs. Just some API documentation is not enough, you need examples.

Watch out with technical jargon and terms. On the one hand: as a scientist you’re often from a different field and you might not know your way around the terms. On the other hand, you do want to mention those terms to help with further investigation.

Bonus points for references to scientific literature.

Documentation gold standard: scikit-learn!

python-graph has no online docs, so that one’s off the shortlist. The other four are pretty OK.

Rule three: it must be python3 compatible. On 1 january 2020 he’s going to wipe python2 from all the machines that he has write access to.

All four packages are OK.

Rule four: it must be easy to install. So pypi (or a conda channel). You want to let pip/coda deal with your dependencies. If not, at least list them.

Pure python is desirable. If you need compilation of c/c++/fortran, you need all the build dependencies. This also applies to your dependencies.

He himself is a computer scientist, so he can compile stuff. But most scientists can’t really do.

He himself actually wants to do research: he doesn’t want to solve packaging problems.

graph-tool is not on pypi, networkx is pure python. scipy is fortran/c, but provides wheels. igraph is C core and not on pypi.

So scipy and networkx are left.

Rule five: it must be versatile. Your package must do everything. If your package does a lot, there are fewer dependencies in the future. And I have to learn fewer packages.

If it doesn’t do everything, it might still be ok if it is extendable. He might even open a pull request to add the functionality that he needs.

Note: small projects that solve one project and solve it well are OK, too.

networkx: does too much to count. Nice. scipy.sparse.csgraph has six functions. So for now, networkx is the package of choice.

The first and third rules are hard rules: if it is a python2-only package it is out and if you can’t find a package, you can’t find it, period.

Conclusions

  • You need to invest effort to make ME try your package.

  • “My software is so amazing, so you should invest time and effort to use it”: NO :-)

  • If it doesn’t work in 15 minutes: next candidate.

PyGrunn: data processing and visualisation of tractor data - Erik-Jan Blanksma

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

He works for Dacom, a firm that writes software to help farmers be more effective. Precision farming is a bit of a buzzword nowadays. You can get public elevation data, you can let someone fly over your fields to take measurements or a cart can do automatic ground samples. This way you can make a “prescription map” where to apply more fertilizer and where less will do.

Another source of data is the equipment the farmer uses to drive over his field. As an example, the presentation looks at a potato harvester.

  • Which route did the harvester take through the field?

  • What was the yield (in potatoes per hectare) in all the various spots?

Some tools and libraries that they use:

  • Numpy: very efficient numerical processing. Arrays.

  • Pandas: dataseries.

  • Matplotlib: graph plotting library.

  • Postgis: geographical extension to the postgres databases.

Pandas is handy for reading in data, from csv for instance. It integrates nicely with matplotlib. With a one-liner you can let it create a histogram from the data.

With the .describe() function, you get basic statistics about your data.

Another example: a map (actually a graph, but it looks like a map) with color codes for the yield. The locations where the yields are lower are immediately clear this way.

When converting data, watch out with your performance. What can be done by pandas itself is much quicker than if it has to ask python to do it. For instance, creating a datetime from a year field, a month field, etc, that takes a long time as it basically happens per row. It is way quicker to let pandas concatenate the yyyy/mm/dd + time info into one string and then convert that one string to a datetime.

He showed the same example for creating a geometric point. It is quickest to create a textual POINT(1.234,8.234) string from two x/y fields and only then to convert it to a point.

Use the best tool for the job. Once he had massaged the data in pandas, he exported it to a postgis database table. Postgis has lots of geographical functions, like ST_CENTROID, ST_BUFFER, and ST_MAKELINE, which he used to do the heavy geographical lifting.

He then used the “geopandas” extension to let pandas read the resulting postgis query’s result. Which could again be plotted with matplotlib.

Nice!

PyGrunn: embedding the python interpreter - Mark Boer

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Writing scripts inside applications is often hard. Some of them luckily have an embedded version of python, but not all of them.

Two important terms: extending and embedding. Lots of scientific software is made available via extending: a python wrapper. Numpy and tensorflow, for instance.

The other way around is embedding: you put python inside your application. Useful for plugins, scripting. He doesn’t know if jupyter notebooks are a good example of embedding, but in any case, jupyter is doing funny things with the python interpreter.

CPython, which is the version of python we’re talking about, consists of three parts:

  • Bytecode compiler

  • Python virtual machine (the one running the compiled bytecode).

  • Python’s C API, which allows other programs to call python. The C API is the opposite of python: it is hard to read and write :-) Oh, and the error messages are horrendous.

But… starting python from C and sending lines to the REPL, that’s quite easy. PyRun-SimpleString(). He showed a 10-line C program that reads from stdin and lets python execute it.

He then expanded it to run in a separate thread. But soon his program crashed. The solution was to explicitly acquire and release the GIL (“global interpreter lock”).

A problem: multiprocessing doesn’t work. At least on windows. Starting another process from within python opens another version of the whole program you’re in…

A suggestion: pybind11, a handy library for helping you embed python into c++. It especially helps with managing the GIL and for embedding python modules.

Something he sees often, is that it is used to parallellize code for the benefit of python:

  • Convert python types to c/c++ types

  • release GIL

  • Perform computation

  • acquire GIL

  • Convert c/c++ types to return type.

A note on deployment: just include python in your installer.

PyGrunn: lessons from using GraphQL in production - Niek Hoekstra & Jean-Paul van Oosten

2019-05-10

Tags: pygrunn, python, django

(One of my summaries of a talk at the 2019 PyGrunn conference).

GraphQL is a different way to create APIs. So: differently from REST. You describe what you want to recieve back, instead of having a fixed REST api. With REST you often get too much or too little. You may need to do a lot of different calls.

REST often leads to a tight coupling between the front-end and the back-end. Changes to a REST api often break the front-end…

What they especially like about GraphQL: it is designed to have documentation build-in.

What they use: “graphene”, “graphene-django” and “relay”. On the front-end it is “apollo” (react-native, react, native ios/android).

With graphene-django you first have to define the data you’re exposing. The various object types the types of the attributes, the relations, etc.

A tip: differentiate between “a user” and “me”. Don’t add more data to a user object if it turns out to be you. Just have a separate endpoint for “me”. Way easier.

Caching: that needs to be done outside of graphene, it only can do a bit of caching right at then end on the resulting json. You’re better off caching at the django object level.

A potential problem spot is the flexibility that GraphQL gives you in querying relations. You need to do quite some clever hacking to use django’s select_related/prefetch_related speedups. You need to pay attention.

Uploading files is tricky. GraphQL itself does not handle file uploads. Their solution was to have a POST or PUT endpoint somewhere and to return the info about the uploaded file as GraphQL.

A downside of GraphQL: it is hard to predict the costs of a query. You can ask for adresses of contacts living at addresses of contacts and so on: you can kill the server that way. You could prevent that by, for instance, limiting the depth of the query.

There are reasons to stay with REST:

  • GraphQL is not a silver bullet. Yes, it has advantages.

  • The django/python tooling is still not very mature.

  • Determining the cost of a query is hard to predict beforehand.

But: just use GraphQL, it is fun!

PyGrunn: testing your infrastructure code - Ruben Homs

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Servers used to be managed by proper wizards. But even wizards can be killed by a balrog. So… what happens when your sysadmin leaves?

  • The point of failure is the sysadmin.

  • Knowledge about infrastructure is centralised.

  • It is non-reproducible.

A solution is configuration management. Chef, ansible, saltstack, puppet. Configuration that’s in source control instead of information in a sysadmin’s head.

  • It is a reproducible way to build your infrastructure.

  • Source code, so everyone can see how a system works.

  • You can even version your infrastructure.

He’ll use saltstack as an example, that’s what they’re using in his company. It is a master/minion system. So a central master pushes out commands to the minion systems.

For testing, he uses a tool called “kitchen”, originally intended for puppet, which can however also be used with saltstack: https://kitchen.saltstack.com/ . He showed a demo where he created a couple of virtualbox machines and automatically ran the salt scripts on them.

You can then ssh to those boxes and check if they’re OK.

But… that’s manual work. So he started using testinfra and pytest. Testinfra helps you test infrastructure. There are build-in tests for checking if a package has been installed, for instance:

def test_emacs_installed(host):
    assert host.package("emacs").is_installed

You can run those tests via “kitchen”. They use that to test their infrastructure setup from travis-ci.com.

PyGrunn: a day has only 24 ± 1 hours - Miroslav Šedivý

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Time zones… If you do datatime.datetime.now() you’ll get a date+time without timezone information. You can get different results on your laptop (set to local time) and a server (that might be set to UTC).

You can use datetime.datetime.utcnow() that returns UTC time. But… without a timezone attached. Best is to request the time in a specific timezone.

There are gotchas regarding time. Check your time only once in a calculation. If you call .utcnow() multiple times, you can get different dates when your code runs around 0:00.

Same with time.time(): if the “ntp” daemon adjusts your system clock in the mean time you get weird results. For that, there is time.monotonic().

The original source for all time zone information is the time zone database (tzdata). You can download it and look at all the files per timezone. Interesting reading! Look at Istanbul’s timezone. Daylight savings time being delayed by a day in a specific year because of a nationwide school exam. It was anounced a few weeks before. That’s all in the time zone database.

So if you make a Docker now and still use it in two years’ time, you might run into problems because summer time might have been abolished by the EU by then. So make sure you keep your time zone libary up to date.

PyGrunn: monitoring and profiling Flask apps - Patrick Vogel & Bogdan Petre

2019-05-10

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Patrick and Bogdan are students at Groningen University and they made the Flask Monitoring Dashboard. Some questions you might be interested in:

  • What is the performance of your flask-based apps?

  • How much is my web app being used?

  • Mean response time per endpoint?

  • Are there a lot of outliers? Most customers might have 20 items and one customer might have a couple of thousands: that’ll hurt performance for only that specific customer. Important to understand.

  • Monitor performance improvements in case you deploy a new version.

What are your options?

  • Commercial monitoring like google analytics or pingdom.

  • Write your own monitoring in flask middleware.

  • No middleware.

  • Best: use the flask monitoring dashboard!

It offers several levels of monitoring that you can configure per endpoint. From just monitoring the last time the endpoint has been called to full profiling including outlier detection. They showed a webpage with the profiling information: it sure looked useful. As an example, there’s a table view (hours vertically, days of the week horizontally) showing the relative usage per day/hour.

It works by “monkeypatching” the view functions. Flask has an internal dictionary with all the view functions! When profiling, a separate thread is started that periodically collects stacktrace info from the function being monitored.

Such monitoring of course has a performance impact. They’re actually researching that right now. The lower levels of monitoring (“time it was last used” and “amount of usage”) have no discernable impact. The two levels that do profiling have more overhead. For cpu/memory intensive tasks, the overhead is around 10%. For disk intensive tasks, it can hit 60%.

Pygrunn preparations

2019-05-09

Tags: pygrunn, python

Tomorrow (friday 2019-05-10), I’m going to the nice one-day Dutch python (and friends) pygrunn conference in Groningen (NL) again. Sold out, as usual. And rightfully so.

Anyway, to be honest, this blog entry is mostly about me testing my blog setup on a new laptop. I’ve got a linux laptop at work (which has its advantages) and since a few months I’ve got a second-hand macbook pro at home (because linux also has its disadvantages). But I never got around to setting up my blog software here till now. And I’m taking the macbook to the conference, so high time to get everything working :-)

I’m giving a talk myself on cookiecutter, the handy tool to create initial project structures. I’m a big fan of it, so I’m glad to tell the fine folk at pygrunn about the joys and advantages of using it. I gave the talk before at work and at the Amsterdam python meetup, but that was a couple of months ago. So I invested two days to polish and re-order and rehearse the presentation again. And I updated the presentation with some more photos and videos of my model railway…

Summaries of the Python meetup in Amsterdam

2019-04-11

Tags: python, pun

I’ve made notes again at the 2019-04-11 Amsterdam Python meetup in the byte office. Here are the summaries.

Ethics in IT - Nick Groenen

Computer systems are taking over the world. They’re influencing everything. A lot of good has come out of this. But there’s also a downside.

We’re talking (at meetups like this, for instance) mostly about tech, not about the people that build it or the people affected by it. Nick Groenen calls that an ethics problem.

As an example, Uber’s “greyball” that was originally written for a good purpose but that was later used to mislead governments. Same with Volkswagen’s diesel emissions scandal: detecting when there’s an official test and adjusting the motor parameters to seem more environmentally friendly.

How ethical is it to work on something like that?

The above examples are of big companies. But what about working for smaller companies? You have the same ethical challenges there. What are you doing with your logfiles? How much information do you mine about your customers? Do you give customers’ data to your boss when he asks about it even though it isn’t allowed by your privacy policy?

And how do you treat new programmers? Is there budget for training? How do you treat interns? That’s also an ethical question.

A good first step would be to actually acknowledge the problem in our IT industry.

Ethics… we didn’t learn for that. There are some pointers to get you started:

  • https://www.ieee.org/about/corporate/governance/p7-8.html

  • https://ethics.acm.org

  • Talk to other departments in your organisation, like the legal department. Or your local UX designer.

  • Try to apply ethics during design and code review.

  • Design for evil. Try to abuse what you’re building. It is actually quite a funny exercise!

  • Understand the communication and decision-making structure of your company. Who do you have to talk to? Who makes the actual decisions?

We need a better testing framework - Noam Tenne

Noam Tenne works at Van Moof, a bicycle company. He made his own testing framework, Nimoy, named after the actor who played Spock.

TDD ought to be a given now. It isn’t always, but it ought to be. Test Driven Development.

What about BDD: Behaviour Driven Development? A test for a feature is written like this:

  • SCENARIO some description

  • GIVEN some starting point

  • WHEN some action

  • THEN some expected result

You can do it with the python behave library. It works with one file per scenario with decorators inside them. Which is a bit cumbersome. And not business-people-friendly.

Nimoy goes back to the old way. One test method per feature. The steps (“given”, “when”) are indicated by using the with statement:

from nimoy.specification import Specification

class ExampleSpec(Specification):

    def example_feature(self):
        with setup:
            a = 5
        with when:
            a += 1
        with expect:
            a == 6  # Look! No

It functions like a DSL (domain specific language) this way. It isn’t real regular python code, but it uses python syntax in a specific way. (Note by Reinout: I suspected he was using metaclasses behind the scenes to adjust the way python treats keywords and variables within the Specification classes, but he later mentioned he used the AST (python’s Abstract Syntax Tree module). ast.NoteTransformer and so.).

And DDT? Data Driven Testing? There’s support in nimoy for parametrizing existing data (for instance from a file or database) and to use that in the tests.

Nimoy has some nice syntactic sugar for using mock classes. Easy to specify expected output, for instance.

He’s got a series of blog posts on how/why he created Nimoy.

He showed some internal code examples afterwards. Including Nimoy tests for nimoy itself. Looked nice.

 
vanrees.org logo

About me

My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.

Weblog feeds

Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):