PyCon.de: the snake in the tar pit, complex systems with python - Stephan Erb¶

Tags: pycon, python

(One of my summaries of a talk at the 2017 PyCon.de conference).

He started with an xkcd comic:

Often it feels this way at the beginning of a project, later on it gets harder. You cannot just run import greenfield to get back to a green field again.

• Most engineering time is spend debugging.

• Most debugging time is spend looking for information.

• Most time spend looking for information is because the code/system is unfamiliar.

Unfamiliar, unknown code: now we’re talking about team size. You’re probably debugging someone elses code. Or someone else is debugging your code.

What can we do to understand such code? How can we spread the knowledge?

• You can do informal reasoning. Try to “run the code in your head”. Code reviews. Pair programming.

  By getting better here, we create less bugs.

• Try to understand it by testing. Treat it as a black box. See what comes out. Add integration tests. Do load tests. Perhaps even chaos engineering.

  By getting better here we find more bugs.

The first is better than the second way, right?

Both get harder when it becomes more complex. Complexity destroys understanding. But I need understanding to have confidence.

Keep in mind the law of leaky abstractions. All non-trivial abstractions are leaky. When something goes wrong, the actual implementation underneath “leaks through”.

Abstractions help us save time building something. It doesn’t save us time learning it afterwards. For python: watch out with dependencies. There are so many nice libraries we could use…

He showed some examples of how to make code more robust. One of the things he used was the pybreaker library, that one looks interesting.

State and mutability: if you have state and want to move from one to the other, there are lots of possibilities and lots of ways in which it can go wrong. If you “just” wipe the state and rebuild it to the new state you want to have, there’s less complexity.

If you pass along a dict, that dict might get modified. if you pass a named tuple, it cannot be modified. So that is less complex.

Keep in mind: complexity breeds complexity.

End-to-end is a networking concept. Try to keep functionality as close to the application as possible. That way, everything in between doesn’t have to deal with the extra functionality/complexity.

He suggested the paper end-to-end arguments in system design (PDF) as reading material. It is very good, even though it is old.

Photo explanation: some 1:87 scale figures on my model railway (under construction).