Pycon NL: practical observability - Daria Korsakova

Tags: python, pycon

(One of my summaries of the 2026 one-day Pycon NL conference in Utrecht, NL).

Full title: stop firefighting: practical observability for Python APIs, workers & jobs. The most important points:

  • What does it mean for a system to be observable?

  • Does that answer look the same for an API, a worker and a job?

  • Metrics tell us something is wrong, but how do we find out what happened and where?

  • How do we get the system to tell us, before our users do?

  • Where do we actually start without doing everything at once?

They started with a prototype of a new system. They had questions like “does it work” and “is it fast enough”. So they added a bit of monitoring: just a couple of logs and some simple graphs. And pushed it to production right away…

What does “observable” mean? It means you can answer questions like: is it up? Is it ready to serve traffic? Is it doing useful work? Is it failing? Is it slow? What changed recently? Which dependency/workload/customer/job/event-type is involved?

Is the answer the same for everything? No. You can have an API that restarts just fine on an error. So: it is up again, no problem. But because of the restart, it might have gotten a new IP address that isn’t recognised by something else in your stack. So the API works, but the rest of the app doesn’t. (That’s a real case that they saw.) Or a nightly database import job might succeed just fine (but there’s no new data as an upstream pipeline failed).

Metrics can help you notice problems before they become critical. Like API response times that slowly creep upwards. Time to investigate! Or something really goes wrong and the number of “error 500” messages spikes. Then you have to move from your metrics to the logs. There you can hopefully see what went wrong. If you need to dive deeper, you can start to use tracing (she doesn’t recommend activating this right away).

Notifications. There’s a sliding scale between “no observability” (the user has to report errors) and “alert on everything” (leading to alert fatigue). She recommends starting with a low number of alerts and slowly working upwards from there.

How to start? Best approach: roll it out in phases.

  • Start with basic visibility: health/readiness, structured logging.

  • Then workload metrics: request rate, error rate, number of processed jobs.

  • Then high-confidence alerts: api not ready and so. “P99” latency. Job queue backlog without progress.

  • Then dashboards.

  • And last tracing and deeper instrumentation.

She wrote an article about it: part one.

https://reinout.vanrees.org/images/2026/pyconnl-3.jpeg

Unrelated photo from our Nelen&Schuurmans company trip to Cologne/Köln in Germany. Hohenzollern railway bridge plus kathedral.