Pycon.de: opentelemetry - Mika Naylor & Emily Woods¶

Tags: pycon, python

(One of my summaries of the 2025 pycon.de conference in Darmstadt, DE).

I followed a workshop on opentelemetry using grafana, promoteus, tempo and loki.

First opentelemetry. “An observability framework for understanding systems”. The goal: to help engineers understand their systems.

The main concept is “signals”, so metrics, logs, traces, etc. Nice: vendor and language agnostic. You can integrate it with lots of observability platforms like grafana, prometheus and more. What they used for the workshop:

• Metrics: prometheus

• Logging: loki (from grafana)

• Tracing: tempo (also from grafana)

• Visualisation: grafana.

• An “OTel collector” forwards the available metrics to the abovementioned services.

Some of the possible metrics:

• Counters. Just count up.

• Up/down counters. A counter that can decrement, too. Useful for number of current visitors for instance.

• Gauges.

• Histograms. For samples, 95 percentiles and so.

Logging can be structured or unstructured. Log handlers can be simple: these send every log line over immediately, which is inefficient in production. So there are also batch log handlers and so. Note: you can also “just” do some normal log scraping/shipping to loki.

Tracing is extremely valuable, but it is also the most involved and computationally expensive. As you’re basically profiling your code, you probably don’t want to trace everything. You can also do trace sampling.

In the workshop, we configured everything with python code, which could get a bit elaborate. A lot can also be set via environment variables.

Distributed tracing gives you a wider view over regular tracing. Regular “spans” profile individual pieces of code, a “distributed trace” can follow a request throughout your code base, useful for following a web request that touches several parts of your code. Core is the propagation of the trace context (so: the trace ID) into all the various spans.

Doing this by hand is a pain, so the normal method is with instrumentation. Many web frameworks have python packages that set up instrumentation automatically. They add a trace ID into the request, which can then be picked up everywhere. Such an instrumentation often also add basic metrics about request duration and so.

Github url with the workshop code: https://github.com/autophagy/pycon-2025-otel-workshop

Photo explanation: picture from the recent “on traxs” model railway exhibition in Utrecht (NL)