Pycon.de keynote: the future of AI: building the most impactful technology together - Leandro von Werra¶

Tags: pycon, python

(One of my summaries of the 2025 pycon.de conference in Darmstadt, DE).

Can AI be build in the open? Especially LLMs. Key components are data, compute, pretraining, posttraining, scaling. Can these components be build in the open? As open source?

Many models are closed, like claude, gemini, OpenAI o1. There are models with open model weights (but a black box otherwise): LLaMA, deepseek, Mistralai. And you have fully open models: granite, bloom, olmo, starcoder2.

Why would we want to do it? Doing it in the open?

• Transparency on pretraining. What data was used? How was my data used. How was the data filtered? This addresses biases, attribution and trust.

• Transparency on alignment. Models are aligned for safety/values. Which values are in the model? Are values global? In closed models, there are only a few people that define how the model behaves, but it has lots of influence, potentially world-wide.

Deepseek-R1 is not open source, but at least the weights are open. This helped shift the discussion a lot, as previously all the big ones were closed api-only models. At hugging face they try to make a fully open version, open-R1.

Open is closing the gap. When GPT-4 came out, it took a year for open models to catch up. At the moment it is more like just two months.

• In Europe, there are multiple public data centers with 10.000+ GPUs.

• Collaborative training: BOOM.

• Building the know-how together? For that you need large, high quality datasets for pre/posttraining. Good training recipes. Learning to scale to 1000s of GPUs.

• At hugging face, they try to build open datasets, also multilingual. See a “fineweb” blog post. And also small models. And some tools to build LLMs.

Some trends:

• Scaling will continue. Bigger is simply better. But…. it is exponential: it gets harder and harder. More compute, more power consumption.

• A new frontier: scaling the test time compute. This could really help improve accuracy.

• Reasoning, as done by deepseek, is interesting.

• Challenge: agents. Agency requires multi-step reasoning, which is more harder.

• AI in science.

What can you do?

• Build high-quality datasets.

• Fine-tune task specific models.

• Work on open source tooling.

• Join an open collaboration.

It is still early days for AI, there’s a lot of stuff you can do. Don’t think that everything is already figured out and build.

Photo explanation: random picture from Darmstadt (DE)