Pycon NL: building resumable Python pipelines - Yuvraj Singh Pathania¶
(One of my summaries of the 2026 one-day Pycon NL conference in Utrecht, NL).
Full title: fire, forget, regret: building resumable Python pipelines.
If you run a task in an async loop, there’s often no reference to the task. So it can be garbage collected! You can solve that by keeping a reference in some list. But even then, the whole process can die. What happens with your task then?
Often you have to re-run the task. He himself is into LLMs, so failed tasks that you have to re-run can become expensive.
BackgroundTasks is part of FastAPI and it works great for spinning off longer running tasks. Real handy, but there is no retry mechanism and no recoverability in case of crashes. “Fire-and-forget”.
You can use specific task queues. ARQ, Celery, Dramatiq. In this case, the job itself is safe and can be re-tried. But not the individual steps within the job.
What he wants is durable execution. Write down every step you finish; on restart, skip what is written. It can save you lots of time (and possibly money). It might be simpler to set up than you think.
His code + slides are at https://github.com/batman004/fire-forget-regret
Unrelated photo from our Nelen&Schuurmans company trip to Cologne/Köln in Germany. Model of the oldest parts of the St. Gereon church.