Djangocon EU: partitioning very large tables with Django and Postgresql - Tim Bell¶
(One of my summaries of the 2026 Djangocon EU in Athens).
The company where he works (Kraken) had an issue with vacuuming. 2TB table, 9 billion rows. 20-30 million row updates per day. The necessary vacuuming would take 16-20 hours every day.
Partitioning seems a good idea. There aren’t a lot of resources to read or watch. One of the best is a talk by Karen Jex.
When to partition? Size over 100 GB or more than 100 million rows. You should rather not partition unless you really need to. In their case, vacuuming provided the reason. Vaccuuming a partial table takes much less time.
Table partitioning: each row belongs in a partition based on a “partition key”, the value and a set of bounds/rules. So “id 1-1000 in partition 1, 1001-2000 in 2”.
Which partition method to choose? Range, list, or hash? Look at your application, really design it from the ground up to work with partitioning in mind. How do you access your data? How do you want to purge data?
Django abstracts away the complexity of the database, but the abstraction can be a bit leaky. Database partitions can be something that leaks through. Partitioned tables don’t work with django migrations when they create indexes or constraints, for instance.
A partition key needs to be part of all unique constraints, including primary keys. So if you need to partition on anything else than the ID, you need to use composite keys. And that’s something that Django supports, but not really likes.
https://github.com/pgpartman/pg_partman is a well-regarded Postgresql extension for handling partitions. They also wrote their own “psycopack” extension for psycopg at https://github.com/kraken-tech/psycopack . This especially helps with migrating from a non-partitioned database to a partitioned one.
(Note: see also the previous talk, about database scaling )
Unrelated photo explanation: a recent trip to the “Modellbundesbahn” in Germany. The Altenbeken viaduct in 1:87 scale.