(One of my summaries of the one-day Pycon NL conference in Utrecht, NL).
Full title: how bazel streamlines python development: stories from the Uber monorepo.
Monorepo: a single repository that contains multiple distinct projects, with well-defined relationships. There are advantages to monorepo’s like developer productivity: code sharing, component reuse, reducing duplication. Atomic commits across components: you fix everything everywhere in one commit. And you have consistency and a common way of working.
Bazel: https://bazel.build/ . A build system that allows you to define tools and tasks
by writing code. With an ever-growing build-in set of rules to support popular langauges
and packages. Bazel’s language is called starlark
, which is inspired by python. Some
semantics differ, but the behaviour is mostly the same.
Do you even need a build tool for python? Well, perhaps you have compiled cython modules. Or a javascript frontend. Code generation? In any case you probably need to generate deployable artifacts like wheels or debian packages.
They used git-filter-repo to help merge an existing repo into a monorepo, preserving the history. (Correction: I originally said they wrote it it, but Alexey told me it was an existing project.)
There is support for running tests. And optionally caching test results to prevent re-running unneeded tests (important in a huge code base).
Caching is hard, that’s why Bazel emphasises hermeticity. When given the same input source code and the same configuration, the result should always be the same. It should be hermetically isolated from outsice influences.
Some challenges for monorepo’s:
The ecosystem might not be hermetically closed. Some projects don’t publish binary wheels and depend on OS libraries. Those OS libaries might not be under total control.
Flaky test. Sometimes you have tests that don’t always fail. They depend on the speed of execution or a race condition or whatever. This wreaks Bazel’s approach a bit.
Lack of tests. If something is important to you, you should test it.
Some things they want to improve:
IDE support.
Static typing. Mypy is not fast. Bazel could perhaps help here.
Shared build cache accross multiple environments. Perhaps even accross the entire organisation?!?
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):