PyGrunn: platform engineering, a python perspective - Andrii Mishkovskyi¶

Tags: pygrunn, python

(One of my summaries of the 2024 Dutch PyGrunn python&friends conference in Groningen, NL).

There’s no universally accepted definition of platform engineering. But there are some big reasons to do it. “Shift left” is a trend: a shift from pure development into more parts of the full process, like quality assurance and deployment.

But… then you have an abundance of choice. What tools are you going to use for package management? Which CI program? How are you going to deploy? There are so many choices… Jenkins or github actions or gitlab? Setuptools or poetry? Etcetera.

You have lots of choices: hurray! Freedom! But it takes a huge amount of time to make the choice. To figure out how to use it. To get others to use it. There of course is a need to consolidate: you can’t just support everything.

How do you do platform engineering?

• You observe. Observation: you need to be part of a community where you gather ideas and tips and possible projects you can use.

• You execute. Execute: you have to actually set things up or implement ideas.

• You collect feedback. Talk with your colleagues, see how they’re using it. You really have to be open for feedback.

Some “products” that can come out of platform engineering to give you an idea:

• Documentation.

• Self service portal.

• Boilerplates: templates to generate code. They use cookiecutter (https://cookiecutter.org) for it.

• APIs.

In his company, they now support three cookiecutter templates. You shouldn’t have too many. And you should try to keep them as standard and simple as possible. He suggests using a happy path: a basic set of tools that are supported and suggested. That’s where you aim all your automation and templates at: you’re allowed to do something else, but you’ll have to support and understand it yourself.

Cookiecutter has some drawbacks. Replaying templates is brittle. There’s no versioning built-in. Applying it incrementally is painful.

Dependency management: they now use pyproject.toml for everything. Poetry instead of setuptools+pip-tools. And as platform team they recommend Renovate to the actual software teams, which helps keeping your code and dependencies up to date. Renovate turned out to be quite popular.

Regarding API: one of the services the platform team at his company provides is orchestrating the authorization used by the various (micro)services. Previously, it was mostly a manual process via a django admin website. “Chatops”: they’d get lots of requests for access. Lots of manual work, no clarity which “scopes” (+/- permissions) there are, no clarity which permissions were actually handed out.

They fixed it with automation based on yaml files:

• Apps deoclare their scopes.

• Maintainers approve access.

• A CI/CD pipeline checks if the access rights are in place.