Pygrunn: Orchestrating Python projects using CoreOS - Oscar Vilaplana¶

Tags: pygrunn, python

(One of the summaries of the 2015 Pygrunn conference)

(Note: Oscar Vilaplana had a lot of info in his presentation and also a lot on his slides, so this summary is not as elaborate as what he told us. Wait for the video for the full version.)

“Orchestrating python”: why? He cares about reliability. You need a static application environment. Reliable deployments. Easy and reliable continuous integration. And self-healing. Nice is if it is also portable.

A common way to make scalable systems is to use microservices. You compose, mix and extend them into bigger wholes. Ideally it is “cluster-first”: also locally you test with a couple of instances. A “microservices architecture”.

Wouldn’t it be nice to take the “blue pill” and move to a different reality? One in where you have small services, each running in a separate container without a care for what occurs around it? No sysadmin stuff? And similary the smart infrastructure people only have to deal with generic containers that can’t break anything.

He did a little demo with rethinkdb and flask.

For the demo it uses coreOS: kernel + docker + etcd. CoreOS uses a read-only root filesystem and it by design doesn’t have a package manager. Journald for logging (it automatically captures the stdout). Systemd for managing processes.

etcd? It is a distributed configuration store. It has a http API.

Also: “fleet”. “systemd for services”. It starts up the containers. It coordinates accross the cluster. It will re-start containers if they die.

How do we get containers to talk to each other? They’re containerized… For that there’s “flannel”: “dhcp for containers”. Per-cluster specific subnet. Per-machine smaller subnet. The best system to run all this is Kubernetes.

Kubernetes uses “replication controllers”. The basis is a “pod”, from which multiple replicas are made, depending on the amount of instances you need.

He then showed a demo. Including a rolling update. Nice. Similarly for a rethinkdb cluster where he increased the number of nodes halfway the demo. Nice, too.

In development, it might be easy to use “nspawn” instead of docker. It is mostly the same, only less isolated (which is handy for development).