Tibo works as lead developer/devops guy on the https://nu.nl website (very well known in the Netherlands). 12 million hits/day.
Their IT team is growing. It gets impossible to know everything about anything.
Infrastructure provisioning is done with “terrible” (terraform + ansible…) :-) Lots of AWS. Their existing devops practices was on a solid foundation:
All infra is in code.
Terraform
“Terrible” providing mechanisms for authentication and so.
But…. setting up extra test environments is slow. Terraform has a slow feedback loop (difference beteen plan and apply). Ansible could take 20 minutes. The infra isn’t very scalable (due to needing a reasonably fixed ansible inventory). Config and secrets management becomes problematic.
So they wanted to improve something. But where to start? Lots of items are connected, so it is hard to find a starting point. A trigger point occured early 2018. Kubernetes just got ready for production + they had to start a brand new website…
An advantage of kubernetes is that it is a flexible platform. A platform to run containers on. But also a platform as a means to work on better logging, better separation, better 12factor usage, etc …. Kubernetes is a journey, not a destination.
But they didn’t want to get carried away. Not everything needed to be chopped up into miniscule nanoservices. And not every thing needed to be in kubernetes: using AWS’ managed databases and so was fine.
(He then mentioned the various components they used; I’m not that versed in kubernetes yet, so I couldn’t make a good summary of that part of his talk). For CI/CD they use Jenkins.
Some things that went wrong:
Memory usage. Make sure you set memory limits.
CPU usage. Yes: also set CPU limits. During a certain event, the
servers started to use so much CPU that the core kubernetes
components started to suffer…. Reserving memory and CPU for
kubelet also helps (--kube-reserved, --system-reserved).
Having memory limits can also be a problem. They upgraded a component which started to legitimately use a bit more memory. It hit the memory limit, got killed, started again, got killed, etc….
Apart from these problems (which you can have with other solutions, too), kubernetes runs pretty stable.
They’re looking at improvements. Helm charts. Combined with SOPS
(Secrets OPerationS) by mozilla (It manages AWS API access instead of
keys. It is versatile.) They’re quite happy with Helm. A big advantage
is that your git diff is much more informative than without helm.
Full title: “shattering worlds in a good way - from Docker to Kubernetes within an international leads company”. Ruben (https://www.n-ableconsultancy.nl) worked with a small international company to move from docker to kubernetes within four months. He provided consultancy and training to help them “navigate” the very diverse kubernetes landscape.
Fun fact: Ruben previously had a company where he build a kind of “kubernetes-light” before kubernetes even existed. Since he’s switched to kubernetes.
Kubernetes is the next step in virtualization. Kubernetes effectively virtualizes all the OSI layers. “It cuts tethers”. Kubernetes aims at applications. It gets your application running, whereever you want to run it.
Kubernetes enables change. It focuses on computing and state at the level of your application. What he means is that even the infrastructure and the provisioning of your app becomes part of the application. Your “span of control” becomes broader.
Roles will grow together that way. It will have a big influence on the IT landscape.
Back to his use case. He got a call because the company’s CTO was about to leave and almost no-one knew how everything fitted together. The first priority: find out, exactly, the current situation. What is there? Why is it there? You need a good inventory. You need to know your starting point. This phase took a month (one day a month…). The design was both brilliant, creative and crazy.
He experimented with kubernetes the weekend before he started working for the company. He suggested it and they OK’ed it. As a proof of concept they installed kubernetes on bare-metal at https://hetzner.de .
On day one of the proof-of-concept Ruben started them on experimenting right away. A deep dive. One of the developers started to turn red: “I don’t know anything about networking, I can’t work on this” etc. He felt insecure and got angry about it. A nice start :-)
But… in the end they got the front-end running in kubernetes, even though the original setup was quite elaborate. At the end of the three weeks, the developer started to get enthousiastic.
Step two was to get it high available. They switched from self-hosted to google (GCE, GKE). Google seemed to be the best for kubernetes hosting.
For every component, they tried multiple solutions. For central monitoring, they went with datadog, for instance.
Step three was becoming complete. Security, authorization, CI/CD (gitlab). And migrating their big data environment to GCE.
Step four: functional and load testing. The company wanted to go live immediately, but he managed to convince them to first make sure everything worked. They redirected some of the existing application’s load over to the new solution, which worked OK.
They did load testing and it turned out that kubernetes could handle 10 times as many requests per second. Testing like that gives you confidence that everything will work just fine.
So: time to go live! They made a mistake and accidentally deleted the entire cluster an hour before going live…. So they stayed on the existing system. One of the developers went home, re-created the cluster, started up the three-hour-mongoDB-sync, went to bed and the next day they actually went live.
Important point: being able, as a developer, to re-create the entire environment was a big booster for his confidence in kubernetes.
Another important point: they now understand their entire environment. And it is much more secure. Before, they had a black box, now they can see what’s going Ona.
Kubernetes forces you to be explicit. It helps you build a better and safer environment.
Kubernetes is a change enable and a game breaker. It gives control and power back to companies that need their IT to be succesfull. It can make or break a business’s competitiveness.
Ruben thinks kubernetes will impact the way we all work with IT and the way way consume IT in the same way Ford’s assembly line changed the auto industry.
My name is Reinout van Rees and I work a lot with Python (programming language) and Django (website framework). I live in The Netherlands and I'm happily married to Annie van Rees-Kooiman.
Most of my website content is in my weblog. You can keep up to date by subscribing to the automatic feeds (for instance with Google reader):
