Skip to content

Kubernetes Basics Under Deployment Pressure

Under the pressure of a real deployment, the Kubernetes concepts sort themselves by importance – from the pod through deployment and service to the ingress.

≈ 10 min readSoftwarePlatform

Kubernetes Basics Under Deployment Pressure

At two in the morning, a container dies. No attack, no hardware defect – the Node.js process inside tripped over a memory leak, the operating system killed it, and with the process, the container is gone too. From this moment on, a single question decides how the rest of the night goes: Who restarts it when nobody is watching? On a classically operated server, the answer is always a chain of people and semi-automation – monitoring that calls someone, a systemd unit with a restart rule, an admin with SSH access and little sleep. The lesson from such nights is the real reason Kubernetes exists: a system in which the restart isn’t a human reaction but the normal operation of a machine that continuously compares desired and actual state.

The docs explain everything except the order#

If you try to learn Kubernetes from the concept documentation, you get a vocabulary list: pods, ReplicaSets, deployments, services, endpoints, ingress, ConfigMaps, secrets, volumes, namespaces, DaemonSets, StatefulSets, jobs, CronJobs. All correctly described, all side by side with equal weight. That is exactly the problem. The docs are a reference work, not a learning order – and without an order, Kubernetes feels like a foreign language where you are only allowed to speak once you know the entire dictionary by heart.

I noticed this this year in a project in which an existing Node.js application, including its database, was supposed to move into a cluster – with a deadline, not someday. Under this deployment pressure, something interesting happened: the concepts sorted themselves. Not by the structure of the documentation, but by the question of which object the application can’t run without right now. What remained was a surprisingly short list:

  • Pod – what actually runs
  • Deployment – what keeps pods alive
  • Service – a stable address in front of unstable pods
  • Ingress – the door to the outside
  • ConfigMap and Secret – configuration out of the image

Everything else – StatefulSets, DaemonSets, NetworkPolicies, the finer points of RBAC – only came up later, when the application was already running and the pressure had eased. The examples I used to play through this order are public in the repository introduction-kubernetes: a collection of material around a small todo application with frontend, backend and database that you can follow locally with Minikube.

Desired state instead of command choreography#

The most important basic insight isn’t in any YAML example but behind them: Kubernetes is declarative. You don’t tell the cluster what to do; you describe what the world should look like when everything is fine. Three copies of this container, reachable under this name, with this configuration. A controller loop compares this desired state with reality nonstop and corrects the difference. If a pod dies, one is missing – so one gets started. Not because someone gives a command, but because desired and actual state no longer match.

flowchart LR
  M[Manifest<br/>todo-api.yaml] -->|kubectl apply| A[API server]
  A --> E[(etcd<br/>desired state)]
  C[Controller loop] -->|reads desired| E
  C -->|measures actual| P[Pods in the cluster]
  C -->|corrects difference| P

That also explains why kubectl apply -f is the central command – and not a choreography of “start this, then that, then wire both together”. On the old server, a deployment was a screenplay: shell scripts in the right order, and heaven help you if step four fails while step three has already gone through. In Kubernetes, this screenplay no longer exists. There is only a description of the desired state, and apply hands it over. What happens afterwards is the controllers’ business – even at two in the morning.

The pod: what actually runs#

A pod is the smallest unit Kubernetes starts: one or more containers that share network and lifecycle. In practice, almost always one container per pod. The second insight under pressure: pods are disposable. You rarely create them directly, you don’t look after them, you don’t repair them. A pod that dies isn’t healed but replaced – with a new name and a new IP address. Once you accept that, you understand the rest of the system almost by itself. If, on the other hand, you try to cling to individual pods, climb into them with kubectl exec and rescue state, you fight against the platform instead of working with it.

The deployment: self-healing as the core idea#

Because pods are mortal, something is needed to keep them alive. That is the deployment – for me, the actual heart of it all:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: todo-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: todo-api
  template:
    metadata:
      labels:
        app: todo-api
    spec:
      containers:
        - name: todo-api
          image: mikebild/todo-api:1.4.2
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 15

Since Kubernetes 1.9, apps/v1 has been the stable API version for this; the guesswork of the early beta versions is over. The deployment describes three things: how many copies should run, what a pod has to look like and how the cluster recognizes whether a pod is healthy. The readiness probe decides whether a pod gets traffic; the liveness probe decides whether it gets restarted. I underestimated this distinction at first – until a backend needed twenty seconds at startup to connect to the database, and without a readiness probe, requests happily ran into a system that wasn’t ready yet.

The moment Kubernetes “clicked” for me was a deliberately provoked failure:

kubectl apply -f todo-api.yaml
kubectl get pods --watch
kubectl delete pod todo-api-7d9f6c8b45-x2x7k   # simulate a crash
# a replacement pod appears within seconds
kubectl rollout status deployment todo-api
kubectl rollout undo deployment todo-api        # roll back a bad release

You delete a pod, and before you have read the output, the replacement is already there. That is the answer to the two-in-the-morning question – not as a feature promise but observable in the terminal. And rollout undo turns a failed release from an emergency into a step back to the last working desired state.

The service: a stable address in front of unstable pods#

If pods constantly come and go and change their IP addresses in the process, nobody can connect to them directly. The service solves exactly this problem: a stable virtual address and a DNS name, with a changing set of pods behind it, selected via labels.

apiVersion: v1
kind: Service
metadata:
  name: todo-api
spec:
  selector:
    app: todo-api
  ports:
    - port: 80
      targetPort: 3000

From now on, every pod in the cluster reaches the backend at http://todo-api – no matter which three pods happen to be behind it, no matter how often they have been replaced. The todo application’s frontend no longer knows any IP addresses, only this name. For me, this was the point at which the migration work paid off for the first time: service discovery, which previously consisted of maintained configuration files and hope, is simply DNS in the cluster.

The ingress: the door to the outside#

At first, services are only reachable inside the cluster. For the way out, there is the ingress – a rule that maps hostnames and paths to services, executed by an ingress controller such as ingress-nginx:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: todo-api
spec:
  rules:
    - host: todo.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: todo-api
              servicePort: 80
flowchart LR
  B[Browser] --> I[Ingress<br/>todo.example.com]
  I --> S[Service<br/>todo-api]
  S --> P1[Pod]
  S --> P2[Pod]
  S --> P3[Pod]

To be honest, the ingress is the most unfinished corner of this list: the resource has been in beta status for years (extensions/v1beta1), the controller isn’t part of the cluster but has to be installed separately, and depending on the environment – Minikube, your own cluster, GKE – things behave slightly differently. Still, it belongs to the basic equipment, because without a door to the outside, even the nicest application remains a monologue.

ConfigMap and Secret: configuration out of the image#

The last building block on the pressure list sounds unspectacular but has the biggest effect on everyday work: configuration doesn’t belong in the container image. An image that has to be rebuilt for staging because a database address is different isn’t an artifact but a hack. ConfigMaps and secrets pull these values out of the image and into the cluster:

apiVersion: v1
kind: ConfigMap
metadata:
  name: todo-api-config
data:
  DATABASE_HOST: "postgres"
  LOG_LEVEL: "info"

In the deployment, envFrom turns this into a set of environment variables, and the same image runs unchanged locally in Minikube and in the cluster at the cloud provider. With secrets, a sober look is worth it: the values are Base64-encoded, not encrypted. That separates them cleanly from the rest of the configuration and keeps them out of Git, but it isn’t a vault.

Debugging without SSH#

Something else you learn immediately under pressure: debugging works differently than on a server. There is no SSH into “the machine”, because it’s unclear which machine would even be meant. Instead, you ask the cluster what it knows about the state:

kubectl describe pod todo-api-7d9f6c8b45-x2x7k
kubectl logs todo-api-7d9f6c8b45-x2x7k
kubectl logs todo-api-7d9f6c8b45-x2x7k --previous   # logs of the crashed container
kubectl get events --sort-by=.metadata.creationTimestamp

describe shows the events – that is where the classics are spelled out in plain text: ImagePullBackOff when the image can’t be found, CrashLoopBackOff when the container dies at startup. And logs --previous shows the output of the container that just crashed – exactly what you want to know at two in the morning. That isn’t necessarily easier than before, but it is more systematic: the cluster logs what it decided, and you don’t have to piece it together from four different log files.

What the pressure doesn’t argue away#

Honesty also includes the other side. For a single small application, there are now five YAML files in the repository, together well over a hundred lines – for something that Docker Compose would describe in twenty lines. Helm promises relief through packaging, but in version 2, with Tiller, it brings its own server component with far-reaching permissions into the cluster – trading YAML volume for operational and security questions, a trade I wouldn’t make in every project yet. And even though Amazon, as the last of the big providers, has offered managed Kubernetes with EKS since June, and GKE and AKS take cluster setup off your hands: they don’t take the concepts off your hands, and a cluster you set up yourself with kubeadm is a whole operational task of its own.

That’s why the sober counter-check comes before the migration starts. A single host with Docker Compose is simply enough when a few things come together:

  • the application runs on one machine, and short outages are tolerable
  • a restart policy in the Compose file covers the nightly process death well enough
  • there is no team deploying in parallel and no load that requires scaling

Then Kubernetes mostly buys you complexity and gives little back. The point at which the calculation flips can be named surprisingly precisely: as soon as the question “who restarts it when nobody is watching?” may no longer be answered with “the one machine, hopefully”.

What I take away from it#

Deployment pressure is an unpleasant but honest teacher. It didn’t replace the concept docs for me, but it sorted them: first the pod as what actually runs, then the deployment as its guardian, then the service as a stable address, the ingress as the door to the outside, ConfigMap and secret as externalized configuration. Five objects with which a real application runs in production – the rest of the dictionary can wait until it is needed.

The deeper insight is the declarative way of thinking. The switch from the screenplay (“execute these steps”) to the desired state (“this is how it should look”) changes how you think about operations – outages are no longer incidents but differences that a controller evens out. And the two-in-the-morning question from the beginning gets its best answer: nobody restarts it. When the first person looks at the dashboard in the morning, the restart is a line in the events – happened, logged, done.

Further reading#

Revised and translated from German with the help of AI.