Containerization and False Security
Why “runs in a container” means neither isolation nor reproducibility nor well-run operations – and which steps actually close the three gaps.
Containerization and False Security
“It runs in a container” has turned into a kind of seal of quality in project conversations. The sentence is supposed to prove three things at once: the application is isolated, the build is reproducible, and operations are under control. None of these three statements follows from the packaging. A container is first of all just an ordinary process with a restricted view of the system – whether it brings security and repeatability along is decided in the Dockerfile and the startup configuration, not in the word “container”.
I have been noticing this confusion in reviews for a few months now. One team shows a clean deployment on Kubernetes 1.10, everything green, and when asked about the patch level of the base image, they shrug. Another team has been building on node:latest for a year and wonders why the CI build behaves differently from the local one. These three false senses of security are worth a closer look, because they need three different remedies.
First false sense of security: isolation#
A container is not a small virtual machine. A VM brings its own kernel and talks to the host through a hypervisor. A container shares the kernel with the host and with all other containers on the same machine. What separates it are kernel namespaces for processes, network and file system, plus cgroups for resources – useful mechanisms, but a much thinner wall than a hypervisor.
flowchart TB
subgraph VM["Virtual machine"]
A1[App] --> K1[Guest kernel]
K1 --> H1[Hypervisor]
H1 --> HW1[Host kernel<br/>+ hardware]
end
subgraph CT["Container"]
A2[App as a process] --> NS[Namespaces<br/>+ cgroups]
NS --> HW2[shared<br/>host kernel]
endThe practical consequence: if you run as root in the container, you run as root on the host. UID 0 is UID 0. User namespaces, which map container root to an unprivileged host user, have existed in Docker for a while – but they are opt-in via --userns-remap, and in the environments I have seen this year, the feature was switched on exactly zero times. As long as that’s the case, the only thing separating a root process in the container from the host is the question of whether it finds a way out.
A --privileged container is guaranteed to find that way. The flag unlocks all capabilities and grants device access – which lets you mount the host file system from inside the container. I have found the flag in production setups, set months earlier “because otherwise it didn’t run”, and never questioned again. A privileged container is not an isolation layer with a weakness; it is an open door with a curtain in front of it.
For a while now, Docker has shipped a default seccomp profile that filters risky syscalls, plus AppArmor profiles on Ubuntu hosts. That helps, but it doesn’t change the principle: the boundary is the kernel, and the kernel is shared. In January, Meltdown and Spectre additionally showed that even process boundaries can wobble at the CPU level. If you need real tenant separation, that is, if you run third-party or untrusted code, you still use VMs – or take a look at gVisor, which Google introduced in May: a userspace kernel between container and host. An exciting approach, but in July 2018 too fresh to base a production decision on.
What you can already do today is shrink the attack surface at startup:
docker run -d \
--read-only \
--cap-drop ALL \
--cap-add NET_BIND_SERVICE \
--security-opt no-new-privileges \
--memory 256m \
order-service:1.4.2
All of this has been available for years; none of it is the default. That is exactly the point: with containers, isolation is not a state but a configuration that someone has to actively set up.
Second false sense of security: reproducibility#
The second assumption goes: because the application is shipped as an image, the build is reproducible. That is only true for the finished image – the same image ID starts the same state everywhere. It isn’t true for the path there. A Dockerfile like this one is not a description but a snapshot:
FROM node:latest
WORKDIR /app
COPY . .
RUN apt-get update && apt-get install -y imagemagick
RUN npm install
ENV API_SECRET=changeme-in-prod
EXPOSE 3000
CMD ["node", "server.js"]
Almost every line here is a time bomb. node:latest is not a version but a bet on when the build happens – last year you got Node 8, for a few weeks now Node 10, and nobody made a decision. apt-get install without a version pulls whatever the mirror server offers today. npm install without a lockfile rolls the dice on dependency versions at build time; since npm 5 that’s solvable with package-lock.json, but only if the file is checked in and respected. And the secret in the ENV line now sits in the image history forever – docker history shows it to anyone allowed to pull the image.
On top of that come build args as a silent variable: an ARG that is set differently on the CI server than on the developer machine produces two different images from identical source – and because the argument isn’t anywhere in the repository, you only see the difference once you put both images side by side. So two builds of the same Dockerfile six weeks apart can differ in low-level OS layers, system packages and npm dependencies without a single line in the repository having changed. In a project this spring, we lost an afternoon over exactly this: the hotfix build for a six-week-old state behaved differently from the running system, because the base image under the same tag had meanwhile become a different one. The honest formula is: an image is a result. It only becomes reproducible once all inputs are nailed down – base image by tag or, better, by digest (node:8.11-alpine@sha256:...), system packages with versions, application dependencies via the lockfile.
Third false sense of security: operations#
The third assumption is the quietest: the container runs, so operations are taken care of. Yet an image doesn’t only freeze the application but also every library underneath – including its vulnerabilities. On a classically administered server, unattended-upgrades installs security updates at night. In a container, this path doesn’t exist. The only patch route is: new base image, new build, new deployment. If you haven’t set up this chain as a routine, you are running a fleet that goes quietly more out of date every day while the dashboard stays green.
Before that comes a question that surprisingly often goes unanswered: which images are actually running, and from which state were they built? On a cluster with a few dozen deployments, that is pure archaeology without an inventory. When the first Meltdown patches arrived in January, I saw in one project how long the inventory alone takes when tags like latest and stable are in circulation – you then literally don’t know what you are running, only that it runs.
flowchart LR CVE[CVE in the base image<br/>published] --> BASE[Base image<br/>updated] BASE --> BUILD[Rebuild<br/>of all services] BUILD --> SCAN[Image scan<br/>e.g. Clair] SCAN --> DEPLOY[Rollout] DEPLOY -. without routine<br/>the chain breaks here .- BUILD
That operations are the real weak flank was shown this year by a prominent case: in February it became known that Tesla’s Kubernetes administration console was reachable from the internet without password protection. The attackers didn’t need a kernel vulnerability and didn’t break out of any container – they settled in via the open console, deployed crypto-mining pods, throttled the CPU load and hid the mining pool behind Cloudflare to avoid attention. Same category: Docker daemons whose remote API listens unauthenticated on port 2375 and that get grazed by mining scripts one after another. In both cases, the container technology was completely innocent. What was vulnerable was the operating model around it – exposed admin interfaces, missing access control, nobody looking.
For me, this is the most important lesson from the case: the platform shifts the operational questions; it doesn’t answer them. If you used to have to know which packages are on the server, you now have to know which layers are in the images. If you used to secure SSH, you now have to secure the kubelet, API server and dashboard. The effort hasn’t disappeared; it has changed its address.
What concretely helps#
None of this is an argument against containers – it is an argument against the seal of quality. The remedies are unspectacular and have been available for quite some time. In the Dockerfile itself, it looks like this:
# build stage
FROM node:8.11-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build && npm prune --production
# runtime stage
FROM node:8.11-alpine
WORKDIR /app
RUN addgroup -S app && adduser -S app -G app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER app
EXPOSE 3000
CMD ["node", "dist/server.js"]
Multi-stage builds have existed since Docker 17.05, so they have been established for more than a year and are no longer an experiment. The effect: compilers, dev dependencies and build tools stay behind in the first stage, and only what is needed at runtime gets shipped. The USER directive makes sure the process doesn’t start as root in the first place – one line that defuses a large part of the isolation problem from the first section, because a breakout then lands with an unprivileged user. The pinned base image takes the dice out of the reproducibility question. A pleasant side effect of the small runtime stage: the image scanner has less to report, simply because fewer packages are on board – an Alpine image with a handful of libraries produces findings you actually read, instead of a list with three hundred entries that nobody opens anymore after the second week.
Beyond that, a few points have settled in as a checklist for my reviews:
- Choose small base images –
alpinefor everyday use, and for compiled languages the distroless images that Google released last year, which contain nothing except the application and its runtime dependencies, not even a shell. - Pin tags, and for production deployments also record the digest –
latesthas no place in a deployment manifest. - Hook image scanning into the CI chain: Clair or Anchore match the layers against known CVEs and turn “we should patch at some point” into a red build.
- Never put secrets into the image – not as
ENV, not asARG, not as a copied file. They end up in the history or in the build context. Secrets come from outside at runtime, for example as a Kubernetes Secret or via the orchestrator. - Trim capabilities at startup and treat
--privilegedlike open root access: anyone who wants to set it has to justify why.
None of these points is hard. The real obstacle is that they are invisible as long as everything runs – and that’s exactly why they belong in the review and not in the “we’ll do it when there’s time” category.
Bottom line#
Containers have made our delivery noticeably better: the same artifacts in all environments, fast startups, a clean interface between build and operations. What they don’t deliver is what the phrase “runs in a container” so readily claims. Isolation is a configuration, not a property – without USER, without trimmed capabilities and with a shared kernel, the wall is thinner than the word suggests. Reproducibility is a discipline, not a property – unpinned tags and packages turn the Dockerfile into a photo instead of building instructions. And operations are a routine, not a property – an image that nobody rebuilds is a server that nobody patches, only without the nightly updates.
When someone tells me today that an application is secure because it is containerized, I ask three things: as which user does the process run? How does the next build know that it builds the same thing as the last one? And who rebuilds when a vulnerability in the base is published tomorrow? The answers show pretty reliably whether the team has an operating model – or just a seal of quality.
Further reading#
- Repository introduction-docker: https://github.com/MikeBild/introduction-docker
- Docker documentation on multi-stage builds: https://docs.docker.com/build/building/multi-stage/
- Docker Engine Security Overview: https://docs.docker.com/engine/security/
- Clair, static analysis of container images: https://github.com/quay/clair
- Distroless images from Google: https://github.com/GoogleContainerTools/distroless
- Report on the Tesla incident (CNBC, February 2018): https://www.cnbc.com/2018/02/21/hackers-hijack-teslas-cloud-system-to-mine-cryptocurrency-redlock.html
Revised and translated from German with the help of AI.