Skip to content

Secrets in the Build Are Product Decisions

Secrets in the build are a trust problem: who may see, store and deploy what, and when?

≈ 10 min readSoftwareDevOpsArchitecture

Secrets in the Build Are Product Decisions

$ docker history --no-trunc myapp:staging
IMAGE      CREATED BY
<missing>  /bin/sh -c #(nop) COPY file:3f2a… in /app/.env
<missing>  /bin/sh -c #(nop) ENV DATABASE_URL=postgres://app:S3cr3t@db.prod.internal:5432/app

Two lines of layer metadata, readable by anyone with pull rights on the registry. In one project, half the configuration lived in an .env file that was copied into the image via COPY. That went well for months, until someone pulled the image from the registry, ran docker history and read the database URL including the password in plain text. No attack, no exotic bug – just an artifact that knew more than it was allowed to.

The mistake wasn’t forgotten encryption. The mistake was an unspoken boundary: the team had never decided whether a build artifact may know secrets. It had just copied a tutorial. And that is exactly where this stops being an operations question. Where a token, a password or a certificate is created, stored and injected into an artifact determines who may deliver without checking with anyone and who has to wait. That decides a team’s speed, friction and error rate – long before any tool has been chosen.

The real question is trust, not encryption#

Encryption solves a narrow problem: protecting data at rest and in transit. It doesn’t answer the question that gets expensive in everyday work – who in the process gets their hands on the secret at all. A KMS-encrypted value that every CI job may decrypt is convenient and still an open boundary.

Before I choose a tool, I clarify four questions. Which component is authoritative for the value? Who may read it, and in which environment? When is it read – at build time or at runtime? And what has to be traceable later if it is compromised? These answers decide rotation and blast radius. Encryption alone answers none of them.

Build time and runtime are two different trust zones#

The most important boundary runs between build and runtime. A build produces an artifact that travels through several environments and lands in a registry that more people often have access to than you’d think. Everything baked into this artifact is therefore exactly as confidential as the registry – and no more.

The classic anti-pattern looks harmless:

# ANTI-PATTERN: the secret becomes part of a shared, promotable artifact
COPY .env /app/.env
# ANTI-PATTERN: build args stay recoverable via `docker history`
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc \
  && npm ci \
  && rm .npmrc

The rm fakes security. The token remains visible in the layer metadata, and the .env travels along as a layer of its own. Both survive every later docker run in every environment. The real problem is not the command but the mixed trust zone: an artifact that is supposed to be identical in staging and production suddenly carries production secrets.

The architectural answer is a clean separation. The build produces an environment-neutral artifact without secrets. Configuration is only added at runtime, injected from a source that is separate per environment. A multi-stage build makes this intention visible in the code:

# build stage produces an artifact that knows nothing about any environment
FROM node:12 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# runtime stage receives config from the environment, never from the image
FROM node:12-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
CMD ["node", "dist/server.js"]

The same thinking applies to build secrets the build really needs – a private registry token, for example. They come in via a secret mount that isn’t persisted in the final image, instead of via an ARG that stays in the history. The principle remains: a secret that only the build needs must not reach the runtime artifact.

Parameter Store as an interface, not a database#

Once the separation is in place, Parameter Store becomes interesting. Its value comes from the structure it enforces, not from the fact that it stores values. The hierarchical path is the real lever: it encodes the environment, and the environment is the boundary where permissions take hold.

# resolve runtime config just-in-time, scoped by environment path
aws ssm get-parameters-by-path \
  --path "/myapp/production/" \
  --with-decryption \
  --query "Parameters[].{Name:Name,Value:Value}"

The path /myapp/production/ is more than cosmetics. It makes it possible to express least privilege cleanly: a deploy job for staging gets an IAM policy that points exactly at staging/* and can’t even read production.

{
  "Effect": "Allow",
  "Action": "ssm:GetParametersByPath",
  "Resource": "arn:aws:ssm:eu-central-1:123456789012:parameter/myapp/staging/*"
}

This anchors the domain intention in the infrastructure instead of in a sentence in a wiki. The value is resolved at runtime and never written into the repo or the image:

#!/usr/bin/env bash
set -euo pipefail
# inject secrets at container start, keep them out of the image and the repo
export DATABASE_URL="$(aws ssm get-parameter \
  --name "/myapp/production/database-url" \
  --with-decryption --query "Parameter.Value" --output text)"
exec node dist/server.js

Whether Parameter Store, Secrets Manager or a vault sits behind it is the smaller question. The bigger one is that access is separated per environment, auditable and rotatable. The tool expresses the boundary – it doesn’t create it.

Sketch#

flowchart LR
  Repo[Code<br/>without secrets] --> Build[Build<br/>reproducible artifact]
  Build --> Registry[Artifact<br/>same for every environment]
  Store[Parameter Store<br/>separate per environment] --> Runtime
  Registry --> Runtime[Runtime<br/>injects config just-in-time]

The sketch shows the one statement that matters: the left path is identical everywhere, the secret path comes separately and is only added at the very right. Exactly at this junction it is decided whether a team can deliver fast and safely or trades one off against the other.

The trade-offs you really weigh#

No approach is free. An environment-neutral artifact plus runtime injection needs a working secret source in every environment and makes container startup depend on another service. If Parameter Store fails, the service doesn’t start – that is a deliberate price for the boundary, not an oversight. You cushion it with caching and clear error messages, not by baking secrets back in after all.

Conversely, the convenience of the .env in the image is real: one artifact, no dependency at startup, easy to reproduce locally. The price for it only comes due later – at rotation, at a leak, at the question of who had the staging image with production credentials in their hands. Architecture here means moving the price to a place where you can control it, instead of shifting it into a later incident.

The second trade-off concerns centralization. A central secret source simplifies rotation and auditing but itself becomes a critical path and a coveted boundary. That’s why the path and IAM structure is not a formality: it is the difference between “one service knows everything” and “every service knows exactly its own”. The actual product decision lies between the two.

The failure path decides, not the happy path#

A secrets architecture is only robust once the failure path is described. In demo mode, every approach looks clean. It gets interesting exactly where systems normally break.

Rotation is the most honest test. Can I change a compromised value without rebuilding an image and promoting it through all environments? With runtime injection, that is a parameter update plus a restart. With a baked-in secret, it is a full release cycle under time pressure – the worst moment for a long road.

Rollback is the second test. With clean separation, a rolled-back image only brings back old code, not old secrets. If you mix the two, rolling back the code also silently rolls back expired or rotated credentials. And third, the audit: if a value leaks, I want to reconstruct which job read it when. A central store with access logs answers that; a file in a layer doesn’t. These questions seem dry, but they are the difference between a demo and operations.

The business outcome is speed with controlled risk#

The reason I treat secrets as a product decision is measurable. When a team no longer has to pass secrets along manually, a whole class of waiting time disappears. A developer deploys to staging without someone from the ops team copying a password via chat – the IAM policy allows exactly this path and nothing beyond it. That shortens the cycle from idea to running change and at the same time lowers the risk, because access is narrowly scoped. Speed and security are not in conflict here; the right boundary delivers both.

Less friction also comes from the boundary living in the code and not in one person’s head. New team members read from the path structure and the policy what is allowed instead of having to ask. And quality goes up, because the most expensive mistake – a secret in a shared artifact – structurally can no longer happen, instead of only being prevented by discipline. That is exactly what I mean by developer productivity: not typing faster, but waiting less often and cleaning up less often.

The effect is not a soft promise; it can be observed. I measure it by three numbers: the time from “the password has to be changed” to “changed and in operation”, the number of manual handoffs per release, and how many environments a single credential can read. Each of these numbers drops once the build-runtime boundary is in place – and when in doubt, each is more convincing than the claim that things are now more secure.

Typical mistakes#

The first mistake is confusing the tool with the solution. A secret store is not an architecture, KMS is not an access boundary, and a --build-arg is not confidentiality. These tools can express good boundaries, but they don’t create them automatically. If you introduce the store without drawing the build-runtime boundary, you have only moved the file to a different place.

The second mistake is overestimating the happy path. Many setups look clean on the first deploy and only show their weakness at the first urgent rotation. The third is treating everything as equally confidential. A feature flag configuration and a database password need different boundaries; if you throw both into the same pot, you make the harmless case expensive and the critical case convenient.

What I check#

Three checks are usually enough for me. First: is a build artifact environment-neutral, or does it know secrets? A docker history and a look at the layers answer that in a minute. Second: is access separated per environment, so that a staging job can’t read production? Third: can I rotate a secret without rebuilding, and trace access after the fact?

If one of these questions ends with a no, part of the architecture is still missing – not another tool, but a deliberately drawn boundary. I wouldn’t make a big initiative out of it but start with a small, verifiable result: a multi-stage Dockerfile, an IAM policy with path scope, a startup script that resolves the value just-in-time. Concrete enough that another developer can meaningfully disagree. Agreement is good, but disagreement often reveals the hidden assumption.

Conclusion#

Storing a secret safely is the smaller half of the work. The bigger half is the reasoning for why the boundary between build and runtime runs exactly there. What matters is the concrete risk, the permitted access and the question of whether a compromised value can be rotated independently of the artifact. The name of the secret store comes after that.

Revised and translated from German with the help of AI.