Skip to content

AWS Deployment Pipelines Are Architecture

Deployment pipelines are architecture because they determine how often, how safely and how reversibly a system is changed.

≈ 10 min readSoftwareAWSCloud

AWS Deployment Pipelines Are Architecture

The time from a merged commit to the moment the change takes effect in production is under a quarter of an hour for some teams and a whole quarter for others – often with comparable application code. This span appears in no architecture diagram. It comes about where a change makes its way: in the steps between commit and runtime, in the controls it has to pass, and in what is left when it has to be taken back. The way there therefore belongs in the same design discussion as the choice of a database – and needs the same justification.

The concrete occasion for this text is a small GitHub Action that does nothing more than deploy an AWS CDK project. A few lines of YAML. And yet exactly this building block determines which error shows up early, which only in production – and who is allowed to trigger a change to the infrastructure at all.

The pipeline determines changeability#

Architecture answers the question of what can be changed independently of each other. That is exactly the core question of a pipeline too. A system that can only be deployed in a quarterly release has a different architecture from one that rolls out the same codebase twenty times a day – even if both contain identical application code. Changeability isn’t only in how modules are split, but in the path from a commit to the runtime.

On AWS, this path has become especially visible since infrastructure itself became code. With the CDK, I describe Lambda functions, API gateways, tables and permissions in TypeScript. cdk synth translates that into a CloudFormation template, cdk deploy brings it into an account. This shifts something fundamental: the question “what does my infrastructure look like?” and the question “how does a change get there?” coincide. Both live in the same repository and go through the same review and the same pipeline. I wrote about this in more detail in Infrastructure in a programming language; here I’m interested in the step after that – what happens when this code is rolled out automatically.

A minimal stack looks like this:

import * as cdk from "@aws-cdk/core";
import * as lambda from "@aws-cdk/aws-lambda";
import * as apigw from "@aws-cdk/aws-apigateway";

export class ApiStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const handler = new lambda.Function(this, "ApiHandler", {
      runtime: lambda.Runtime.NODEJS_12_X,
      handler: "index.handler",
      code: lambda.Code.fromAsset("dist/api"),
    });

    new apigw.LambdaRestApi(this, "Api", { handler });
  }
}

The example is deliberately small. It isn’t meant to show a complete application but to mark the boundary: everything here is part of the deployable artifact from now on. If I change the runtime, extend a permission or rebuild the API, it takes the same path as a code change. The exciting decision isn’t how many constructs I write, but which control stands between this code and production.

The “CDK Action” as the smallest honest building block#

The simplest way to automate this deployment is a GitHub Action that reacts to every push to main:

name: deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
        with:
          node-version: 12
      - run: npm ci
      - run: npm test
      - run: npx cdk deploy --require-approval never
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: eu-central-1

That looks harmless, but it makes one architecture decision after another. on: push means: every merge to main is a deployment – the trunk is the delivery, not just an intermediate state. The npm test step before cdk deploy draws a quality line: whatever is red doesn’t go out. --require-approval never skips the CDK’s interactive prompt for security-relevant changes – convenient, but it shifts responsibility for IAM changes entirely to the review before. And the two secrets determine with which permissions this pipeline acts in the account; in practice, that’s where most of the unspoken power lies.

That is exactly why this small building block is more revealing than any big reference pipeline. You can walk through it with the team in five minutes, and every line stands for a conscious or unconscious decision about frequency, safety and responsibility.

The review point comes before the deploy, not after#

With application code, we review the diff. With infrastructure, many people don’t – and are surprised when a merge opens a security group on the side or replaces a table. The CDK has a good tool for this that becomes the actual review artifact in a pipeline:

# what will actually change in the account
cdk diff ApiStack

cdk diff doesn’t show my TypeScript diff but the difference between what is in the account and what I want to deploy – including the IAM statements CloudFormation derives from it. In a clean pipeline, this step runs automatically and attaches the result to the pull request. That way, a person decides about the effect, not about the source code. That’s a methodological difference with a direct operational effect: a reviewer who sees “this change adds iam:PassRole” stops it before it lands in the account. A reviewer who only reads stack.ts often misses it.

This is what the path looks like as a whole:

flowchart LR
  Commit --> Build["Build & Test"]
  Build --> Synth["cdk synth<br/>CloudFormation template"]
  Synth --> Diff["cdk diff<br/>review artifact"]
  Diff --> Approve{"Approval?"}
  Approve -->|yes| Deploy["cdk deploy"]
  Approve -->|no| Stop["Stop"]
  Deploy --> Observe["Metrics &<br/>alarms"]
  Observe -->|failure| Rollback["Rollback to<br/>previous revision"]

Every node in this sketch is an adjustment knob. Where the approval sits decides speed versus control. Whether anything is observed after Deploy at all decides whether a bad release is noticed before the customer reports it. And whether there is a real Rollback path decides whether a change is reversible – or just hopefully right.

The same artifact through several environments#

As soon as more than one environment is involved – staging and production, often in separate AWS accounts – the pipeline becomes the actual definition of what “promoting” means. That only holds up under one condition: the same checked artifact moves through the stages, not a second build with slightly different ingredients. With the CDK, that means synthesizing the same app with a different context.

const app = new cdk.App();

new ApiStack(app, "ApiStack-staging", {
  env: { account: "111111111111", region: "eu-central-1" },
});

new ApiStack(app, "ApiStack-prod", {
  env: { account: "222222222222", region: "eu-central-1" },
});

This puts the environment topology in the same repository as the code, and it goes through the same review. The methodological gain is trust: when staging and production are synthesized from one source, staging actually tests what later runs in production – and not a plausible copy that differs in three places. It’s exactly these differences that otherwise only surface in the most expensive account. Splitting into separate accounts isn’t an operations detail but an architecture boundary: a deploy to staging can’t accidentally touch production because it simply lacks the permissions in the other account.

Trade-offs you have to name#

The interesting work on a pipeline isn’t getting it to run but laying its compromises open. Three of them come up regularly:

  • Auto-deploy on main versus a manual gate. The automatic path shortens lead time drastically but requires a reliable test net and good alerts. The gate gives a feeling of safety, but in practice it collects changes into large, risky batches – the most common reason for rare, fear-ridden releases.
  • One stack versus many stacks. One big stack is easy to deploy, but every change touches everything. Separate stacks for API, data and static frontend let you keep risk small and roll out parts independently – at the cost of more coordination at the interfaces.
  • Rollback via redeploy versus fix-forward. CloudFormation can roll back automatically on a failed deploy. But that only holds as long as the change hasn’t already migrated data or extended a schema. From then on, “just go back” is an illusion, and you need additive, backward-compatible steps instead of an undo button.

None of these decisions is universally right. But they belong in an architecture conversation and in a short ADR, not in a YAML file someone committed a year ago that no one has questioned since.

Why a CTO should care#

For technical leadership, all this translates into numbers that have had an established vocabulary since “Accelerate.” Four metrics describe the performance of delivery fairly completely, and all four are determined by the pipeline, not by the application code:

  • Deployment frequency – how often anything goes live at all. An auto-deploy on main raises it by orders of magnitude compared to a manual quarterly gate.
  • Lead time – how long from “code done” to “in production.” cdk diff in the PR plus an automatic deploy pushes that down from days to minutes.
  • Change failure rate – what share of releases causes a failure. The test step and the diff review before the deploy lower it without sacrificing frequency.
  • Time to restore – how quickly the normal state is back. A real, practiced rollback path is the difference here between minutes and a lost afternoon.

The important thing about these four: they make different sides of delivery capability visible. Small changes limit the scope of a review and of a possible failure; automated checks shorten feedback; a practiced recovery path lowers the time to repair. Whether a pipeline actually has this effect only shows in Lead Time for Changes, Change Failure Rate and Time to Restore Service in your own system. It is therefore not an add-on for the operations side but part of the architecture.

How I check a pipeline#

A pipeline design only holds up once the failure path is described too. The questions that uncover the most aim exactly there. What happens if cdk deploy aborts in the middle of a change – is a half-finished state left behind? Is the deploy idempotent enough that a second run heals it? Where is the line beyond which a rollback is no longer enough because data has already been migrated? And quite simply: who may trigger this pipeline, and is that logged in a traceable way?

What I run into most often is overrating the happy path. In the demo account, everything runs cleanly. It gets interesting with partial failures, permissions that are just barely insufficient, resources that CloudFormation wants to replace instead of change, and that one deploy that hangs at 5 p.m. on a Friday. That’s exactly where it shows whether the pipeline model holds up – and whether “reversible” is a real promise or just a word in a presentation.

Conclusion#

The temptation is to treat a pipeline as a tool question: which Action, which runner, which provider. The real substance lies elsewhere. A pipeline encodes how often a team may change its mind, how safely it does so and how easily it takes back a mistake. These are architecture properties, and they don’t come from the tool’s name but from the few deliberate decisions behind it – auto-deploy or gate, one stack or many, undo or fix-forward. If you name these decisions and measure their effect, you can change bottlenecks in a targeted way. If you leave them to an old YAML file, you make them anyway – only unconsciously and usually at the worst possible moment.

Further reading#

Revised and translated from German with the help of AI.