Skip to content

Cloud Guardrails Instead of Cloud Gatekeeping

Why central approval processes in the cloud breed shadow infrastructure and how technically enforced guardrails – SCPs, Permission Boundaries, Config Rules, budgets – give teams their freedom of movement back.

≈ 10 min readSoftwareArchitecture

Cloud Guardrails Instead of Cloud Gatekeeping

An approval process with a three-day wait doesn’t create control. It creates shadow infrastructure. The team waiting for approval of a DynamoDB table isn’t really waiting – it finds a way around: a colleague’s private AWS account, the project lead’s credit card, the “temporary” bucket in the sandbox account that still holds production data two years later. The process that was supposed to prevent risks shifts them exactly to where nobody looks anymore: no review, no central logging, no cost control, no backups.

On a project last year, I saw this up close. A central cloud team, a Jira board, every new resource a ticket. Official lead time: three days. Actual lead time with follow-up questions: a week or more. The cloud team was neither malicious nor lazy – it was simply too small for twelve product teams, and every ticket needed context that only the requesting team had. What struck me was that the actual discussion was never “allow or forbid.” It was almost always: “In which region? With which tagging? Who pays for it?” In other words, questions that can be phrased as a rule. And whatever can be phrased as a rule, a machine can enforce – without any waiting time.

What the ticket process actually protects#

Before you abolish the process, it’s worth looking at which legitimate concerns it carries. On that project, there were four at the core:

  • Cost: nobody should accidentally start an expensive cluster with an exotic instance size.
  • Data sovereignty: data stays in European regions, period.
  • Security baselines: no public S3 bucket, no disabled CloudTrail, no wide-open security groups.
  • Privilege escalation: anyone who may create roles could otherwise grant themselves arbitrarily far-reaching permissions.

All four concerns are real. But none of them needs a person who reads a ticket three days later. They need boundaries that apply at the moment of the API call – not at the moment of approval. The ticket process checks a declaration of intent; it never again checks what actually happens afterward. The guardrail checks every single action, every time.

AWS itself now has a name for this way of thinking: Control Tower, generally available since summer 2019, literally calls its preconfigured rules “guardrails.” So the term isn’t a buzzword from conference slides; it comes from the platform’s product practice. More interesting than the name is the distinction Control Tower makes: preventive guardrails prevent an action, detective guardrails detect a deviation. This distinction carries the entire architecture.

flowchart LR
  A[Team changes<br/>infrastructure] --> B{Preventive:<br/>SCP / Boundary}
  B -->|violates| C[API call rejected<br/>immediately]
  B -->|allowed| D[Resource<br/>is created]
  D --> E{Detective:<br/>AWS Config}
  E -->|compliant| F[Nothing happens]
  E -->|deviation| G[Alarm or<br/>auto-correction]

Hard limits: Service Control Policies#

The outermost guardrail is Service Control Policies in AWS Organizations. An SCP applies to entire accounts or organizational units and overrides every IAM policy in the account – not even the account administrator gets past it. That’s exactly why only the rules that are truly non-negotiable belong here. On that project, there were two: allowed regions and a short list of forbidden services.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideAllowedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["eu-central-1", "eu-west-1"]
        }
      }
    },
    {
      "Sid": "DenyUnapprovedServices",
      "Effect": "Deny",
      "Action": ["gamelift:*", "robomaker:*", "chime:*"],
      "Resource": "*"
    }
  ]
}

Two details matter to me here. First, the NotAction exceptions: IAM, CloudFront and Route 53 are global services, and their API calls technically run through us-east-1 – if you forget them in the region lock, you lock yourself out. We learned that the hard way, luckily in a test account. Second, the nature of the list of forbidden services: it’s short and justified, not an allowlist of a hundred entries. An SCP that forbids everything except what someone once requested is just the old gatekeeping in JSON form. The guardrail marks the edge of the road, not the only permitted lane.

Important for understanding: an SCP grants no permissions itself. It defines the upper limit of what IAM policies in the account can effectively allow at all. Within this limit, the teams move freely – and that’s exactly the point.

Self-service for IAM: Permission Boundaries#

The most uncomfortable question on the project came from the security officer: “If developers may create their own IAM roles, can’t they give themselves any permissions they want?” Without a countermeasure: yes. Anyone with iam:CreateRole and iam:AttachRolePolicy can build a role with administrator permissions and assume it. That’s why IAM was historically the last territory of the central team – and the most common reason for waiting, because every Lambda function and every pipeline needs a role.

Since 2018, there have been Permission Boundaries for this: a managed policy attached to a role as an upper limit. The effective permissions are the intersection of the boundary and the attached policies. The trick lies in self-reinforcement – the boundary only allows creating new roles if the new role carries the same boundary again:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowWorkloadServices",
      "Effect": "Allow",
      "Action": [
        "s3:*", "dynamodb:*", "lambda:*", "sqs:*",
        "logs:*", "cloudwatch:*", "apigateway:*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyCreateWithoutBoundary",
      "Effect": "Deny",
      "Action": ["iam:CreateRole", "iam:CreateUser"],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "iam:PermissionsBoundary": "arn:aws:iam::111122223333:policy/workload-boundary"
        }
      }
    },
    {
      "Sid": "DenyBoundaryTampering",
      "Effect": "Deny",
      "Action": [
        "iam:DeleteRolePermissionsBoundary",
        "iam:DeletePolicy",
        "iam:CreatePolicyVersion"
      ],
      "Resource": "arn:aws:iam::111122223333:policy/workload-boundary"
    }
  ]
}

With this, the teams could create roles for their services without anyone having to approve them – but no role could ever allow more than the workload-boundary. In the CDK, which we used to describe the infrastructure anyway, that’s one line per role:

import * as iam from '@aws-cdk/aws-iam';

const boundary = iam.ManagedPolicy.fromManagedPolicyName(
  this,
  'WorkloadBoundary',
  'workload-boundary'
);

const serviceRole = new iam.Role(this, 'OrderServiceRole', {
  assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
  permissionsBoundary: boundary,
});

That got rid of the three-day wait for “please create a role for the order service.” Not because someone approved faster, but because there was nothing left to approve: the boundary was already drawn before the request came up.

Detect, don’t just prevent: AWS Config#

Preventive rules have a gap: they can only forbid what can be expressed as an IAM condition. “No S3 bucket without encryption” or “no security group with 0.0.0.0/0 on port 22” are properties of a resource, not API actions – for these, you need the second kind of guardrail. AWS Config records the state of all resources and continuously checks it against rules. On deviation: an alarm and, where it can be done safely, automatic correction.

import * as config from '@aws-cdk/aws-config';
import * as budgets from '@aws-cdk/aws-budgets';

new config.ManagedRule(this, 'BucketsNotPublic', {
  identifier: 'S3_BUCKET_PUBLIC_READ_PROHIBITED',
});

new config.ManagedRule(this, 'VolumesEncrypted', {
  identifier: 'ENCRYPTED_VOLUMES',
});

new budgets.CfnBudget(this, 'TeamMonthlyBudget', {
  budget: {
    budgetType: 'COST',
    timeUnit: 'MONTHLY',
    budgetLimit: { amount: 500, unit: 'USD' },
  },
  notificationsWithSubscribers: [
    {
      notification: {
        notificationType: 'ACTUAL',
        comparisonOperator: 'GREATER_THAN',
        threshold: 80,
      },
      subscribers: [
        { subscriptionType: 'EMAIL', address: 'team-orders@example.com' },
      ],
    },
  ],
});

For me, the budget belongs in the same category, even though it’s technically a different service. Fear of costs was the emotional core of the old approval process – and a budget alarm at 80 percent answers it better than any ticket, because it reacts to actual spending instead of estimated spending. On that project, each team got its own budget with its own notification. The discussions about costs didn’t become fewer, but they took place with numbers instead of guesses, and they took place within the team – not between the team and a gatekeeper.

The second benefit of Config is less conspicuous: the history. If a security group looks different today than yesterday, Config shows when what changed – that saved us hours in more than one debugging session, quite apart from the rules. An approval process only knows what was requested; Config knows what actually happened.

I have become more cautious with automatic correction. A rule that immediately removes a public bucket policy is a security gain. A rule that deletes resources at night whose tags are wrong once cost us a morning of debugging, because a legitimate prototype had disappeared. Detecting and reporting is almost always right; intervene automatically only where the correction is guaranteed not to destroy anything.

Guardrails are code, not click configuration#

The point where such setups tip over, in my experience, isn’t the technology but the maintenance. An SCP that someone clicked together in the console a year ago is just as opaque as the old approval process – except that now nobody knows why port 22 is blocked and who may change that. Guardrails are infrastructure, and for infrastructure, what I already described when reviewing infrastructure code applies: it should be versioned, reviewed and changed traceably.

For us, that specifically means: SCPs, boundary policies and Config Rules live as code in the repository, changes go through a pull request, and the review includes someone from a product team alongside the platform team. The pull request replaces the ticket – with one crucial difference: what gets discussed is a permanent rule for everyone, not a one-off approval for one team. So the discussion is worth it, and its result keeps having an effect. I described how such rules can also be checked when synthesizing your own stacks in Policy as Code for CDK – together, the two give you two checkpoints: one early in the build, one at the account boundary that can’t be bypassed.

By the way, the pattern isn’t AWS-specific. The same thing is happening right now in the Kubernetes world with Open Policy Agent and Gatekeeper – rules as code, enforced at the admission controller instead of in a meeting. If you run both worlds, you’ll recognize the line of thinking immediately.

flowchart TD
  Root[Organization Root] --> Sec[OU Security<br/>Logging, Audit]
  Root --> Work[OU Workloads]
  Root --> Sand[OU Sandbox]
  Work --> A1[Account Team Orders]
  Work --> A2[Account Team Billing]
  SCP1[SCP: EU regions,<br/>service blocklist] -.applies to.-> Work
  SCP2[SCP: additionally<br/>tight cost limits] -.applies to.-> Sand

The organizational structure itself is part of the guardrails: a sandbox OU with tight limits and a cleanup rule takes the risk out of experimenting, and a workload OU carries the binding rules. If you start with a multi-account structure today, Control Tower gives you exactly this scaffolding preconfigured – the guardrails behind it are the same SCPs and Config Rules, just already wired up.

What’s different in the end#

After half a year with this setup, the cloud team’s Jira board was almost empty – not because less was happening, but because most things no longer needed an approval. The tickets that remained were the interesting ones: requests to change a guardrail. That’s exactly where human attention belongs. And the platform team suddenly had time for the work that had been left lying around before: better templates, sharper rules, a new account in an hour instead of a week.

I take away a few observations. The approval process and the guardrail answer the same question – “what may happen here?” – but at different times: the process once, beforehand, based on a description; the guardrail at every action, based on reality. That makes guardrails not only faster but more honest. Second: the split into preventive and detective isn’t a subtlety but the central design decision. Everything that is strictly forbidden belongs in SCPs and boundaries; everything that concerns properties of resources belongs in Config Rules with an alarm. If you try to enforce everything preventively, you rebuild the gatekeeping in JSON. And third: shadow infrastructure isn’t a discipline problem of the teams but a symptom. It doesn’t disappear through stricter processes, but through an official path that is faster than the detour.

Further reading#

Revised and translated from German with the help of AI.