Skip to content

Serverless Workflows with AWS Step Functions

Step Functions turn nested Lambda handlers into a declarative state machine: a visible flow, explicit retry and catch paths, Standard or Express.

≈ 12 min readSoftwareAWSCloud

The first function validates and calls the second, which calculates and calls the third, which sends a notification. On the whiteboard, that makes a clean chain. At runtime, all that remains is a sequence of calls that nobody can pause, inspect or resume from the outside. If you chain Lambda functions together inside their handlers, you have taken the flow away from operations: when an order gets stuck, the search through the logs begins.

AWS Step Functions turns this relationship around. Instead of hiding the flow in nested handlers, you describe it declaratively as a state machine – and the state becomes visible instead of disappearing into call chains.

A state machine makes the flow visible as a series of named states and explicitly handled results.

An explicit workflow

  1. Input · take over the business request and the context it needs
  2. Task · run exactly one bounded capability
  3. Choice · pick the next permitted path based on a result
  4. Parallel · start independent branches only when there is real concurrency
  5. Catch · route expected errors into a named handling path
  6. Completion · record success or failure as an unambiguous end state

The diagram doesn’t pretend there is a linear happy path. Branching and error handling stay part of the model instead of disappearing into scattered function calls.

The problem: orchestration in code#

Let’s first look at what the reflex solution typically looks like. A Lambda function that processes an order might orchestrate internally like this:

exports.handler = async (event) => {
  const order = await validateOrder(event);
  if (!order.valid) {
    await notifyFailure(order);
    return { status: "rejected" };
  }

  if (order.amount > 1000) {
    await requestManualReview(order);
  } else {
    await autoApprove(order);
  }

  const [inventory, payment] = await Promise.all([
    reserveInventory(order),
    chargePayment(order),
  ]);

  return finalize(order, inventory, payment);
};

At first glance, this is readable. But every one of these calls can fail, and as soon as you want robustness, the code grows. What happens if chargePayment throws a transient error? Then you need a retry with backoff. What if reserveInventory succeeds but chargePayment fails for good? Then the reservation has to be rolled back. Bit by bit, the handler fills up with try/catch blocks, counters and timeout logic. The actual business flow – validate, decide, run in parallel, finish – disappears under the error handling.

And the most unpleasant part comes in operations. When an order gets stuck, there is no place where you can read its state. You have CloudWatch logs and have to reconstruct how far the process got. With a single function, that’s still manageable. With five functions calling each other, it turns into detective work. Coordinating several functions is one of the limits you quickly run into with plain code.

The idea: the flow as a state machine#

Step Functions moves orchestration out of the code into a declarative description. This description is written in the Amazon States Language (ASL), a JSON format. You define states, their transitions and the error paths – the individual Lambda functions stay small and each do exactly one thing.

The key gain: the flow no longer lives in a handler but exists as a standalone artifact that the console renders graphically. For every execution, you can see which state it is in, which transition was taken and where something failed.

An ASL definition has only a few required fields at the top level. StartAt names the entry state, States holds the states as an object. Every state has a Type and either points to the next state with Next or marks itself as terminal with "End": true.

Before we get into a concrete definition, here are the eight state types available in 2021 – that’s the complete toolkit:

  • Task does work, typically a Lambda function.
  • Choice branches based on conditions, the if/else of the state machine.
  • Parallel runs several branches at the same time and collects their results.
  • Map applies a flow to every element of a list, in 2021 only as the inline variant.
  • Pass passes input through or injects fixed data, useful for testing.
  • Wait pauses for a duration or until a point in time.
  • Succeed ends the execution successfully.
  • Fail ends the execution with an error.

With these eight building blocks, you can model a surprising number of processes without writing a single line of orchestration code.

A task with retry and catch#

Let’s start with the core: a task state that calls a Lambda function and handles errors declaratively. This is the part that costs the most code in the handler approach – and shrinks to a few fields in ASL.

{
  "Comment": "Order processing workflow",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:ValidateOrder",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.SdkClientException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "NotifyFailure"
        }
      ],
      "Next": "CheckAmount"
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:NotifyFailure",
      "End": true
    }
  }
}

There is a lot in here, so let’s go step by step. The Retry block is an array of retrier objects. ErrorEquals names the errors this retrier responds to – here two transient Lambda service errors that you shouldn’t blame on the business code. IntervalSeconds sets the first wait time, BackoffRate multiplies it with every further attempt (default 2.0), MaxAttempts caps the retries (default 3). Without explicit values, AWS would use these defaults.

The Catch block kicks in when the retries are exhausted or when an error occurs that no retrier covers. Every catcher needs ErrorEquals and a Next that describes the transition. States.ALL is the catch-all error that matches everything – and that is exactly why there are two rules you need to know: States.ALL has to stand alone in its ErrorEquals and has to come last. If another retrier or catcher follows it, the definition is invalid.

An important detail about the order: the matching retriers are processed first, and only when none of them applies anymore is the catcher transition taken. Retry before catch, always in this order. That is exactly the behavior you would painstakingly rebuild by hand in code – here it is a convention of the language.

One point that regularly causes discussion: not every States. error is caught by everything. States.TaskFailed, for example, matches pretty much everything except States.Timeout – so if you only retry on States.TaskFailed and expect a timeout, you’ll be surprised. And States.DataLimitExceeded, which occurs when the payload limit is exceeded (in 2021 that is 256 KB between states), is not caught by States.ALL. You have to deliberately avoid errors like these, not catch them.

Branching with Choice#

After validation, the process should make a decision. Small orders are approved automatically, large ones go to manual review. In the handler this was an if; in ASL it is a choice state.

{
  "CheckAmount": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.amount",
        "NumericGreaterThan": 1000,
        "Next": "ManualReview"
      }
    ],
    "Default": "FanOut"
  },
  "ManualReview": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:RequestManualReview",
    "Next": "FanOut"
  }
}

The choice state contains a Choices array. Each rule consists of a Variable (a path into the current input, here $.amount), a comparison operator such as NumericGreaterThan, StringEquals or BooleanEquals, and a Next. You can also nest conditions with And, Or and Not. If no rule matches, Default applies.

This very Default is a common pitfall. Choice has no End and no implicit exit. Without Default and without a matching rule, the execution runs into a States.NoChoiceMatched error. That’s why I recommend always setting Default, even if you believe you have covered all cases in theory.

Parallel: several branches at once#

For reservation and payment, which run independently of each other, there is the parallel state. It runs several Branches at the same time – each branch is itself a small state machine with its own StartAt and States.

{
  "FanOut": {
    "Type": "Parallel",
    "Branches": [
      {
        "StartAt": "ReserveInventory",
        "States": {
          "ReserveInventory": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:ReserveInventory",
            "End": true
          }
        }
      },
      {
        "StartAt": "ChargePayment",
        "States": {
          "ChargePayment": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:ChargePayment",
            "End": true
          }
        }
      }
    ],
    "Next": "Finalize"
  },
  "Finalize": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:Finalize",
    "End": true
  }
}

The output of a parallel state is an ordered array – one entry per branch, in the order of the definition. So Finalize receives both results bundled together. Parallel also supports Retry and Catch at the state level, and so does the map state when you process a list iteratively.

How data flows between states#

The first time you work seriously with ASL, you stumble over a question that never comes up in handler code: what exactly does the next state receive as input? In a JavaScript function, you simply pass values on as arguments. In a state machine, passing data on follows rules, and you need to know them, or data gets lost without anyone noticing.

Every state receives a JSON object as input and returns a JSON object as output. Four fields control this flow. InputPath filters which part of the input reaches the state at all. Parameters builds the object the Lambda function actually receives from it. ResultPath determines where in the input the result is inserted. Finally, OutputPath filters what gets passed on to the next state.

The most common pitfall is in ResultPath. Without a value, the default is $, which means the task’s result replaces the entire input. If you still want to access $.amount in the choice state after ValidateOrder, you have to put the result under its own key with "ResultPath": "$.validation"; otherwise the original order is gone. This very default is the reason a choice state then reaches into nothing: the key the condition expects no longer exists after the task. Errors like this look like a wrong condition but are really a data flow problem.

The flow as a picture#

The real argument for Step Functions is visibility. What I described above in JSON, the console renders as a graph. And exactly this structure can be traced as a diagram:

flowchart TD
    Start([Start]) --> Validate[ValidateOrder<br/>Task]
    Validate --> Check{CheckAmount<br/>Choice}
    Check -->|amount > 1000| Review[ManualReview<br/>Task]
    Check -->|else| FanOut[FanOut<br/>Parallel]
    Review --> FanOut
    FanOut --> Reserve[ReserveInventory]
    FanOut --> Charge[ChargePayment]
    Reserve --> Finalize[Finalize<br/>Task]
    Charge --> Finalize
    Finalize --> Done([Succeed])
    Validate -. "Catch States.ALL" .-> Notify[NotifyFailure<br/>Task]

The dashed edge is the error path described by the catcher. For a running execution, the console colors the states it has passed through, so you can see at a glance where the process stands or where it got stuck. I never get this view with the handler approach – and that’s the difference between “we have to search the logs” and “just look at the execution.”

Standard or Express#

When you create a state machine, you have to choose a type, and that choice is fixed afterward. If you get it wrong, you build a new state machine. That’s why it pays to understand the two types early.

Standard is the type for long-running, auditable processes. An execution may last up to a year, execution is exactly-once, and the complete history is available through the API for up to 90 days. Billing is per state transition. Important for the cost calculation: every retry attempt is its own transition and counts too. Visual debugging in the console is only available for Standard.

Express is meant for high-frequency, short flows. An execution may run for five minutes at most. Run asynchronously, the guarantee is at-least-once; run synchronously via StartSyncExecution (available since November 2020), it is at-most-once. Billing is based on the number of executions plus duration and memory. There is no execution history in the console – you depend on CloudWatch Logs, and for that, logging has to be enabled.

The dividing line follows the domain: order processes, approval workflows, anything with human involvement or audit requirements belongs to Standard. High-throughput event processing, short transformations, IoT ingestion – that’s Express territory. The idempotent nature of short Express flows fits at-least-once semantics, while for a payment you want the exactly-once guarantee of Standard.

When to use Step Functions – and when not#

I don’t recommend turning every sequence of two functions into a state machine. If one function calls another and that’s it, a direct call or an event is perfectly fine. The effort of Step Functions pays off from the point where there is real flow logic: branches, parallelism, retries with different strategies, compensation for partial failures, or when operations needs to see the state.

A second criterion is error handling. As soon as you notice that the orchestrating handler contains more try/catch than domain logic, that’s a strong signal. Moving this handling out declaratively makes the code of the individual functions small and readable again – each function focuses on its one job, and the coordination lives in the state machine.

And Step Functions doesn’t replace the other serverless building blocks; it coordinates them. The Lambda functions stay, the database stays, authentication – for example with Cognito User Pools – stays where it is. Step Functions is the layer on top that describes the flow.

Conclusion#

Moving from nested handlers to a state machine is above all a change of perspective. Instead of hiding the flow in code, you make it an artifact: StartAt, a few states, Next transitions, and the error paths as Retry and Catch right on the state. The individual functions get smaller because they no longer have to carry the coordination, and operations get calmer because the state of every execution is visible.

The eight state types are quickly learned, and with Task, Choice and Parallel you already cover most real-world flows. With the order of retry and catch in mind, a Default set on every choice and a deliberate choice between Standard and Express, you have the foundation in place. In the projects I was involved in, what really won people over was the colored execution in the console anyway. From then on, the flow is no longer buried in logs but is a picture everyone can look at.

Further reading#

Revised and translated from German with the help of AI.