Skip to content

Resilience Patterns: Timeout, Retry, Circuit Breaker, Idempotency

Why resilience is no accident: timeout, retry with backoff and jitter, circuit breaker and idempotency as deliberately designed patterns against the failure modes of distributed systems.

≈ 13 min readSoftwareArchitecture

A distributed system rarely falls over because of a spectacular crash. It falls over because a service still answers – just a little slower than usual. Add a call that nobody ever gave a time limit. And a client that dutifully keeps pushing requests while the dependency is long since down. That is how a single shaky component drags half the system down with it, without a single process crashing anywhere.

Distributed calls fail. Over a longer period, that is part of normal operation. The network is slow now and then, a node gets restarted, a database hits its limit. The question is never whether a remote call fails, but what happens when it does. And this is exactly where it is decided whether a system is resilient or only looks stable until things get serious for the first time.

Resilience is designed, not hoped for. I check four patterns for distributed calls: timeout, retry, circuit breaker and idempotency. Not every call needs all four; the choice depends on the failure mode, the cost and the semantics of the operation. The TypeScript code below is illustrative – it is about the mechanics behind it, not a production-ready library.

The patterns differ mainly in the failure state they react to. Their names alone don’t tell you whether they make sense at a given boundary.

Resilience patterns by failure question

  • Question · How long may a single attempt take?
  • Effect · limits blocked resources
  • Question · Is the failure transient and the attempt safe to repeat?
  • Effect · repeats with bounded backoff and jitter
  • Question · Should a visibly impaired target keep getting load?
  • Effect · interrupts calls and checks again later in a controlled way
  • Question · How does a failure stay confined to one part of the system?
  • Effect · separates resources and areas of contention
  • Question · What happens to messages that can never be processed?
  • Effect · keeps failure cases for inspection and targeted reprocessing

Only the failure question shows which patterns complement each other and which would just create extra traffic or hidden queues.

The patterns at a glance#

Before I get into the code, a short overview. Four patterns, each with a clear responsibility:

  • A timeout limits how long a single call may wait. Without a timeout, a call may wait forever – and block a thread, a connection, a slot in the pool while doing so.
  • A retry repeats a failed call – but only for transient errors, only with exponential backoff and jitter, and only with a limited number of attempts.
  • A circuit breaker stops further calls to a visibly failed dependency so you don’t keep hammering it endlessly.
  • Idempotency is what makes retries safe in the first place: a call sent several times has the same effect as a single one.

These four can interlock, but they are not an inseparable package. A timeout may deliberately fail fast without a retry. A retry on a non-idempotent operation can produce duplicate effects. A circuit breaker needs a usable failure signal, which often comes from a timeout. You only combine the patterns that fit the call at hand.

Timeout: never wait forever#

The most expensive timeout is the one that doesn’t exist. A remote call without a time limit isn’t slow, it is potentially infinite. And infinity is a failure mode: threads pile up, connection pools run dry, and a service that is really just waiting on a slow downstream becomes unreachable itself.

The distinction between the connection timeout (how long may establishing the connection take) and the request timeout (how long may the entire response take) matters. Set both. And both belong to the specific call, not nailed globally to one value – a health check has a different tolerance than a report export.

In practice, I wrap this in a small helper. Promise.race against a timer, plus an AbortController, so the underlying operation is actually canceled and doesn’t keep running in the background:

export async function withTimeout<T>(
  fn: (signal: AbortSignal) => Promise<T>,
  timeoutMs: number,
): Promise<T> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);

  try {
    return await fn(controller.signal);
  } finally {
    clearTimeout(timer);
  }
}

// usage: the signal is forwarded to fetch so the socket is released
const result = await withTimeout(
  (signal) => fetch("https://payments.internal/charge", { signal }),
  2_000,
);

The AbortSignal is the part that is easy to forget. A Promise.race that only cancels the waiting but leaves the HTTP connection open has merely moved the problem: the socket stays occupied. Only the forwarded signal releases the resource.

A subtle interaction that Marc Brooker points out in the AWS Builders’ Library: a timeout set too low can itself trigger retry storms. If regular latency occasionally exceeds the timeout limit, the client interprets healthy responses as failures and retries – and creates exactly the load that then really tips the server over. You set timeouts based on the real latency distribution, not on wishful thinking.

Retry: backoff and jitter, but with judgment#

Retry is the pattern that backfires fastest. The naive variant – “on error, try again immediately, and again, and again” – has nothing to do with resilience. It is a load weapon aimed at your own downstream. A server that is already under pressure gets finished off by synchronized repetitions from all clients.

Three ingredients make retry usable. Exponential backoff: the wait time grows with each attempt (base * 2^attempt) so an overloaded service gets some air. A cap: the wait time doesn’t run to infinity but toward an upper limit. And jitter: a random component that keeps all clients from aligning their retries to the same millisecond and creating synchronized load spikes.

There are several variants of jitter – Marc Brooker compared them in 2015 on the AWS Architecture Blog: Full Jitter, Equal Jitter, Decorrelated Jitter. The practical recommendation from that analysis is usually Full Jitter, because it spreads the retries most evenly across the time window. The formula is pleasantly simple:

type RetryOptions = {
  maxAttempts: number;
  baseDelayMs: number;
  capMs: number;
  isRetryable: (error: unknown) => boolean;
};

export async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  opts: RetryOptions,
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 0; attempt < opts.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      if (!opts.isRetryable(error) || attempt === opts.maxAttempts - 1) {
        throw error;
      }

      // full jitter: random point in the exponentially growing window
      const window = Math.min(opts.capMs, opts.baseDelayMs * 2 ** attempt);
      const delay = Math.random() * window;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }

  throw lastError;
}

The isRetryable callback is deliberately pulled out. Not every error is retryable. A 503 Service Unavailable, a dropped connection or a timeout can be transient. An unchanged 400 Bad Request, on the other hand, stays wrong. Even 409 Conflict isn’t a blanket no: after re-reading, with a new version number or an adjusted operation, another attempt can make sense; blindly repeating the identical request usually doesn’t. Retry decisions therefore need the error code, the operation and the remaining time budget.

The AWS Well-Architected guidance sums this up under REL05-BP03 (“Control and limit retry calls”): backoff, jitter and a maximum number of attempts belong together. The AWS SDKs already implement exactly that in their built-in retry modes standard and adaptive – if you work at that level, you shouldn’t put a second retry loop of your own on top. Which brings us to the most dangerous pitfall.

The retry storm#

Retries stack up. The client retries, the SDK underneath retries, the API gateway in front retries. Three layers with three attempts each are not three but up to twenty-seven calls for a single user action. If jitter is missing on top of that and all clients start at the same time, you get a retry storm: multiplicative load that hits the server exactly at the moment it can least handle it.

The remedy is discipline, not more technology. For every call chain, it should be clear which layer owns the retry and which shared time and attempt budget applies. Lower layers such as an SDK may have their own retries if the layers above know about them and don’t multiply them in an uncoordinated way. Jitter and cap belong in the responsible place. If you coordinate many clients, add a retry budget or client-side rate limiting with a token bucket.

Circuit breaker: stop hammering#

Retry with backoff cushions short disruptions. But what if a dependency doesn’t just cough briefly but really fails – for minutes? Then every further attempt is wasted load, wasted latency and wasted resources. This is exactly where the circuit breaker comes in, a pattern that Michael Nygard coined in “Release It!”.

The idea comes from electrical engineering: a circuit breaker interrupts the circuit before the wire burns through. Applied to software, that means – when a dependency is visibly down, the breaker opens and lets calls fail immediately (fail fast) instead of running them into a dead timeout. The breaker knows three states:

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN: Failures >= threshold
    OPEN --> HALF_OPEN: Cooldown expired
    HALF_OPEN --> CLOSED: Probe succeeds
    HALF_OPEN --> OPEN: Probe fails

In the CLOSED state, everything runs normally; the breaker just counts the failures. If the number of failures exceeds a threshold, it switches to OPEN: calls are no longer let through but fail immediately – the client gets a quick answer or a fallback instead of a hanging connection. After a cooldown, the breaker moves to HALF_OPEN and lets a single probe request through. If it succeeds, it closes again (CLOSED). If it fails, it opens again and waits for the next cooldown.

As a small state machine, this is manageable:

type BreakerState = "CLOSED" | "OPEN" | "HALF_OPEN";

export class CircuitBreaker {
  private state: BreakerState = "CLOSED";
  private failureCount = 0;
  private openedAt = 0;

  constructor(
    private readonly failureThreshold: number,
    private readonly cooldownMs: number,
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "OPEN") {
      if (Date.now() - this.openedAt < this.cooldownMs) {
        throw new Error("circuit is open");
      }
      this.state = "HALF_OPEN";
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess(): void {
    this.failureCount = 0;
    this.state = "CLOSED";
  }

  private onFailure(): void {
    this.failureCount++;
    if (this.state === "HALF_OPEN" || this.failureCount >= this.failureThreshold) {
      this.state = "OPEN";
      this.openedAt = Date.now();
    }
  }
}

AWS files the circuit breaker in Well-Architected under REL05-BP01 – “graceful degradation”, turning a hard dependency into a soft one. Instead of a failed recommendation service dragging down the whole product page, the breaker in the OPEN state delivers a fallback (such as a static list) and the page stays usable. That is the real gain: not the perfect answer, but an acceptable one that keeps the overall system alive.

In production, you don’t have to write this yourself – established libraries such as opossum for Node.js, Polly for .NET or resilience4j for Java come with it, including metrics and half-open sampling. The code above is illustrative so the state logic stays visible.

Idempotency: what makes retries safe#

Now the circle closes back to retry. A retry is only harmless if the repeated operation has the same effect as a single one. That is exactly what idempotency means. For a read, this is trivial – two GETs do no harm. For a write, it gets serious: if the first attempt at a payment actually went through but the response got lost in the timeout, the retry charges a second time. The customer pays twice.

HTTP semantics provide a frame here: GET, PUT and DELETE count as idempotent, POST in principle does not. Idempotent means the same intended effect on repetition; the status code or response don’t have to be identical. Where you can design operations accordingly – a PUT on a natural ID instead of a POST that creates a new resource every time – you should. Where that isn’t possible, for example with “charge” or “send”, an explicit idempotency key helps. The client generates a unique key per business operation and sends it along; the server remembers key plus result and thereby prevents the second side effect.

The pattern has been established for years – Stripe, for example, uses an Idempotency-Key header. At the time, an IETF standard was merely an Internet-Draft (draft-ietf-httpapi-idempotency-key-header), but the pattern itself works regardless. On the server side, the core looks like this:

type Result = { status: string; id: string };

interface IdempotencyStore {
  get(key: string): Promise<Result | undefined>;
  set(key: string, value: Result): Promise<void>;
}

export async function handleRequest(
  key: string,
  op: () => Promise<Result>,
  store: IdempotencyStore,
): Promise<Result> {
  const existing = await store.get(key);
  if (existing) {
    // key already processed: return the stored result, no second side effect
    return existing;
  }

  const result = await op();
  await store.set(key, result);
  return result;
}

Two details matter in real life. First, the store needs an expiry (TTL) – keeping idempotency keys forever is neither necessary nor cheap. Second, there is a concurrency problem: if two retries arrive almost simultaneously, the second can complete its get before the first writes its set. In production, you solve this with an atomic reservation of the key (for example a conditional insert), not with the naive get-then-set above. The code shows the principle, not the production-ready lock.

Bulkhead and DLQ, briefly#

Two patterns round out the picture without me going into detail here. The bulkhead – named after the watertight compartments in a ship’s hull – isolates resources so that one overloaded path doesn’t flood the whole system. Separate connection pools or thread pools per downstream make sure a hanging dependency only exhausts its own pool and doesn’t drag all other calls into the abyss with it. This also comes from “Release It!”.

And the dead letter queue (DLQ) is the catch basin for the case where all retries are exhausted. Instead of spinning an unprocessable message in circles forever or losing it silently, it lands in a separate queue – for inspection, for manual intervention or for a later retry. In AWS, this is a standard building block of SQS and SNS. If you want to orchestrate the chaining of such steps cleanly, Serverless Workflows with AWS Step Functions gives you the right frame, and I described how to spread load across several processes in child_process and Scaling.

Conclusion#

Three mistakes show up in post-mortems again and again. No timeout – a call that may wait forever and ties up resources while doing so. Retry without jitter and cap – the well-meant retry that turns into a retry storm and finishes off the weak server for good. And non-idempotent writes that get repeated – the double charge nobody saw coming.

All three have a common root: resilience was hoped for, not designed. The four patterns – timeout, retry, circuit breaker, idempotency – are not a library you import but decisions per call. Remote calls need a justified time limit. Retries need a budget, backoff and jitter. A breaker pays off for dependencies whose sustained failure would otherwise keep generating load and latency. An idempotency key belongs to operations where repetition is possible and a duplicate effect is harmful. That way, a system under load can degrade in a controlled manner instead of multiplying its failure.

Further reading#

Revised and translated from German with the help of AI.