Event Sourcing on AWS: DynamoDB Streams and Lambda
Event Sourcing implemented serverless: DynamoDB as an append-only event store, DynamoDB Streams as the change feed and Lambda as an idempotent projector into CQRS read models.
An event store is not a table whose rows you update. You only append: every state change is a new, immutable item, and the state of an aggregate only emerges when you fold its events in version order. In my conceptual look at Event Sourcing, I described this as a logbook – the sequence of events carries the truth, and the state is merely its result.
On the whiteboard, that is quickly explained. It only becomes viable with the answers to the next questions: Where does the store run? Who guarantees the order? What happens when a consumer sees the same event twice? On AWS, you can assemble this from managed services, and the sharp edges then lie in concurrency when appending, in the order of the change feed and in duplicate delivery to the projection. That is exactly what this article is about – including the code I write for it.
The three building blocks#
Before I get into the code, the mental model. If you separate the roles cleanly, you make far fewer mistakes later:
- The event store is a DynamoDB table, append-only: one item per event, immutable. This is the source of truth, the only truth in the system.
- The streams form the change feed of this table. Every new row creates a stream record. No second store, just a window onto the changes.
- The projection is a Lambda that reads the stream records and writes denormalized read models from them. This is exactly the projection idea behind read models, just cast in AWS services.
Important for debugging later: the stream is not an archive. Records are trimmed after 24 hours. If you need the complete history, you read it from the table, not from the stream. The stream only carries the flow of deltas from the write side to the read side.
Put together, the path of a command looks like this:
graph LR A[Command] --> B[CommandHandler] B -->|Conditional write| C[(DynamoDB<br/>event store)] C --> D[DynamoDB Streams] D --> E[Lambda<br/>projector] E --> F[(Read model table)] F --> G[Query]
The write side is on the left, the read side on the right. The stream connects both sides asynchronously. DynamoDB Streams holds each change record once; processing by Lambda, on the other hand, is at-least-once. That is exactly why the projection has to tolerate repetitions.
The event store as a table#
The data model is deliberately simple. The partition key is the aggregateId, the sort key is the version – zero-padded so that numeric sorting works as a string. An aggregate is thus one partition, and its events sit there in ascending version order. This maps directly to the principles of single-table design: the access path is in the key.
I enable streams when creating the table via the StreamSpecification. The StreamViewType has exactly four valid values – KEYS_ONLY, NEW_IMAGE, OLD_IMAGE and NEW_AND_OLD_IMAGES. For Event Sourcing, I need the new image, because the event lives in the freshly written row:
Resources:
EventStore:
Type: AWS::DynamoDB::Table
Properties:
TableName: event-store
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
- AttributeName: pk
AttributeType: S
- AttributeName: sk
AttributeType: S
KeySchema:
- AttributeName: pk
KeyType: HASH
- AttributeName: sk
KeyType: RANGE
StreamSpecification:
StreamViewType: NEW_IMAGE
A detail that once cost me hours: the StreamViewType cannot be changed after creation. If you want to switch it, you have to disable the stream and create it again – with a new stream ARN and therefore a new event source mapping. Getting this right on the first attempt saves you an unpleasant migration step later.
Appending events with optimistic concurrency#
The interesting part of writing is not the writing itself, but the question: What if two commands want to advance the same aggregate to version N+1 at the same time? Exactly one may win. I solve this with optimistic concurrency through a conditional write.
The idea: I write the event for version = expectedVersion + 1 and tie it to the condition that no item exists under this key yet. If it already exists, someone else has taken the version, and DynamoDB throws a ConditionalCheckFailedException. The caller then rereads the current version and retries the command.
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import {
DynamoDBDocumentClient,
PutCommand,
} from "@aws-sdk/lib-dynamodb";
import { ConditionalCheckFailedException } from "@aws-sdk/client-dynamodb";
const client = DynamoDBDocumentClient.from(new DynamoDBClient({}));
export class ConcurrencyError extends Error {}
interface DomainEvent {
type: string;
payload: Record<string, unknown>;
}
export async function appendEvent(
aggregateId: string,
expectedVersion: number,
event: DomainEvent,
): Promise<void> {
const nextVersion = expectedVersion + 1;
try {
await client.send(
new PutCommand({
TableName: "event-store",
Item: {
pk: aggregateId,
sk: String(nextVersion).padStart(12, "0"),
version: nextVersion,
type: event.type,
payload: event.payload,
createdAt: new Date().toISOString(),
},
ConditionExpression: "attribute_not_exists(pk)",
}),
);
} catch (err) {
if (err instanceof ConditionalCheckFailedException) {
throw new ConcurrencyError(
`version ${nextVersion} already exists for ${aggregateId}`,
);
}
throw err;
}
}
attribute_not_exists(pk) checks whether the complete item key is free – DynamoDB evaluates the condition against exactly the item whose key is in the PutCommand. This makes the combination of aggregateId and version the uniqueness anchor. No additional lock, no separate sequence table, no transaction across several systems.
When I have to write several events atomically – for example, for a command that triggers two domain events – I use TransactWriteItems instead of individual puts. This bundles up to 100 items in one atomic transaction, and I can put the same condition on each event. For the rule “one event = one item”, the simple conditional write is enough.
The projection as a stream handler#
Now for the read side. As soon as an event is written, it appears in the stream, and the event source mapping pulls it into a Lambda. Its job: fold the events into a read model.
The handler receives a DynamoDBStreamEvent with a batch of records. Each record carries an eventName – for our append-only store, that is always INSERT. I read the new image from dynamodb.NewImage and unmarshall it into a plain object.
import type { DynamoDBStreamEvent, DynamoDBBatchResponse } from "aws-lambda";
import { unmarshall } from "@aws-sdk/util-dynamodb";
import {
DynamoDBDocumentClient,
GetCommand,
UpdateCommand,
} from "@aws-sdk/lib-dynamodb";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
const client = DynamoDBDocumentClient.from(new DynamoDBClient({}));
class ProjectionGapError extends Error {}
export async function handler(
event: DynamoDBStreamEvent,
): Promise<DynamoDBBatchResponse> {
const batchItemFailures: { itemIdentifier: string }[] = [];
for (const record of event.Records) {
if (record.eventName !== "INSERT") continue;
if (!record.dynamodb?.NewImage) continue;
try {
const item = unmarshall(record.dynamodb.NewImage as never);
await project(item);
} catch (err) {
batchItemFailures.push({
itemIdentifier: record.dynamodb!.SequenceNumber!,
});
}
}
return { batchItemFailures };
}
Two things matter here. First, I return batchItemFailures at the end. If ReportBatchItemFailures is enabled on the event source mapping, Lambda uses the smallest reported SequenceNumber as a checkpoint and retries from there. This reduces repetitions, but doesn’t rule them out for subsequent records that already succeeded. Without partial feedback, the entire batch counts as failed.
Second, the actual projection. This is where the most expensive mistake of the whole setup lurks, so it gets its own section.
Idempotency is not optional#
The event source mapping delivers at-least-once. The stream itself contains each record exactly once – but processing by Lambda can deliver the same batch several times on retries, with BisectBatchOnFunctionError or on a timeout. Secondary sources often mix up these two statements: the stream is exactly-once and ordered per item; Lambda processing is at-least-once. For the code I write, the second statement is what counts.
In concrete terms: my projection has to cope with the same event twice without corrupting the read model. With a count, duplicate processing would otherwise be a double count. The guard against it is a conditional write on a gapless, increasing version in the read model. I only write the new projection if its version directly follows the last stored one:
async function project(item: Record<string, unknown>): Promise<void> {
const aggregateId = item.pk as string;
const version = item.version as number;
const current = await client.send(
new GetCommand({
TableName: "read-model-orders",
Key: { pk: aggregateId },
ConsistentRead: true,
}),
);
const lastProcessedVersion =
(current.Item?.lastProcessedVersion as number | undefined) ?? 0;
// A repeated delivery has no effect.
if (version <= lastProcessedVersion) return;
// A later event must wait until the missing versions were applied.
if (version !== lastProcessedVersion + 1) {
throw new ProjectionGapError(
`expected version ${lastProcessedVersion + 1}, received ${version}`,
);
}
await client.send(
new UpdateCommand({
TableName: "read-model-orders",
Key: { pk: aggregateId },
UpdateExpression: [
"SET lastProcessedVersion = :v",
"#status = :status",
"updatedAt = :now",
].join(", "),
ConditionExpression:
"attribute_not_exists(pk) OR lastProcessedVersion = :previous",
ExpressionAttributeNames: {
"#status": "status",
},
ExpressionAttributeValues: {
":v": version,
":previous": lastProcessedVersion,
":status": deriveStatus(current.Item?.status, item),
":now": new Date().toISOString(),
},
}),
);
}
If an event arrives twice, the function stops before the update. If, on the other hand, version 5 arrives while the read model is only at version 3, the gap is reported as an error. Lambda redelivers this record; version 4 can be processed first. The condition in the UpdateCommand additionally protects against two projectors working at the same time. If it fails after the read, the record is retried as well – it must not simply vanish as a successful duplicate case.
An edge case that is easy to miss: a PutItem or UpdateItem that doesn’t actually change the data creates no stream record. If you tie your projection expectations to “every write produces an event”, you will otherwise wonder about missing records. In the append-only event store, this doesn’t happen, because every event is a new item – but as soon as you start remodeling the store, you should keep it in mind.
The projection has to check ordering itself#
The second pitfall concerns order. AWS guarantees for DynamoDB Streams: for each individual item, the stream records appear in the same order as the actual changes to that item. In this data model, however, each event version is its own item – same partition key, different sort key. So the per-item guarantee does not imply a guaranteed order across the events of an aggregate.
In normal operation, consecutive writes often arrive in the expected order. The projector still must not rely on that, especially not with retries, partial batch failures or an increased ParallelizationFactor. That is why the code above checks for a gapless version. A gap is retried or deliberately filled from the event store, instead of pulling a later event ahead and then discarding the missing one as outdated.
There is no global ordering across aggregates either. The records are spread across shards, and shards are processed in parallel. If you need a globally chronological view – say, a feed of all orders sorted by time – you have to model an order suited for that. A timestamp can serve for display, but on its own it is not a conflict-free global sequence.
Rebuild and snapshots#
A read model can be derived. That is one of the nicest advantages of the approach: if I change the projection logic or a read model is corrupt, I throw it away and rebuild it. The rebuild is a replay – I read all events of an aggregate in ascending order and fold them:
import {
DynamoDBDocumentClient,
QueryCommand,
} from "@aws-sdk/lib-dynamodb";
export async function rebuildProjection(
aggregateId: string,
fromVersion = 0,
): Promise<ReadModelState> {
const result = await client.send(
new QueryCommand({
TableName: "event-store",
KeyConditionExpression: "pk = :id AND sk > :from",
ExpressionAttributeValues: {
":id": aggregateId,
":from": String(fromVersion).padStart(12, "0"),
},
}),
);
const events = result.Items ?? [];
return events.reduce(applyEvent, initialState());
}
For aggregates with a very large number of events, this gets expensive – every replay reads the whole stream. Against that, the AWS Prescriptive Guidance recommends snapshots: the materialized state of an aggregate at version N, stored periodically. The rebuild then loads the snapshot and folds only the delta events since that version. The fromVersion parameter above is exactly the seam for this – I start the query at snapshot.version instead of at zero. The complete rebuild without a snapshot remains the emergency brake for the case where the fold logic itself has changed.
Hot partitions and the blocked shard#
The third pitfall is the price of clear partitioning. The aggregateId as the partition key means: a very active aggregate concentrates all writes and all stream traffic on one partition and one shard. That leads to throttling and latency – a classic hot partition.
On top of that comes a reader limit: at most two processes may read from a shard at the same time. A batch that the Lambda cannot process also blocks the shard’s progress until it either succeeds or expires. This is exactly what the settings on the event source mapping are for:
BisectBatchOnFunctionErrorsplits a failed batch recursively to isolate the one poisonous record.MaximumRetryAttemptsandMaximumRecordAgeInSecondslimit how long a record may block.- An on-failure
DestinationConfigpushes batches that have failed for good to SQS or SNS instead of holding them in the stream.
Together with ReportBatchItemFailures from the handler, this gives you error handling that doesn’t let a single broken event halt the entire projection. If you set this up from the start, you spare yourself the incident where a read model table stands still for hours because a single record could not be parsed.
Conclusion#
Event Sourcing on AWS needs surprisingly little: a DynamoDB table with aggregateId as the partition key and version as the sort key, a conditional write for optimistic concurrency, DynamoDB Streams as the change feed and a Lambda that folds idempotent read models from it. That is Event Sourcing plus CQRS, assembled serverless from managed services.
The concepts hold up just as well as on the whiteboard – but the operational details decide whether the system survives the night. I take three things into every project: projections have to be idempotent, because processing is at-least-once. The order of the separate event items has to be checked via their version. And the error handling on the event source mapping decides whether the stream clears itself again or gets stuck.
Once you have set up this stack, you quickly notice how well it can be extended – for example, toward a GraphQL read side with AppSync and DynamoDB that sits directly on the projected read models. The event store remains the truth, the projections remain derivable, and the API only reads views that are already fully shaped.
Further reading#
- Slide deck on this topic: https://slides.mikebild.dev/introduction-cqrs-event-sourcing.en.html
- DynamoDB Streams – retention, StreamViewType, ordering, shards
- DynamoDB Streams and Lambda triggers
- Using AWS Lambda with DynamoDB – ESM, Batch, ParallelizationFactor, Error Handling
- Condition Expressions
- Optimistic locking with versioning
- AWS Prescriptive Guidance – Event Sourcing Pattern
- AWS Database Blog – Build a CQRS event store with Amazon DynamoDB
Revised and translated from German with the help of AI.