Node.js Streams: Understanding and Using Backpressure
Why a fast producer must not overrun a slow consumer – and how pipe, pipeline, the return value of write() and the drain event handle flow control in Node.js.
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
This crash does not come from processing too much data. It comes from data arriving too fast. A source reads a log file from a fast SSD, every chunk goes via write() to a slow destination – and because nobody looks at what write() returns, the internal buffer grows until the heap is full. The amount of data does not matter; what matters is the speed gap between the two ends.
Many people think of streams as file stuff around createReadStream. But they are the fundamental model of how Node.js pushes data piece by piece through a pipeline without ever holding everything in memory at once. The part of it that decides in real systems between “runs stable” and “process dies at three in the morning” is called backpressure. The stream API looks harmless: you chain a few pipe calls, and it works – until the day the source delivers faster than the destination can swallow.
Why streams at all#
The classic way to process a file is fs.readFile. The function reads the entire file into a buffer and then calls the callback. For a 2 MB configuration file, that is perfectly fine. For a 4 GB log file, you have a problem: Node.js tries to load four gigabytes into the heap, and the process falls over long before a single line has been processed.
A stream does it differently. It reads the file in small bites – chunks – and passes each chunk on individually. At any point in time, only a small buffer sits in memory, regardless of whether the file is two megabytes or two terabytes. Memory consumption stays constant. That is the core benefit, and it applies not only to files: HTTP requests, TCP sockets, compression, encryption – all of these are streams in Node.js.
The four types#
Before we get to flow control, an overview of the four stream types is worthwhile. All four inherit from EventEmitter, so they communicate through events – that is the common thread running through the entire asynchronous side of Node.js, which I describe in more detail in Node.js Training: From the Event Loop to Architecture Decisions.
- A Readable is a source you read from.
fs.createReadStream()returns a Readable, as does the incoming body of an HTTP request. - A Writable is a destination you write to.
fs.createWriteStream()or the HTTP response are Writables. - A Duplex is readable and writable at the same time, but with separate channels. A
net.Socketis the prime example: in and out over the same connection. - A Transform is a Duplex whose output is a function of its input.
zlib.createGzip()takes raw bytes in and puts compressed ones out. This is exactly where you add your own processing logic.
You combine these types into a pipeline: a source, any number of Transforms in the middle, a destination at the end.
flowchart LR
A["Readable<br/>fs.createReadStream"] -->|chunks| B["Transform<br/>gzip / uppercase"]
B -->|chunks| C["Writable<br/>fs.createWriteStream"]
C -.->|drain signal| A
style A fill:#2b6cb0,color:#fff
style B fill:#2f855a,color:#fff
style C fill:#c05621,color:#fffThe solid arrows are the forward data flow. The dashed arrow going back is the decisive one – it is backpressure. More on that in a moment.
The naive approach and why it tips over#
Suppose we want to copy a large file into another one and process each chunk along the way. The intuitive code looks like this:
const fs = require('node:fs');
const source = fs.createReadStream('huge-input.log');
const target = fs.createWriteStream('huge-output.log');
source.on('data', (chunk) => {
target.write(chunk);
});
That runs. Most of the time. And then comes the case where huge-input.log comes from a fast SSD and huge-output.log is written to a slow network drive. The source fires data events as fast as it can, and target.write() accepts every chunk. If the destination cannot write fast enough, Node.js buffers the chunks internally – and this buffer has no upper limit. Memory fills up, RSS climbs, the garbage collector works against an ever-growing pile, and at some point the operating system kills the process.
That is the pitfall in one sentence: If you ignore the return value of write() and just keep writing, you produce unlimited buffering.
What write() actually tells you#
writable.write(chunk) returns a Boolean, and this Boolean is half the battle:
truemeans: the internal buffer is still belowhighWaterMark; you may keep writing.falsemeans: the buffer has reached or exceededhighWaterMark. Stop writing, or Node.js will buffer into the void.
highWaterMark is the buffer limit. In the Node 16/18 covered here, the general default for byte streams was 16384 bytes, that is 16 KiB; since Node 22 it is 64 KiB. In object mode – more on that later – the default stays at 16 objects. It is important to understand: this limit is a threshold, not a hard wall. Node.js does not stop buffering when you exceed highWaterMark. It just politely tells you with a false return value that it has had enough. If you ignore the warning, it keeps buffering anyway – at your own risk.
Once write() has returned false, you should pause and wait for the 'drain' event. The Writable emits 'drain' as soon as its buffer has been emptied again – the signal that you may continue writing. Done by hand, the correct cycle looks like this:
source.on('data', (chunk) => {
const ok = target.write(chunk);
if (!ok) {
source.pause();
target.once('drain', () => source.resume());
}
});
We write, check the return value, pause the source on false and resume it only after 'drain'. That is exactly flow control: a fast producer is slowed down so that it does not overrun a slow consumer.
You can also draw the cycle as a state machine:
stateDiagram-v2
[*] --> Schreiben
Schreiben --> Schreiben: write() returns true
Schreiben --> Pausiert: write() returns false<br/>source.pause()
Pausiert --> Schreiben: 'drain' event<br/>source.resume()pipe does this automatically#
The good news: in practice, you rarely have to write this whole pause/resume dance by hand. pipe() does exactly that in the background.
const fs = require('node:fs');
fs.createReadStream('huge-input.log')
.pipe(fs.createWriteStream('huge-output.log'));
pipe() connects source and destination, checks the return value of write() internally, pauses the Readable on backpressure and resumes it after 'drain'. So backpressure is solved without you writing a single line of flow control yourself. pipe() also returns the destination so that you can chain – handy when a Transform sits in between:
const fs = require('node:fs');
const zlib = require('node:zlib');
fs.createReadStream('huge-input.log')
.pipe(zlib.createGzip())
.pipe(fs.createWriteStream('huge-input.log.gz'));
And yet I advise against just chaining pipe() in production. The reason is error handling.
Why pipeline is the better choice#
pipe() has a treacherous weakness: it does not propagate errors. If a stage in the middle of the chain throws an error – the network connection drops, the disk is full – a pipe() chain does not notice it cleanly. The other streams are not destroyed, file descriptors stay open, the stream hangs. In a long-running server process, such leaks accumulate until the resources eventually run out.
stream.pipeline() solves this. It chains the stages the same way, but it propagates errors across all stages and calls destroy() on every stream involved on error or regular end. Nothing leaks. There is the callback variant:
const fs = require('node:fs');
const zlib = require('node:zlib');
const { pipeline } = require('node:stream');
pipeline(
fs.createReadStream('huge-input.log'),
zlib.createGzip(),
fs.createWriteStream('huge-input.log.gz'),
(err) => {
if (err) {
console.error('pipeline failed', err);
return;
}
console.log('pipeline succeeded');
}
);
Since Node.js 15 – and thus firmly established in the release lines 16 and 18 that are current in 2022 – there is also a promises variant from node:stream/promises. It reads much more cleanly in async functions because you catch errors with try/catch:
const fs = require('node:fs');
const zlib = require('node:zlib');
const { pipeline } = require('node:stream/promises');
async function compress() {
try {
await pipeline(
fs.createReadStream('huge-input.log'),
zlib.createGzip(),
fs.createWriteStream('huge-input.log.gz')
);
console.log('done');
} catch (err) {
console.error('pipeline failed', err);
}
}
compress();
The promises variant also takes options, such as a signal for an AbortSignal that lets you abort the pipeline from the outside. The point that tips the scales for me: pipeline() handles backpressure just as automatically as pipe() – but it also cleans up. For production code, pipeline() is the default choice; pipe() at most for short, isolated scripts.
Writing your own Transform#
The point where streams become really useful is your own Transform in the middle. A Transform takes chunks in, processes them and pushes the result out. The skeleton consists of a transform function that receives three things: the chunk, the encoding and a callback you have to call at the end.
const { Transform } = require('node:stream');
const upperCase = new Transform({
transform(chunk, encoding, callback) {
const transformed = chunk.toString().toUpperCase();
this.push(transformed);
callback();
}
});
this.push() passes the result on to the next stage; callback() signals that this chunk has been fully processed. If you call callback() with an error – callback(err) – pipeline() propagates this error cleanly through the entire chain. Built into a pipeline, it looks like this:
const fs = require('node:fs');
const { pipeline } = require('node:stream/promises');
await pipeline(
fs.createReadStream('input.txt'),
upperCase,
fs.createWriteStream('output.txt')
);
And because the upperCase Transform takes part in flow control like any stream, backpressure applies here too: if the destination is slow, the pipeline slows down across the Transform back to the source. You do not have to do anything extra for that.
Async iteration as a modern alternative#
Since Node.js 10, every Readable implements Symbol.asyncIterator, so you can consume it directly with for await ... of. In async functions, this often reads more clearly than on('data') and keeps flow control intact without you having to touch pause and resume.
const fs = require('node:fs');
async function countLines(path) {
const stream = fs.createReadStream(path, { encoding: 'utf8' });
let lines = 0;
for await (const chunk of stream) {
lines += chunk.split('\n').length - 1;
}
return lines;
}
Backpressure arises implicitly here: as long as the loop body is busy with a chunk – and an await inside it holds it up further – the loop does not request a new chunk, and the Readable only delivers the next one then on its own. So you get the same brake as with pipe(), just in ordinary async/await code. The rule that comes up next still applies: if you iterate like this, you should not also consume the stream in parallel via on('data').
Object mode and a few rules#
By default, streams work with buffers and strings. If you set objectMode: true, the chunks may be arbitrary JavaScript objects – useful when you build a pipeline of parsed records instead of raw bytes. Note that in object mode, highWaterMark counts the number of objects, not bytes; the default is 16 objects. And an existing stream cannot be safely switched to object mode afterward – you decide that at construction.
A rule that keeps snagging in reviews: do not mix consumption methods. Per Readable, you choose exactly one way of reading – either on('data'), or on('readable'), or pipe(), or async iteration. If you combine two of them, you produce race conditions, lost chunks and bugs that are hard to reproduce. One Readable, one consumption method.
What should stick#
Streams are the tool for processing arbitrarily large amounts of data with constant memory – the buffer stays small no matter how large the source is. Backpressure is the mechanism that makes this work: the return value of write() and the 'drain' event form the flow control that keeps a fast producer from overrunning a slow consumer. highWaterMark is the threshold at which write() flips from true to false – a warning, not a constraint.
You rarely have to write this cycle by hand, because pipe() and above all pipeline() handle it automatically. The decisive difference lies in error handling: pipeline() propagates errors and cleans up all streams; pipe() does not. With this difference in mind, you get Node.js processing code that does not fall over even with four gigabytes and a slow destination – it just gets a bit slower, the way it should.
Further reading#
- Repository for the Node.js introduction: mikebild/introduction-nodejs – teaching material on async patterns including
10-streams.md. - Node.js Stream API (v18) – Readable, Writable, Duplex, Transform,
pipe,pipeline,write,drain,highWaterMark. - Current Node.js Stream API – today’s defaults
- stream.pipeline in the official docs
- Guide: Backpressuring in Streams
Revised and translated from German with the help of AI.