Skip to content

Streams in Object Mode: Data Pipelines Beyond Bytes

Node.js streams carry more than bytes: with objectMode:true they become composable data pipelines that parse, filter and aggregate – including the bridge to async iterators.

≈ 11 min readSoftwareNode.jsBackend

If you switch a stream to objectMode: true and bring along your usual highWaterMark: 16384, you no longer buffer 16 KiB but 16,384 records. Same number, different unit, and a memory footprint that is off by orders of magnitude. Object mode looks like an incidental constructor option, but it changes what a stream pushes through the pipeline at all – and with it, when throttling kicks in.

In the earlier post on Node.js streams and backpressure, everything revolved around bytes: buffers flowing through a pipe, and the art of throttling the flow before memory overflows. That is the standard case for file and network I/O. But as soon as you are not copying files but processing records – CSV lines, JSON objects, log entries – that picture no longer goes far enough.

When a stream works in object mode, it no longer carries buffers or strings, but arbitrary JavaScript values – whole objects, arrays, numbers. That turns a tool for raw byte streams into a tool for data processing. And the nice part: the mechanics stay the same. You chain stages, each stage does one thing, and in the end you have a pipeline that passes one record after another without loading everything into memory at once.

What object mode really changes#

A stream in standard mode knows exactly two data types: Buffer and String. Everything you push into it has to be one of the two. Object mode lifts this restriction. A new stream.Transform({ objectMode: true }) accepts and emits arbitrary JS values. The only exception is null – this value is reserved and signals the end of the stream. More on that later, because this is exactly where a pitfall lies.

Object mode is set per stream via the constructor option, and the default is false. So you turn it on deliberately. The pattern already shows up throughout the examples in my Node.js repository – there, however, still in the ES5 style with util.inherits and Readable.call(this, { objectMode: true }). A Readable that passes on two small objects looks like this there:

const { Readable } = require('stream');
const util = require('util');

// ES5 pattern from the training repo
const ReadStream = function () {
  Readable.call(this, { objectMode: true });
  this.data = [
    { id: 0, value: 1 },
    { id: 1, value: 2 },
  ];
  this.curIndex = 0;
};
util.inherits(ReadStream, Readable);

ReadStream.prototype._read = function () {
  if (this.curIndex === this.data.length) {
    return this.push(null); // signals end of stream
  }
  this.push(this.data[this.curIndex++]);
};

That works and grew historically. Still, I prefer to show it in modern class syntax, because it is closer to what you find in codebases today. With class ... extends Transform, the constructor ritual becomes a super({ objectMode: true }), and the processing logic sits in a clearly named method.

The one core method: _transform#

At its core, a transform stream has a single method you write yourself: _transform(chunk, encoding, callback). It is called once for every incoming value. You emit results with this.push(value) and report with callback() that this value has been fully processed. In object mode, chunk is the raw JS value – the encoding argument is ignored because there is nothing to decode here.

The key point: push and callback are two separate things. push sends output to the next stage, callback signals “I am done with this input value.” You can push zero, one or several times per input and call callback exactly once at the end. If you only call callback() without push, the value disappears – that is the natural way to leave something out in a filter.

Here is a processing stage that parses raw CSV lines into objects:

const { Transform } = require('stream');

class ParseRow extends Transform {
  constructor() {
    super({ objectMode: true });
  }

  _transform(line, _encoding, callback) {
    const [id, name, status] = String(line).split(',');
    this.push({ id: Number(id), name, status });
    callback();
  }
}

And a filter stage that only lets active records through. This is where leaving out shows: if the condition is not met, nothing is pushed; only callback() is called.

const { Transform } = require('stream');

class FilterActive extends Transform {
  constructor() {
    super({ objectMode: true });
  }

  _transform(record, _encoding, callback) {
    if (record.status === 'active') {
      this.push(record);
    }
    // no push means the record is dropped
    callback();
  }
}

A formatting stage rounds out the picture. It takes the object and shapes an output line from it:

const { Transform } = require('stream');

class FormatOutput extends Transform {
  constructor() {
    super({ objectMode: true });
  }

  _transform(record, _encoding, callback) {
    this.push(`#${record.id} ${record.name}\n`);
    callback();
  }
}

Three small stages, each with a single responsibility. That is exactly how composability comes about.

Chaining stages: parse, filter, format#

The actual pipeline is then almost unremarkable – and that is a good sign. I chain the stages with pipeline() from the stream module, not with a chain of pipe() calls. The reason: pipeline() propagates errors and cleans up the streams involved when one fails. A plain pipe() chain does not – there you have to handle errors and cleanup at every seam yourself, and that is exactly what gets forgotten in practice.

const { Readable, pipeline } = require('stream');

const csvLines = ['0,Ada,active', '1,Linus,inactive', '2,Grace,active'];

pipeline(
  Readable.from(csvLines),
  new ParseRow(),
  new FilterActive(),
  new FormatOutput(),
  process.stdout,
  (err) => {
    if (err) {
      console.error('pipeline failed:', err);
    } else {
      console.log('pipeline finished');
    }
  }
);

The source here is Readable.from(csvLines) – more on that in a moment. The sink is process.stdout, a perfectly ordinary writable stream. In between, one record after another flows through: first as a string, after ParseRow as an object, filtered by FilterActive, shaped back into a line in FormatOutput. At no point is the entire data set in memory – there is only ever a small supply of buffered values in flight.

As a diagram, this data flow looks like this:

flowchart LR
  A["Source<br/>Readable.from"] -->|"CSV line (string)"| B["Parse<br/>_transform"]
  B -->|"JS object"| C["Filter<br/>_transform"]
  C -->|"JS object"| D["Format<br/>_transform"]
  D -->|"output line"| E["Sink<br/>Writable"]

The edges show what actually flows: first a line as a string, then JS objects through the middle stages, and text again at the end. Each stage only takes care of its own conversion and knows nothing about its neighbors.

Aggregating with _flush#

Filtering and reshaping work record by record. For aggregations – sums, counts, groupings – you need a different rhythm: collect while the values arrive, and emit the result only once the source is exhausted. That is what _flush(callback) is for. This method is called exactly once, just before the stream ends. You collect in _transform and emit in _flush.

const { Transform } = require('stream');

class CountByStatus extends Transform {
  constructor() {
    super({ objectMode: true });
    this.counts = {};
  }

  _transform(record, _encoding, callback) {
    this.counts[record.status] = (this.counts[record.status] || 0) + 1;
    callback(); // nothing pushed yet, just accumulate
  }

  _flush(callback) {
    this.push(this.counts); // emit the aggregate once at the end
    callback();
  }
}

This stage swallows many objects and emits a single one at the end – the aggregate. It shows nicely that a stream stage does not have to keep its input and output counts one to one.

Backpressure now counts objects#

In object mode, highWaterMark means something different than in byte mode – people regularly read past this, with noticeable consequences. highWaterMark is the threshold at which a stream counts as “full enough” and triggers backpressure. In byte mode, this threshold is measured in bytes; in 2022 the general default was 16 KiB, since Node 22 it has been 64 KiB. In object mode it is measured in objects, still with a default of 16 objects.

That is no small detail. If you come from byte mode and read highWaterMark: 16, you reflexively think of 16 bytes and consider it absurdly small. In object mode, though, it means 16 buffered objects – a perfectly sensible supply. Conversely, a highWaterMark: 16384 set out of byte-mode habit can mean in object mode that you buffer 16,384 objects before backpressure kicks in. That is rarely what you want.

It is also important that highWaterMark is a threshold, not a hard barrier. The buffer may exceed it – the threshold is the signal from which push() returns false and asks upstream stages to throttle. With pipeline() and pipe(), this pausing and resuming happens automatically. So normally you rely on this automatic interplay and only have to check the return value of push yourself when you drive a source by hand. Backpressure works just as reliably in object mode as in byte mode; only the unit is different.

To tell them apart, here are the two modes side by side:

  • Byte mode (standard) carries Buffer or String; highWaterMark counts bytes here. The general default depends on the Node version and has been 64 KiB since Node 22.
  • Object mode (objectMode: true) carries arbitrary JS values except null, highWaterMark counts objects here with a default of 16, and the typical uses are data pipelines that parse, filter, transform and aggregate.

The bridge to async iterators#

So far I have built streams as classes. That is not always necessary, and it is often even cumbersome. Node offers two modern connectors that bring object mode together with the world of async iterators: Readable.from(iterable) and for await ... of.

stream.Readable.from(iterable) takes any iterable or async iterable – an array, a generator, an async generator – and turns it into a readable stream. By default, this stream works in object mode. It is the most elegant way to feed a data source into a pipeline without writing your own Readable class. That is exactly what I already used above with Readable.from(csvLines).

The other direction is at least as handy: a readable stream is itself an async iterable. So you can consume it directly with for await ... of, and the language construct takes care of pausing, resuming, backpressure and the end – all automatically.

const { Readable } = require('stream');

async function* generateRecords() {
  yield { id: 0, value: 1 };
  yield { id: 1, value: 2 };
  yield { id: 2, value: 3 };
}

async function main() {
  const source = Readable.from(generateRecords());

  for await (const record of source) {
    console.log('received:', record);
  }
}

main();

You can even use async generators as a pipeline stage. A generator that consumes a source and passes on filtered values replaces an entire transform class with a few lines:

async function* filterEven(source) {
  for await (const n of source) {
    if (n % 2 === 0) {
      yield n;
    }
  }
}

Whether you write a transform class or an async generator is to a good extent a matter of taste. I reach for the class when I have to keep internal state across many values, for example in an aggregation with _flush. For pure filters and reshaping, async generators are often the shorter, more readable way.

Two pitfalls that strike again and again#

The first pitfall is mixing byte and object mode in one pipeline. You start with fs.createReadStream(...), which is a byte stream and delivers buffers. Right after it you attach an object mode stage that wants to work with record.status – and get a buffer instead of an object. Between a byte readable and the first object stage there always has to be a parse layer that decodes buffers into values. The reverse seam is just as tricky: if you push an object into a non-object-mode writable, Node answers with a type error (ERR_INVALID_ARG_TYPE) that demands a string, Buffer or Uint8Array. At every seam you should be aware of what data type is flowing right now.

The second concerns null as a legitimate payload value. Because push(null) means the end of the stream, you cannot send a real null value through an object mode stream – it would be mistaken for the end signal and cut off the remaining data. If null has a real meaning in your payload, you need a stand-in object, a so-called sentinel, such as { value: null } instead of a bare null.

A third point is a matter of diligence: you pass errors in _transform to callback(err) instead of throwing them. Only then do they end up in the stream’s error channel and get propagated cleanly by pipeline(). An error thrown in the middle of asynchronous processing is much harder to catch – another argument for pipeline() over a bare pipe() chain.

Conclusion#

Object mode turns Node.js streams from an I/O tool for bytes into a tool for data processing. The mechanics stay familiar – _transform with push and callback, _flush for aggregating, chaining via pipeline() – but the content is now whole JS values. This lets you build pipelines that parse, filter, transform and aggregate as separate, composable stages that only ever hold a small supply of records in memory.

I take away two things as the core message. First: in object mode, highWaterMark counts objects, not bytes – mix that up and you size your buffer wrong by orders of magnitude. Second: Readable.from and for await ... of are the modern bridge between streams and async iterators and in many cases save you from writing your own stream class. If you have understood the byte-oriented part from the post on buffers and binary data and add object mode to it, you have both halves of the stream world together – the raw one and the structured one.

Further reading#

Revised and translated from German with the help of AI.