Skip to content

Node.js: Async I/O Explains Architecture

The calculation “one thread times 200 connections” shows which systems fit Node.js and which work belongs in separate processes.

≈ 10 min readSoftwareNode.jsBackend

Node.js: Async I/O Explains Architecture

Fifty milliseconds isn’t a conspicuous number. A JSON.parse over a few megabytes, a synchronous template render, some hashing – 50 ms of CPU time doesn’t stand out unpleasantly in any profiler. Now the second number: 200 concurrent connections. In Node.js, all JavaScript code runs on a single thread. If each of these 200 requests contains a synchronous 50 ms block, the last response doesn’t wait 50 milliseconds but 200 × 50 ms = 10 seconds. Not because the machine is overloaded – CPU and network are bored – but because all requests share the same thread and have to pass through the same bottleneck one after the other.

This small calculation is the complete architecture talk on Node.js in one paragraph. Once you have really understood it, you can derive most system decisions around Node: which services belong there, which don’t, and why the scaling answer is “more processes”. The following sections show what that means in detail.

One thread, many connections#

Node answers the question “How do I serve many concurrent connections?” differently than the classic servers. Apache with PHP or a Java servlet container give each request its own thread or process; if one blocks, it hardly bothers the others. Node turns the model around: there is exactly one thread for JavaScript, and it must never wait. Everything that takes time – a network call, a file access, a DNS lookup – is handed off to the operating system or to a small internal thread pool. The JavaScript thread merely registers a callback and turns to the next task.

This is organized by libuv, the C library Node is built on. It asks the kernel via epoll (Linux), kqueue (macOS) or IOCP (Windows) which of the open operations are done and works through the associated callbacks in a fixed order – that is the event loop. The loop goes through phases: timers, pending callbacks, polling for I/O, setImmediate, close callbacks. In between, microtask processing with process.nextTick and promise resolutions slips in. File system, DNS and parts of crypto don’t run through the kernel mechanism but through the thread pool mentioned above – four threads by default, configurable via UV_THREADPOOL_SIZE.

flowchart LR
  subgraph proc[Node process]
    JS[JavaScript<br/>one single thread] --> Loop[Event loop<br/>libuv]
  end
  Loop --> Kernel[Kernel<br/>epoll / kqueue / IOCP]
  Loop --> Pool[Thread pool<br/>fs, dns, crypto]
  Kernel -. callback ready .-> JS
  Pool -. callback ready .-> JS

For everyday work, two rules of thumb are enough. First: I/O is practically free in Node, because the waiting happens elsewhere – in the kernel or in the thread pool. Second: CPU time in the JavaScript thread is the most expensive resource of the whole system, because it is exclusive. Every millisecond a request spends computing is a millisecond in which all other requests stand still.

One blocking call blocks everyone#

This first really became clear to me in a production incident, not while reading the documentation. A reporting endpoint accepted JSON documents, normally a few kilobytes. One day, a client started sending exports of around 30 MB. The code looked harmless:

const http = require('http');

const server = http.createServer((req, res) => {
  let body = '';
  req.on('data', chunk => { body += chunk; });
  req.on('end', () => {
    // synchronous: the event loop stops while parsing
    const report = JSON.parse(body);
    res.end(JSON.stringify({ items: report.items.length }));
  });
});

server.listen(3000);

JSON.parse over 30 MB takes several hundred milliseconds. During that time, nothing else happens in the process: no new connections, no ongoing responses, no timers. The health check endpoint stopped answering too, the load balancer took the instance out of rotation, and monitoring reported the service as dead – while the process was running perfectly and simply computing. The incident was understood after an hour, but the lesson was bigger than the incident: in Node, there is no such thing as “one slow request”. There are only slow requests that take all the others down with them.

That makes the failure model fundamentally different from thread-per-request servers. There, a CPU hog shows up as one slow request in the statistics. In Node, it shows up as a latency spike across the entire system – including for requests that have nothing to do with the culprit.

Which systems fit – and which don’t#

If you take the model seriously, architecture decisions almost sort themselves. Node is strong when the process mostly waits and computes only briefly:

  • API gateways and proxies that check, enrich and forward requests
  • REST APIs in front of databases, where the actual work happens in storage
  • Aggregators that query several backends in parallel and assemble results
  • Push and real-time services with many open, mostly idle connections

All these systems have the same character: thousands of connections, almost all of which are waiting for something at any given moment. That is exactly what an event loop is built for – the waiting connections cost almost nothing, and one thread is enough for all of them.

Conversely, work with serious CPU load doesn’t belong in the request path of a Node process: image processing, PDF generation, encrypting large files, elaborate computations over data sets. Not because V8 is slow – the JIT compiler is astonishingly fast – but because every one of these milliseconds belongs to all requests at the same time.

In one project, we cut a gateway along this line: Node took over authentication, routing and assembling responses from three backends. Thumbnail generation for uploaded images, initially right in the middle of the upload handler, moved into a separate process with its own queue – started via child_process.fork and connected through messages. Upload latency dropped, but the real gain was a different one: the 99th percentile of all other endpoints calmed down, because no image occupied the shared thread anymore. The event loop had dictated the boundary along which to split the system – and it was a good boundary.

Backpressure: waiting needs organizing, too#

Async I/O doesn’t mean data flows at the right speed on its own. If you load a 2 GB export file with fs.readFile to send it to the client, you hold 2 GB in memory – per request. Streams solve this by passing data through in chunks:

const fs = require('fs');
const zlib = require('zlib');
const http = require('http');

http.createServer((req, res) => {
  res.setHeader('Content-Encoding', 'gzip');
  fs.createReadStream('./export.csv')
    .pipe(zlib.createGzip())
    .pipe(res);
}).listen(3000);

The interesting thing about pipe isn’t the compact notation but the built-in flow control. If the destination can’t keep up – that is, write() returns false – the reading stream pauses until the drain event signals that there is room again. This interplay is called backpressure. Without this brake, the disk reads at a few hundred MB/s while a mobile client receives at a fraction of that – the difference piles up as a buffer in the heap, and the process grows until memory runs out.

Backpressure is thus the second place where the runtime model becomes an architecture question: if you route uploads, exports or event streams through Node, you have to design data flow as something boundable – with streams as the standard tool and pipe as the connector, not with “everything into one variable and then send”.

Scaling means: more processes#

A Node process uses one core for JavaScript. On a machine with eight cores, seven lie idle as long as only one process runs. The platform’s answer is the cluster module: a master process starts one worker per core, all share the same port, and the master distributes incoming connections round-robin.

const cluster = require('cluster');
const os = require('os');

if (cluster.isMaster) {
  os.cpus().forEach(() => cluster.fork());
  cluster.on('exit', worker => {
    console.log(`worker ${worker.process.pid} died, forking a new one`);
    cluster.fork();
  });
} else {
  require('./server');
}
flowchart TB
  In[incoming connections<br/>port 3000] --> M[Master process<br/>round-robin distribution]
  M --> W1[Worker 1<br/>own event loop]
  M --> W2[Worker 2<br/>own event loop]
  M --> W3[Worker N<br/>own event loop]

Each worker has its own event loop and its own heap. That has two pleasant side effects and one inconvenient consequence. Pleasant: a crashed worker only takes itself down, and the master restarts it; and a blocked worker now only blocks an eighth of the system instead of everything. Inconvenient: there is no shared memory. Sessions, caches, counters – everything that used to sit comfortably in a module variable has to move out of the process, for example into Redis or the database. In my view, that isn’t a loss but an early push toward a state you need anyway for horizontal scaling across machines. If you get to eight workers with cluster, you later get to eight machines almost without changes.

The classification matters: cluster mitigates the blocking problem, it doesn’t solve it. The 50 ms calculation from the beginning applies unchanged in every worker. For real CPU work, the clean answer remains a separate process with a queue in front of it – cluster is load distribution, not a free pass for computation in the request path.

async/await: new syntax, same model#

Since a few weeks ago – Node 7.6, released in February 2017 – async/await runs without a flag. On Node 6 LTS, which many production systems run, you still need Babel for it. After years of callback pyramids and promise chains, the same flow suddenly reads like synchronous code:

async function loadDashboard(userId) {
  const user = await fetchUser(userId);
  const orders = await fetchOrders(user.id);
  return { user, orders };
}

// independent calls: run them concurrently instead
async function loadDashboardFast(userId) {
  const [user, orders] = await Promise.all([
    fetchUser(userId),
    fetchOrders(userId)
  ]);
  return { user, orders };
}

There is nothing synchronous about it. At every await, the function releases the thread; the event loop serves other callbacks and resumes the function as soon as the promise is resolved. That is exactly why two things belong in the same thought. First: two await lines one after the other run sequentially, even if they are independent – Promise.all remains responsible for parallelism, as in the second example. Second, and I consider this the real trap of the new syntax: await makes waiting cooperative, but CPU work between two await points blocks exactly as before. A JSON.parse over 30 MB doesn’t get more polite because there is an await in front of it. If you have moved from callbacks through promises to async/await, you have much better code – and an unchanged runtime model. The syntax is new; the math stays the same.

I collect runnable examples for all of this – event loop, streams, Express, async patterns from callbacks to streams – in a public repository: MikeBild/introduction-nodejs. The examples are deliberately kept small; each one shows exactly one of the limits described here.

The bottom line#

The event loop is a useful architecture test, and a cheap one. For every planned feature, you can ask: Does this code wait, or does it compute? Waiting work – database, HTTP, files, open connections – fits into the Node process in almost any quantity. Computing work belongs in separate processes, behind a queue, with cluster and child_process as built-in tools. Streams with working backpressure keep memory flat where large amounts of data flow.

I find it remarkable how much architecture follows from a single runtime property. Separating the gateway from image processing, stateless processes, queues in front of CPU work – none of this is in a framework handbook, all of it is in the calculation 200 × 50 ms. Platforms shape their systems, whether you want them to or not. Node’s upbringing is strict, but it points in a direction that pays off beyond Node, too: small processes that mostly wait, keep nothing important in local memory and limit their data flows. Once you have seen the 10 seconds from the beginning in a real dashboard, you won’t forget this direction.

Further reading#

Revised and translated from German with the help of AI.