Buffers in Node.js: Working with Binary Data
Why JS strings don’t work for raw bytes and Buffer is Node’s tool for them: alloc versus allocUnsafe, reading and writing encodings cleanly, and the memory trap behind slice and subarray.
A Buffer isn’t a broken string. If <Buffer 89 50 4e 47 0d 0a 1a 0a ...> shows up in the console after reading a file, nothing went wrong: Node shows exactly what’s in the file – a sequence of bytes that has no meaning as text and isn’t supposed to get one.
In Node, such bytes come from many directions: from fs, from open network connections, from crypto. The Buffer is the type Node holds them in. What makes it interesting are mainly the places where it behaves differently from what you’re used to with strings and arrays – when creating it and when splitting it up.
Why strings aren’t enough here#
A JavaScript string is a sequence of Unicode characters, not a sequence of bytes. That sounds like hair-splitting, but it’s the core of the problem. As soon as I have a string, the runtime has already made an interpretation: these bytes are text, in a specific encoding. But raw binary data – a PNG header, a compressed block, a cryptographic key – has no meaning as text at all. If I force it through a string, I lose or corrupt bytes, because invalid byte sequences get replaced with replacement characters. Often this is only noticed late, for example when an uploaded file arrives damaged at the other end.
Node solves this with Buffer: a region of fixed byte length that contains exactly the bytes you put into it, without any reinterpretation. One detail is crucial here and explains a lot: a Buffer is a subclass of Uint8Array, so it’s a TypedArray itself. Everything you know from Uint8Array – index access, length, for loops, the TypedArray methods – works on a Buffer too. Buffer merely adds the extras that are practical for Node, first and foremost the encodings.
const fs = require('node:fs');
const data = fs.readFileSync('image.png');
console.log(Buffer.isBuffer(data)); // true
console.log(data instanceof Uint8Array); // true
console.log(data.length); // number of bytes
console.log(data.subarray(0, 8)); // <Buffer 89 50 4e 47 0d 0a 1a 0a>
By the way, the first eight bytes are the fixed signature of a PNG file. In practice, you read exactly such headers to detect file types without relying on the extension.
Creating a buffer: alloc, allocUnsafe, from#
There are three ways to get a Buffer, and the differences matter – for security reasons too.
Buffer.alloc(size)reservessizebytes and sets them all to0. Optionally, you pass a fill byte and an encoding. This is the safe default when you need an empty buffer of a certain size.Buffer.allocUnsafe(size)also reservessizebytes but doesn’t initialize them. The content is old memory – whatever was there before is still there. In return, creating it is measurably faster, because zeroing is skipped and the memory may come from an internal pool.Buffer.from(...)creates a buffer from existing data: from a string (with an encoding), from an array of byte values, from another buffer (copied) or from anArrayBuffer.
The difference between the first two isn’t a detail. allocUnsafe returns memory that may contain leftovers from earlier allocations – possibly data from completely different parts of the program.
const zeroed = Buffer.alloc(8);
console.log(zeroed); // <Buffer 00 00 00 00 00 00 00 00>
const uninitialized = Buffer.allocUnsafe(8);
console.log(uninitialized); // <Buffer ?? ?? ?? ?? ?? ?? ?? ??> old memory
That’s why alloc is the default; allocUnsafe is only worth it where the buffer is completely overwritten right afterward anyway. If you use allocUnsafe and only partially fill the memory, you must never output or send the rest, because sensitive data may be in there. That’s the first of the two classic traps, and it’s treacherous because it never shows up in a local test – in a freshly started process, the memory often happens to be zeroed. Only under load, when the pool is reused, do the data leftovers appear.
As an aside, because it still haunts old code: the former new Buffer(...) call has been deprecated at runtime since Node 10 (DEP0005) and shouldn’t be used anymore. And Buffer.from(10) doesn’t create a buffer of size 10 but throws a TypeError – a number isn’t a valid argument for Buffer.from. If you mean a size, use alloc or allocUnsafe.
Encodings: the same bytes, different glasses#
The second big benefit of Buffer over a bare Uint8Array is the encodings. An encoding isn’t a conversion of the bytes but a rule for how to read the same bytes as text or how to break text down into bytes. The round trip between the two sides looks like this:
graph LR
S["String<br/>(Unicode text)"] -->|"Buffer.from(str, enc)"| B["Buffer<br/>(raw bytes)"]
B -->|"buf.toString(enc)"| S
B -->|"buf.toString('hex')"| H["Hex text"]
B -->|"buf.toString('base64')"| B64["Base64 text"]When writing in with Buffer.from(str, enc), I say: break this text down into bytes according to this rule. When reading out with buf.toString(enc), I say: interpret these bytes as text according to this rule. What matters is that both sides mean the same encoding, otherwise you get nonsense.
The common encodings fall into two groups. The first describe text to bytes: utf8 (the default), utf16le, latin1 (also called binary) and ascii. The others are byte-to-text representations that make arbitrary binary data printable: hex, base64 and base64url. You need hex and base64 in particular all the time, for example to make a hash readable or to transport binary data in JSON.
const buf = Buffer.from('Grüße', 'utf8');
console.log(buf.length); // 7 bytes, not 5
console.log('Grüße'.length); // 5 characters
console.log(buf.toString('hex')); // 4772c3bcc39f65 byte view as hex
console.log(buf.toString('base64')); // R3LDvMOfZQ== byte view as base64
const b64 = buf.toString('base64');
const back = Buffer.from(b64, 'base64').toString('utf8');
console.log(back); // 'Grüße' again
The interesting point is in the first two lines: the string has five characters, but the buffer has seven bytes. The ü and the ß each take up two bytes in UTF-8. Bytes and characters simply aren’t the same thing, and if you mix up length, you calculate offsets wrong. In practice, a related problem arises from exactly this: with streams, a multibyte character can be cut apart at the point where one data chunk ends and the next begins. For clean decoding across such chunk boundaries, there is the string_decoder, which buffers half characters in between – but that’s a topic of its own, covered in more detail in the post Node.js streams and backpressure.
Numbers as bytes: byte order#
Encodings take care of text. But as soon as you take apart a binary format – a file header, a network protocol, a frame – it’s about numbers spread across several bytes. For this, Buffer comes with a series of read and write methods that interpret a fixed number of bytes at an offset as an integer: readUInt8, readUInt16BE, readUInt32LE, plus the signed variants and, for 64 bits, the BigInt versions such as readBigUInt64BE. The counterpart for writing is called writeUInt32BE and so on.
The suffix at the end is where many people stumble: BE stands for big-endian, LE for little-endian – that is, for the question of whether the most significant byte comes first or last. If you pick a different order when reading than the writer used, you get a completely different number, without any error being thrown. Network protocols are traditionally big-endian, and many file formats define their own order; so you have to look it up in the respective format and must not guess.
const buf = Buffer.alloc(4);
buf.writeUInt32BE(0x89504e47, 0); // PNG signature bytes as one number
console.log(buf); // <Buffer 89 50 4e 47>
console.log(buf.readUInt32BE(0).toString(16)); // 89504e47
console.log(buf.readUInt32LE(0).toString(16)); // 474e5089, reversed bytes
The same four bytes produce two different numbers depending on the chosen order. The buffer itself doesn’t know what’s right – the meaning lies in the format, not in the bytes.
The memory trap: slice and subarray share memory#
Now to the second big trap, and it snaps shut especially often because arrays behave differently. When you call slice on an ordinary JavaScript array, you get a copy. With a buffer, it’s different: buf.subarray(start, end) and the older buf.slice(start, end) both create no copy but a view onto the same underlying memory. If you write into the view, you change the original too.
graph TD M["Byte block in memory<br/>[1, 2, 3, 4, 5]"] O["original<br/>(whole block)"] --> M V["view = original.subarray(1, 3)<br/>(bytes 1 and 2)"] --> M
Both names, original and view, point to the same memory. A write through one is visible through the other.
const original = Buffer.from([1, 2, 3, 4, 5]);
const view = original.subarray(1, 3); // shares memory, no copy
view[0] = 99;
console.log(view); // <Buffer 63 03>
console.log(original); // <Buffer 01 63 03 04 05> byte 1 changed too!
The byte at position 1 in the original has changed, even though I seemingly only wrote into view. In a function that returns a partial buffer which the caller then modifies, this leads to bugs that are hard to find, because cause and effect sit in completely different places in the code.
If you need an independent copy, you have to create it explicitly – for example with Buffer.from over the view or with buf.copy into a freshly reserved target buffer.
const original = Buffer.from([1, 2, 3, 4, 5]);
const copy = Buffer.from(original.subarray(1, 3)); // independent copy
copy[0] = 99;
console.log(copy); // <Buffer 63 03>
console.log(original); // <Buffer 01 02 03 04 05> untouched
On the naming question, because it comes up often: the Node documentation now recommends subarray. The name slice only still exists on buffers for compatibility reasons and deliberately differs from the behavior of TypedArray.prototype.slice, which returns a copy. This inconsistency grew historically and is a common source of confusion. I recommend consistently writing subarray for buffers – then anyone reading the code immediately knows that memory is shared here and no copy is created.
Where buffers show up in everyday work#
You rarely create buffers by hand; usually the platform hands them to you. The fs module returns a buffer when reading without an encoding. Streams deliver their data in buffer chunks unless you configure them as text. The net network module passes received packets on as buffers. And crypto works with buffers almost throughout – a hash, for example, is created as a buffer that you then make readable with toString('hex').
const crypto = require('node:crypto');
const hash = crypto.createHash('sha256')
.update('hello world') // accepts string or Buffer
.digest(); // returns a Buffer
console.log(Buffer.isBuffer(hash)); // true
console.log(hash.toString('hex')); // b94d27b9934d3e08a52e52d7da7dabfa...
Here you can see the two roles of the buffer nicely together: update accepts text or bytes, digest returns raw bytes, and toString('hex') makes those bytes readable for logs and comparisons. If you want to understand which execution model these inputs and outputs run in and why that doesn’t block, you’ll find the background in the post Node.js Training on event loop architecture.
What sticks#
Three things go a long way. First: a buffer is a Uint8Array with Node extras – if you keep that in mind, you immediately understand index access and length and don’t expect any string magic. Second: alloc is the safe default, allocUnsafe a deliberate compromise for cases where you overwrite the memory completely anyway; anything else risks exposing old data. Third: subarray and slice share memory and don’t copy – if you need an independent version, copy by hand.
The two traps – uninitialized memory from allocUnsafe and the shared view from subarray – have one thing in common: they don’t show up in a quick test and only strike later or under load. That’s exactly why it pays to play through the behavior deliberately once instead of leaving it to chance. File access, networking and cryptography in Node run on raw bytes; once you have understood the buffer, you read their output in the console as what it is: a representation of the data without any reinterpretation.
Further reading#
Revised and translated from German with the help of AI.