Observability with OpenTelemetry: Logs, Traces and Metrics
Instrument logs, traces and metrics consistently with OpenTelemetry, connect them through trace IDs and export them via OTLP to suitable backends.
{"ts":"2024-07-09T12:14:03.221Z","level":"error","msg":"stock reservation failed","order_id":"4711"}
This line is structured, has a timestamp, a clean field for the order ID – and still answers none of the questions that matter after an outage. Which request triggered it? Which call ran before it, which one after? Is this an isolated case or the hundredth since noon? The line knows nothing about the operation it came from, and so searching the log volume remains exactly that: searching.
What’s missing is the connection – between the single event, the path of the request through the services and the aggregated view over time. That’s what logs, traces and metrics are for, and they only become really useful once they reference each other. OpenTelemetry is the standard I use to collect them consistently in Node.
The three telemetry signals aren’t interchangeable representations of the same information. Each answers a different question.
Three signals, three investigative questions
- Question · How does an aggregated value change over time?
- Strength · trends, thresholds and alerting
- Limit · doesn’t explain a single slow request
- Question · Which path did a specific execution take?
- Strength · latency and dependencies across service boundaries
- Limit · doesn’t replace long-term aggregation
- Question · Which discrete event was recorded with which context?
- Strength · details, error messages and domain hints
- Limit · hard to analyze without correlation and limits
The practical gain comes from switching between these perspectives: from the trend to the affected trace, and from there to the correlated events.
Three signals, three angles#
Before I talk about tools, it’s worth getting the terms straight. Logs, traces and metrics aren’t three ways of doing the same thing – they answer different questions and have different costs.
- Logs are discrete events with a timestamp: “Order 4711 rejected because stock is zero.” They deliver the detail, the specific case, the debug context. In exchange, they are expensive to retain and quickly get confusing at high cardinality.
- Traces show the path of a single request through the system: which service calls which, how long each hop takes, where the latency comes from. A trace is a tree of spans, and it doesn’t end at the service boundary.
- Metrics are aggregated number series over time: requests per second, error rate, latency distribution. They’re cheap to store and good for alerts and trends – but they never tell me which request was slow, only that there were some.
The mistake in thinking I see again and again is the assumption that one of these signals is enough. Metrics alone tell me that latency went up, but not why. Logs alone drown in volume. Traces alone show the path, but not the trend over a week. Only together do they form a picture – and the real gain comes where I can jump between them: from a latency spike in the metric to the specific trace, and from there into the correlated logs.
OpenTelemetry separates instrumentation and backend#
OpenTelemetry (OTel for short) is a vendor-neutral CNCF project, and that neutrality is exactly why I recommend it. Observability used to mean tying yourself to a backend: the vendor’s agent SDK was woven into the code, and switching meant rewriting the instrumentation. OTel separates the generation and transport of telemetry from the backend. I instrument my code against a standardized API and export the data via OTLP or a suitable exporter. That lowers switching costs considerably, but it doesn’t make backends interchangeable: query languages, aggregations, retention, supported signals and dashboards remain product-specific. A switch can therefore start with the collector configuration and still require adjustments to semantics and analysis.
A word about maturity, because in mid-2024 it isn’t the same everywhere. Tracing is stable in JavaScript, and so are metrics. The logs signal in JS, on the other hand, is still experimental – @opentelemetry/api-logs and sdk-logs carry the alpha stamp and are meant as a bridge for existing loggers, not as a direct log API for application code. In practice, that means I keep writing my logs with the logger of my choice and make sure myself that the trace correlation is in there. More on that later.
Setting up the SDK – early enough#
Instrumentation starts with the Node SDK. The key point I flag most often in reviews: the SDK has to start before any other module is loaded. OTel works by monkey-patching the libraries – if http or graphql are already loaded before the SDK could patch them, I get a no-op tracer and wonder about empty traces. That’s why I load the instrumentation via --require, not as a regular import in app.ts.
// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
const sdk = new NodeSDK({
resource: new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: 'checkout-service',
[SemanticResourceAttributes.SERVICE_VERSION]: '1.4.0',
}),
traceExporter: new OTLPTraceExporter({
url: 'http://collector:4318/v1/traces',
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: 'http://collector:4318/v1/metrics',
}),
}),
instrumentations: [
getNodeAutoInstrumentations({
// File-system spans are noisy; turn them off.
'@opentelemetry/instrumentation-fs': { enabled: false },
}),
],
});
sdk.start();
process.on('SIGTERM', () => {
sdk.shutdown().finally(() => process.exit(0));
});
The whole thing is started so that the instrumentation is guaranteed to run first:
node --require ./instrumentation.js ./app.js
getNodeAutoInstrumentations() covers a surprising amount without me touching a single line of business code: HTTP and HTTPS, Express or Fastify, GraphQL, gRPC, common database clients. To start with, I get a complete request tree through the standard layers practically for free. If you’re wondering in general how to keep an eye on a running Node process, Debugging and Observing Node.js covers the basics this builds on.
Manual spans for business logic#
Auto-instrumentation knows HTTP and GraphQL, but it doesn’t know my domain. Whether a checkout fails the credit check or runs out of stock is something no generic instrumentation plugin sees. That’s exactly what I use manual spans for – where the interesting domain decisions are made.
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('checkout', '1.4.0');
export async function processCheckout(order: Order): Promise<Receipt> {
return tracer.startActiveSpan('checkout.process', async (span) => {
span.setAttribute('order.id', order.id);
span.setAttribute('order.item_count', order.items.length);
try {
await reserveStock(order);
const receipt = await charge(order);
span.setAttribute('payment.provider', receipt.provider);
return receipt;
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}
Three things matter here. startActiveSpan makes the span the active context, so everything that happens inside the callback – including the auto-instrumentation spans for reserveStock and charge – is correctly attached as children. recordException plus setStatus marks the span as failed, which makes it findable in the backend. And span.end() in finally makes sure the span is closed even in the error case – a forgotten end() is a leak that shows up in the backend as a hanging span.
I’m careful with attributes. The order ID belongs there, and so does the item count. What doesn’t belong there is anything that blows up cardinality or is sensitive – full shopping carts, personal data, free text.
The trace doesn’t stop at the service boundary#
The real value of a trace shows as soon as several services are involved. A request hits the gateway, which calls the checkout service, which talks to the database and, on the side, to a payment provider. For this to remain one connected trace instead of falling apart into four unconnected fragments, the trace context has to travel across the boundaries.
By default, OpenTelemetry uses the W3C Trace Context for this: a traceparent header that carries the trace ID, span ID and sampling decision. Auto-instrumentation injects and extracts it automatically on HTTP calls – the receiver attaches its spans to the parent from the header, and the tree keeps growing across process boundaries.
graph TD
A[Client Request] -->|traceparent| B[API Gateway<br/>Span: http.request]
B -->|traceparent| C[Checkout Service<br/>Span: checkout.process]
C --> D[Span: db.query]
C -->|traceparent| E[Payment Provider<br/>Span: payment.charge]
B -.->|one trace_id| F[(One trace =<br/>tree of spans)]
C -.-> F
D -.-> F
E -.-> FIt gets interesting where there’s no HTTP header. In a Lambda Function or with a message queue, there’s no automatic transport for the context. Then I have to pass it through the event payload myself – write it into the message with propagation.inject when sending, pull it out with propagation.extract when receiving, and set the context as active. With serverless, two more points come in: the cold start lengthens the init phase, and before the function is frozen I have to trigger a force flush, otherwise the last spans get lost because the export couldn’t send them out anymore.
Linking logs to traces – through the trace_id#
Now to the point where, in my view, observability stands or falls: correlation. A log without a trace reference is a line in the middle of nowhere. A log with trace_id and span_id is a springboard – from the error message straight into the complete request path that produced it.
The active span lives in the context, and in Node the context is carried by AsyncLocalStorage – which is also the default context manager OTel uses internally. So all I have to do is read the active SpanContext and attach its IDs to every structured log. I described in detail how AsyncLocalStorage carries this implicit context through asynchronous call chains in Async Context with AsyncLocalStorage; here it’s the carrier that makes trace correlation possible in the first place.
import { trace } from '@opentelemetry/api';
type Fields = Record<string, unknown>;
export function log(level: 'info' | 'warn' | 'error', msg: string, fields: Fields = {}): void {
const span = trace.getActiveSpan();
const ctx = span?.spanContext();
const entry = {
ts: new Date().toISOString(),
level,
msg,
...(ctx && { trace_id: ctx.traceId, span_id: ctx.spanId }),
...fields,
};
process.stdout.write(JSON.stringify(entry) + '\n');
}
A call like log('error', 'stock reservation failed', { order_id: order.id }) in the middle of the business logic now produces a JSON line that carries the same trace_id as the span it was created in. Later, in the backend, I filter on that ID and have the log and trace side by side. In practice, I don’t wrap this myself but let an established logger like Pino handle it through a mixin – the principle stays the same: read the active SpanContext and attach it.
RED: Rate, Errors, Duration#
Metrics are the aggregated health of the system, and I don’t start with a hundred custom metrics, but with three. The RED method – Rate, Errors, Duration – covers the essentials for every service: how many requests come in, how many of them fail, how long they take.
import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('http-server');
const requestCounter = meter.createCounter('http.server.requests', {
description: 'Total number of HTTP requests',
});
const errorCounter = meter.createCounter('http.server.errors', {
description: 'Total number of failed HTTP requests',
});
const latencyHistogram = meter.createHistogram('http.server.duration', {
description: 'HTTP request duration in milliseconds',
unit: 'ms',
});
export function recordRequest(route: string, statusCode: number, durationMs: number): void {
const attributes = { route, status_class: `${Math.floor(statusCode / 100)}xx` };
requestCounter.add(1, attributes);
if (statusCode >= 500) {
errorCounter.add(1, attributes);
}
latencyHistogram.record(durationMs, attributes);
}
These three calls live in a middleware and run for every request. The counter for the rate, the counter for the errors, the histogram for the duration – the latter because with latency I’m not interested in the average but in the distribution, the p95 and p99. An average of 80 milliseconds hides that every twentieth user waits two seconds.
What matters with attributes is restraint. I label with route and a status class – values with a limited, manageable range. What I don’t do: use the full URL with an embedded order ID as a label, or the user ID. Every unique label combination creates its own time series, and unbounded values make cardinality explode until the metrics backend buckles. Cardinality belongs in traces, not in metrics.
The pitfalls that cost me time#
I see a few mistakes so regularly that I name them explicitly here. The most expensive one is the logging without correlation mentioned above – logs that are structured but carry no trace_id. They look tidy and still help little when it counts, because I can’t assign them to any request. The second is initializing the SDK too late, which makes the monkey-patching come to nothing.
The third concerns sampling. Capturing every trace in production rarely makes sense – volume and costs explode. So you sample. But naive head sampling that blindly keeps every hundredth trace is very likely to throw away exactly the error traces I want to see. That’s why I go with ParentBased(TraceIdRatio): the sampling decision is made at the start, propagated through the traceparent header and stays consistent for the whole trace – no service decides on its own, so I don’t get half traces. The fourth pitfall is the cardinality trap with custom metrics already mentioned.
Conclusion#
Observability isn’t a question of the right dashboard but of being able to answer any question about a running system – including the ones I didn’t think to ask beforehand. The three signals provide the raw data for that: logs for the specific event, traces for the request path, metrics for the aggregated health. They only unfold their real power through correlation, when I can jump from the metric spike to the trace and from there to the log without switching tools.
OpenTelemetry keeps instrumentation more vendor-neutral and lets you configure the export targets separately from it. Auto-instrumentation gets me close quickly, manual spans cover the domain, trace context propagation holds everything together across service boundaries, and linking logs to the trace_id via AsyncLocalStorage closes the loop. The start can be small: three RED metrics, auto-instrumentation, correlated logs. When you switch backends, you still need to check its query model, dashboards and quirks. By the next incident at the latest, the groundwork pays off – then the system is visible before anyone starts fixing it.
Further reading#
Revised and translated from German with the help of AI.