Skip to content

Central Logging with the ELK Stack in Docker

Container logs are scattered across many containers and they are ephemeral. This article shows how an ELK stack of Elasticsearch, Logstash and Kibana in a Compose setup centralizes logs, indexes them and makes them searchable.

≈ 14 min readSoftwarePlatform

docker logs <container> scales up to exactly one. With a frontend, an API, a database and a cache, that’s four calls in four terminals; with three hosts and a few replicas, it quickly becomes fifteen – and each one shows only its own slice, and only as long as the container lives.

Editorial note (July 2026): The example deliberately uses Docker Compose and ELK 6.4 from 2018. It is not a current production template. Regardless of the version, the Docker GELF driver supports both UDP and TCP; the earlier claim that it could only do UDP was already wrong in 2018.

The decision behind it is less a technical one than a strategic one. The question isn’t “which logging driver do I configure?” but “how much visibility into my system do I want, and what do I pay for it in storage, compute and operational effort?” Central logging isn’t a feature you switch on. It’s an investment with running costs. If you treat the ELK setup as a pure API exercise, you will reliably build a system that either shows too little or fills up the disk. That’s why placing logs in context comes here before the first reconfigured container.

This is exactly where the ELK stack comes in. In this article I walk the path I also take in projects: first place logs within the observability strategy, then understand where Docker actually writes the logs and which logging drivers exist, then set up a complete ELK stack with Docker Compose, send the logs of an app container there and finally make them searchable in Kibana. Along the way I collect the pitfalls I’ve run into most often in practice.

Logs as one of three pillars#

Before you switch a single driver, the context needs to be clear: logs are only one of three pillars that visibility into a distributed system rests on, and none of the three replaces the others. Metrics are aggregated numbers over time – requests per second, error rate, memory usage – and answer “is something broken right now, and since when?”. Logs are individual events with context and answer “what exactly happened in this one case?”. Traces follow a request across several services and answer “where in the path is it stuck?”. The ELK stack covers the middle pillar, and it helps enormously to know that consciously: if you try to watch an error rate over time with logs alone, you pay far more than a simple metric would have cost.

And that brings us to the heart of the matter: cost. Logs are the most tempting and at the same time the most expensive of the three pillars, because every single line is stored and, in Elasticsearch, indexed by default. The cost driver is called cardinality. A field with few possible values – say a log_level with info, warn, error – is cheap to index. A field with an almost unlimited number of distinct values – a request_id, a user_id, a full query string – drives up the size of the index and with it the storage and RAM requirements. With metrics, such high cardinality quickly becomes an existential threat; with logs it’s expensive but manageable, as long as you control it deliberately.

That control is the actual design work, and it comes down to a few levers. First: what ends up in the log at all, and at which level? A DEBUG that keeps running in production is a pure cost driver with no return. Second: how long do I keep logs? The daily index I build further down is exactly the lever for that – entire days can be deleted in one go. Third: does every event really need to be fully indexed, or is a sample enough at high volume? The rule behind it is simple and still annoying to implement: you don’t log everything that’s possible, you log what you actually want to read when something goes wrong – and you don’t pay for the rest. The rest of this article shows the mechanics; whether they are set up with cost awareness decides whether the setup still holds after three weeks.

Where Docker logs end up by default#

Docker has a concept that surprises many people at first: it doesn’t care what happens in your application, only what gets written to STDOUT and STDERR. Everything the process with PID 1 in the container writes to these two streams is captured by Docker and handed to a so-called logging driver.

The default driver is called json-file. It writes each log line as a JSON object to a file on the host, typically under /var/lib/docker/containers/<id>/<id>-json.log. This is exactly the file docker logs reads from. That’s also the reason for a quirk that causes confusion in practice: docker logs only works with json-file (and journald). If you switch the driver to gelf or syslog because you want to send the logs somewhere else, the local docker logs command for that container is empty. The logs haven’t disappeared, they just aren’t kept locally anymore.

The drivers available as of 2018 are quite a list:

  • json-file and journald – local, readable with docker logs.
  • syslog, gelf, fluentd, splunk – forwarding to a central receiver.
  • awslogs, gcplogs, etwlogs – platform-specific targets.
  • none – logging switched off entirely.

For centralization, gelf and syslog are the most interesting. They send the log lines over the network to an endpoint – and that endpoint can be Logstash.

You can set a logging driver per container with docker run:

docker run \
  --log-driver=gelf \
  --log-opt gelf-address=udp://logstash:12201 \
  --log-opt tag="myapp" \
  myorg/myapp:latest

In Compose, the same thing per service looks like this:

version: "3"
services:
  app:
    image: myorg/myapp:latest
    logging:
      driver: "gelf"
      options:
        gelf-address: "udp://logstash:12201"
        tag: "myapp"

A detail worth knowing: the gelf driver supports UDP and TCP. This example uses UDP. It’s fast and lean, but it doesn’t confirm receipt – if a message gets lost under load, it’s gone. TCP is connection-oriented, but that still doesn’t turn a log transport into a tamper-proof audit trail.

The three roles in the ELK stack#

Before we write Compose, it’s worth a look at the three letters. ELK stands for Elasticsearch, Logstash and Kibana, and each component has exactly one clearly defined job. Elasticsearch stores and indexes: it’s the search engine at the center that makes every log line searchable and holds the data. Logstash receives, parses and transforms – it’s the front door that accepts logs from various sources, gives them structure and passes them on to Elasticsearch; its pipeline consists of input, filter and output. And Kibana visualizes and searches: the web UI where you filter logs, build dashboards and make patterns visible.

The data flow is a short chain:

flowchart LR
    A[App Container] -->|stdout / gelf| B[Logstash]
    B -->|index| C[Elasticsearch]
    C -->|query| D[Kibana]
    E[Browser] -->|http :5601| D

The app writes to STDOUT, the Docker driver forwards to Logstash, Logstash indexes into Elasticsearch, and Kibana makes the whole thing visible in the browser. The important part: the application itself knows nothing about any of this. It logs to STDOUT and STDERR – the way the Twelve-Factor principle intends – and transport is the infrastructure’s job. No custom log file handling in the container, no rotation logic in the app. That keeps the application clean and makes the log path replaceable.

The Compose stack#

Now let’s build the stack. I deliberately pin the images to a fixed version – in this case the 6.4 line – because an untagged elasticsearch:latest is the most reliable source of “but it looks different on my machine.” Here’s the elk-compose.yml:

version: "3"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"

  logstash:
    image: docker.elastic.co/logstash/logstash:6.4.0
    volumes:
      - ./pipeline:/usr/share/logstash/pipeline
    ports:
      - "12201:12201/udp"
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:6.4.0
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

volumes:
  es-data:

A few points in the configuration are there on purpose. Elasticsearch gets discovery.type=single-node so it doesn’t look around in vain for cluster peers. The heap is fixed at 512 megabytes with ES_JAVA_OPTS – that’s enough for trying things out; in production you would size it more generously, but always with the same Xms and Xmx. And the data lives on a named volume es-data so it survives a container restart. That’s not a detail: if you forget the volume, every docker-compose down loses all indexed logs – especially annoying for, of all things, the log system.

Kibana gets the address of Elasticsearch through ELASTICSEARCH_URL. The service name elasticsearch works as a hostname because Compose puts all services on the same network and provides name resolution.

What’s still missing is the Logstash pipeline. It lives as pipeline/logstash.conf next to the Compose file and defines how Logstash accepts messages and where it writes them:

input {
  gelf {
    port => 12201
  }
}

filter {
  # room for grok, mutate, date & co.
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "docker-logs-%{+YYYY.MM.dd}"
  }
}

The gelf input listens on port 12201 – exactly the port our app service sends its logs to. The filter block is empty here, but in real setups that’s where the actual work happens: splitting lines into fields with grok, normalizing timestamps with date, renaming fields with mutate. The output block writes to Elasticsearch, into a daily index docker-logs-2018.09.18. This pattern is chosen on purpose: one index per day makes cleaning up old logs easy later, because you can delete whole indexes instead of hunting for individual documents.

Connecting the app#

What’s still missing is the application whose logs we want to see. I add an app service to the same stack and give it the gelf logging driver:

  app:
    image: myorg/myapp:latest
    logging:
      driver: "gelf"
      options:
        gelf-address: "udp://localhost:12201"
        tag: "myapp"
    depends_on:
      - logstash

The gelf driver has a stumbling block that almost certainly shows up the first time you set it up: the gelf-address is resolved by the Docker daemon on the host, not from inside the container network. That’s why it says udp://localhost:12201 with the mapped host port, not the service name logstash. The daemon doesn’t know Compose’s internal DNS – it only sees the port Logstash publishes on the host.

When you start the stack with docker-compose -f elk-compose.yml up, Elasticsearch comes up first, then Logstash and Kibana, then the app. From now on, every log line of the app flows as a GELF message to Logstash, gets indexed and is visible in Kibana.

A word about ordering: depends_on only controls the order in which Docker starts the containers – not whether the respective service is ready yet. After starting, Elasticsearch needs a few seconds until the index is green, and during that time Kibana likes to report “Kibana server is not ready yet” in the browser. That’s not an error, that’s the reality of asynchronous startup. Take a breath, reload the page.

Searching in Kibana#

Kibana runs on port 5601. On first access it asks for an index pattern – you enter docker-logs-*, pick the time field, and the Discover tab already shows the log lines trickling in. From here you filter by the tag field (myapp), run full-text searches on the messages, narrow down time ranges and build visualizations if needed. The aha moment reliably comes right here: scattered docker logs calls across several terminals turn into a single searchable UI.

Pitfalls from practice#

Three things go wrong most often with this setup, and I’ve experienced all three live more than once.

The classic is that Elasticsearch doesn’t even start and the log says max virtual memory areas vm.max_map_count [65530] is too low. Elasticsearch 6.x requires vm.max_map_count=262144 on the host. That’s a kernel setting, not a container parameter, and it has to be set on the Docker host:

sudo sysctl -w vm.max_map_count=262144

On a Docker Machine with VirtualBox, you set it in the VM accordingly. If you forget it, no container restart helps – Elasticsearch dies reproducibly on startup.

The second pitfall is log volume. The default driver json-file writes to disk without limit. A chatty container can fill up a development machine overnight. That’s why json-file always needs a limit:

    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

And for the UDP transport of gelf chosen here, what I touched on above applies: under load, messages can get lost without anything blowing up anywhere. For critical logs, GELF over TCP or a buffered collector are options. Whether a log has to be complete, immutable and provable remains a separate business requirement beyond that.

The third point is the startup coupling already mentioned. Logstash and Kibana need a reachable Elasticsearch. depends_on only guarantees the start, not readiness. In production you solve that with health checks and retry logic; when trying things out locally, it’s usually enough to know that “not ready” in the first few seconds is normal and no reason to panic.

Two ways of forwarding#

Finally, a bit of context that shows the logging driver isn’t the only way. There are basically two approaches to getting logs from containers to Logstash. The first is the one we took here: the Docker logging driver on the app service forwards directly. The second is a log collector as a sidecar – an additional container such as logspout that reads the Docker socket and forwards the collected logs of all containers via syslog. The advantage of the collector approach is that you don’t have to change anything on the app services and docker logs keeps working, because the containers stay on the default driver. The downside is one more moving part in the stack. Which way fits better depends on how much control you have over the individual services – with third-party images, the collector is often the more pragmatic choice.

If you want to embed the ELK stack in a larger Compose setup, Node.js and Redis with Docker Compose covers the basics for multi-part stacks. Compose keeps several parallel projects apart through their own project names and the networks derived from them.

Conclusion#

Container logs are local and ephemeral by default – that’s no problem with one container and a big one with twenty. The ELK stack solves this with a clear division of labor: Logstash receives and transforms, Elasticsearch stores and indexes, Kibana makes things searchable. With its logging drivers, Docker provides the bridge to get a container’s logs there in the first place, without having to touch the application.

The effect I care about is a shift in day-to-day work. The path from incident to cause gets shorter, because you no longer piece together docker logs across several terminals but filter in one UI. Friction drops, because the app stays clean – it logs to STDOUT and STDERR, and transport stays the infrastructure’s job and therefore replaceable. And the quality of diagnoses goes up, because a searchable history reveals patterns that get lost in an ephemeral single log. If you know the three big pitfalls – vm.max_map_count, unlimited log volume and readiness at startup – you can cover this path in an afternoon.

Knowing the limits is just as important. Logs are one of several telemetry signals, not all of observability: for trends over time, metrics are cheaper; for the path of a request through several services, traces are the right tool. Costs grow with volume and cardinality, which is why retention, log levels and the question of which fields get indexed have to remain deliberate decisions. And the gelf over UDP used here is fine for non-critical application logs, but not for audit trails or billing data, where no message may get lost. Within these limits, the core stays simple: the app logs to STDOUT and STDERR, everything else is infrastructure – and what that infrastructure costs is up to you.

Further reading#

Revised and translated from German with the help of AI.