Consolidating Logs with Grafana Loki or OpenObserve
When you run one or two Docker containers on your laptop, checking errors is easy. You just run docker logs my-app and read the screen. But in a real production cluster with 30 microservices, containers crash, restart, and move across different servers all the time.
If a container crashes at 3:00 AM and gets replaced, its local logs are gone forever. To troubleshoot clusters efficiently, you need a centralized logging stack that automatically collects, stores, and lets you search stdout and stderr from every running container in one place.
- Why traditional
docker logsfails in multi-container clusters. - How Grafana Loki and OpenObserve compare for modern open-source logging.
- How to build a complete, working centralized logging stack using Docker Compose.
- How to collect
stdoutandstderrautomatically using log shippers like Promtail and Fluent Bit. - How to search, filter, and parse JSON logs using LogQL and SQL.
- Common architectural mistakes (like high cardinality) and real interview questions.
1. Why Centralized Logging Matters
Containers are designed to be temporary. Following the 12-Factor App methodology, your applications should never write logs to local text files inside the container. Instead, they should print logs directly to standard output (stdout) and standard error (stderr).
The container runtime (Docker or containerd) captures those streams and writes them as JSON files on the host machine. A centralized logging stack places a lightweight agent on each host to read those streams and ship them to a central database. Even if a container is deleted, its logs remain safe and searchable.
Grafana Loki vs. OpenObserve
For years, the ELK Stack (Elasticsearch, Logstash, Kibana) was the default choice, but it consumes huge amounts of RAM and disk space. Today, two modern open-source tools dominate lightweight cluster logging: Grafana Loki and OpenObserve.
| Feature | Grafana Loki | OpenObserve |
|---|---|---|
| Core Philosophy | "Like Prometheus, but for logs." Indexes only metadata labels, not the full log text. | Built in Rust. All-in-one observability platform for logs, metrics, and traces. |
| Query Language | LogQL (very similar to PromQL). |
Standard SQL and full-text search. |
| Storage Backend | Object storage (AWS S3, MinIO, GCS) or local filesystem. | Object storage (S3, MinIO) using compressed Parquet files, or local disk. |
| User Interface | Requires a separate Grafana container for the UI. | Built-in modern web UI included in the single binary. |
| Best For | Teams already using Prometheus and Grafana dashboards. | Teams wanting a fast, single-binary setup with SQL queries and low RAM usage. |
2. Architecture & Workflow Visualized
Whether you choose Grafana Loki or OpenObserve, the underlying architecture follows the exact same pipeline. Your application never needs to know where the logs are going.
Centralized Container Logging Pipeline (stdout / stderr to Search UI)
Containers write logs to stdout/stderr. The log shipper tails those streams from the host, attaches container metadata (name, service), and pushes compressed batches to Loki or OpenObserve.
3. Building the Stack with Grafana Loki
Let's build a complete, working logging stack using Docker Compose. This setup runs Grafana Loki for storage, Promtail to automatically discover and read logs from every running Docker container, and Grafana to search them.
Step 1: Create the Promtail Configuration
Promtail needs to know where to find Docker container logs and where to send them. Save this file as promtail-config.yaml.
Example 1: promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: docker_containers
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
- source_labels: ['__meta_docker_container_log_stream']
target_label: 'stream'
Step 2: Create the Docker Compose Stack
Now create a docker-compose.yml file in the same folder. We also include a tiny demo-app container that prints structured JSON logs every 3 seconds so you can immediately see logs flowing.
Example 2: docker-compose.yml (Loki + Promtail + Grafana)
services:
loki:
image: grafana/loki:3.0.0
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
promtail:
image: grafana/promtail:3.0.0
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./promtail-config.yaml:/etc/promtail/config.yml:ro
command: -config.file=/etc/promtail/config.yml
depends_on:
- loki
grafana:
image: grafana/grafana:11.0.0
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
depends_on:
- loki
demo-app:
image: alpine:3.19
container_name: payment-api
command: >
sh -c 'while true; do
echo "{\"level\":\"info\",\"service\":\"payment-api\",\"msg\":\"Order processed\",\"latency_ms\":42}";
sleep 3;
echo "{\"level\":\"error\",\"service\":\"payment-api\",\"msg\":\"Database timeout\",\"status\":503}" >&2;
sleep 5;
done'
Run the stack and verify
docker compose up -d
docker compose ps
Expected output
NAME IMAGE STATUS payment-api alpine:3.19 Up 10 seconds loki-1 grafana/loki:3.0.0 Up 10 seconds promtail-1 grafana/promtail:3.0.0 Up 9 seconds grafana-1 grafana/grafana:11.0.0 Up 9 seconds
Open your browser and go to http://localhost:3000. Add Loki as a data source using the URL http://loki:3100, then click Explore to start searching your container logs.
4. Alternative: All-in-One Logging with OpenObserve
If you prefer a single-container backend with a built-in UI and standard SQL querying, OpenObserve is an excellent choice. You do not need a separate Grafana container.
Example 3: Running OpenObserve with Docker Compose
services:
openobserve:
image: public.ecr.aws/zinclabs/openobserve:latest
container_name: openobserve
ports:
- "5080:5080"
environment:
- ZO_ROOT_USER_EMAIL=admin@upgradix.local
- ZO_ROOT_USER_PASSWORD=ComplexPass#123
- ZO_DATA_DIR=/data
volumes:
- ./o2-data:/data
Once running, open http://localhost:5080 and log in. OpenObserve gives you a ready-to-copy Fluent Bit or Vector configuration under the Ingestion → Logs tab. Fluent Bit tails your container stdout/stderr and sends JSON batches directly to OpenObserve's HTTP API.
SELECT * FROM "default" WHERE level = 'error' AND status = 503.5. Searching & Parsing Logs in LogQL
In Grafana Loki, you query logs using LogQL. Every query starts with a stream selector inside curly braces {}, followed by optional filters and parsers.
Example 4: Common LogQL Troubleshooting Queries
# 1. Get all logs from the payment-api container
{container="payment-api"}
# 2. Show only stderr (error stream) logs
{container="payment-api", stream="stderr"}
# 3. Parse JSON logs and filter where status is 503
{container="payment-api"} | json | status = 503
# 4. Count error logs per minute (great for alerts!)
sum(count_over_time({container="payment-api"} | json | level="error" [1m]))
Expected parsed log match
2026-11-18 20:51:12 {"level":"error","service":"payment-api","msg":"Database timeout","status":503}
Extracted fields: level="error" service="payment-api" status="503" stream="stderr"
6. Step-by-Step Execution Breakdown
Here is exactly what happens under the hood from the moment your code prints an error to the moment you see it in your browser.
stdout or stderr./var/lib/docker/containers/<id>/<id>-json.log.container="payment-api" and stream="stderr".user_id, request_id, or ip_address as Loki labels. Labels create separate index streams. High cardinality will crash Loki or make queries painfully slow. Keep those values inside the JSON log body and parse them at query time using | json.7. Hands-On Practice
Run the Docker Compose stack from Section 3. Open Grafana Explore and write a LogQL query that shows only the stderr logs from the payment-api container.
Hint: Use the label selector {container="payment-api", stream="stderr"}.
Modify the demo-app command to log random latency values, then write a LogQL query to find only requests where latency_ms is greater than 40.
Expected result: Grafana returns only the JSON log lines matching | json | latency_ms > 40.
Even with centralized logging, Docker's local JSON log files can fill up your server's hard drive. Add a logging block to your demo-app service in docker-compose.yml to limit local log files to 10MB and keep a maximum of 3 files.
Hint: Use the json-file driver with max-size: "10m" and max-file: "3".
8. Interview Questions
Answer: Containers are ephemeral (temporary). Writing to local files inside a container couples the app to the filesystem, risks filling up container disk space, and loses logs when the container restarts. Writing to stdout and stderr lets the container runtime and external log shippers handle rotation, collection, and storage uniformly across all services.
Answer: Elasticsearch builds a full-text inverted index for every word in every log line, which requires massive RAM and disk space. Grafana Loki only indexes a small set of metadata labels (like app, env, container) and stores the actual log lines as heavily compressed chunks in cheap object storage like AWS S3.
Answer: Cardinality refers to the number of unique values a label can have. A label like env="prod" has low cardinality (maybe 3 values). A label like trace_id or user_id has high cardinality (millions of unique values). In Loki, every unique combination of labels creates a new log stream in the index. High cardinality explodes index size and degrades query performance.
Answer: OpenObserve is a great choice when a team wants a single-binary, all-in-one platform (logs, metrics, and traces) with a built-in UI, prefers standard SQL over learning LogQL, and wants fast full-text search backed by compressed Parquet files on S3 with very low operational overhead.
Conclusion
Consolidating container logs is one of the highest-impact upgrades you can make to any cluster. By printing structured JSON to stdout and stderr and shipping it to Grafana Loki or OpenObserve, you turn stressful 3:00 AM outages into fast, searchable investigations.
Start small with the Docker Compose stack above, practice writing a few LogQL or SQL queries, and always remember to keep your labels clean and your host log rotation enabled.
Official Documentation
- Grafana Loki Documentation – Architecture & LogQL
- OpenObserve Documentation – Quickstart & Ingestion
- Docker Documentation – Configure Logging Drivers
Upgradix by Shrash Studio – Learn Technology. Build Your Future.