LATEST 🚀 Welcome to Upgradix by Shrash Studio – Your destination for free technology learning and career growth.
🕘 Updated:
Kubernetes

Consolidating Logs with Grafana Loki or OpenObserve

When you run one or two Docker containers on your laptop, checking errors is easy. You just run docker logs my-app and read the screen. But in a real production cluster with 30 microservices, containers crash, restart, and move across different servers all the time.

Consolidating Logs with Grafana Loki or OpenObserve

When you run one or two Docker containers on your laptop, checking errors is easy. You just run docker logs my-app and read the screen. But in a real production cluster with 30 microservices, containers crash, restart, and move across different servers all the time.

If a container crashes at 3:00 AM and gets replaced, its local logs are gone forever. To troubleshoot clusters efficiently, you need a centralized logging stack that automatically collects, stores, and lets you search stdout and stderr from every running container in one place.

What you will learn
  • Why traditional docker logs fails in multi-container clusters.
  • How Grafana Loki and OpenObserve compare for modern open-source logging.
  • How to build a complete, working centralized logging stack using Docker Compose.
  • How to collect stdout and stderr automatically using log shippers like Promtail and Fluent Bit.
  • How to search, filter, and parse JSON logs using LogQL and SQL.
  • Common architectural mistakes (like high cardinality) and real interview questions.

1. Why Centralized Logging Matters

Containers are designed to be temporary. Following the 12-Factor App methodology, your applications should never write logs to local text files inside the container. Instead, they should print logs directly to standard output (stdout) and standard error (stderr).

The container runtime (Docker or containerd) captures those streams and writes them as JSON files on the host machine. A centralized logging stack places a lightweight agent on each host to read those streams and ship them to a central database. Even if a container is deleted, its logs remain safe and searchable.

Grafana Loki vs. OpenObserve

For years, the ELK Stack (Elasticsearch, Logstash, Kibana) was the default choice, but it consumes huge amounts of RAM and disk space. Today, two modern open-source tools dominate lightweight cluster logging: Grafana Loki and OpenObserve.

Feature Grafana Loki OpenObserve
Core Philosophy "Like Prometheus, but for logs." Indexes only metadata labels, not the full log text. Built in Rust. All-in-one observability platform for logs, metrics, and traces.
Query Language LogQL (very similar to PromQL). Standard SQL and full-text search.
Storage Backend Object storage (AWS S3, MinIO, GCS) or local filesystem. Object storage (S3, MinIO) using compressed Parquet files, or local disk.
User Interface Requires a separate Grafana container for the UI. Built-in modern web UI included in the single binary.
Best For Teams already using Prometheus and Grafana dashboards. Teams wanting a fast, single-binary setup with SQL queries and low RAM usage.

2. Architecture & Workflow Visualized

Whether you choose Grafana Loki or OpenObserve, the underlying architecture follows the exact same pipeline. Your application never needs to know where the logs are going.

Centralized Container Logging Pipeline (stdout / stderr to Search UI)

API Container stdout / stderr Worker Container stdout / stderr DB Container stdout / stderr Log Shipper Promtail / Fluent Bit Adds Labels & Batches HTTP Log Engine Loki or OpenObserve Compresses to S3 / Disk Search UI LogQL / SQL

Containers write logs to stdout/stderr. The log shipper tails those streams from the host, attaches container metadata (name, service), and pushes compressed batches to Loki or OpenObserve.

3. Building the Stack with Grafana Loki

Let's build a complete, working logging stack using Docker Compose. This setup runs Grafana Loki for storage, Promtail to automatically discover and read logs from every running Docker container, and Grafana to search them.

Step 1: Create the Promtail Configuration

Promtail needs to know where to find Docker container logs and where to send them. Save this file as promtail-config.yaml.

Example 1: promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker_containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'

Step 2: Create the Docker Compose Stack

Now create a docker-compose.yml file in the same folder. We also include a tiny demo-app container that prints structured JSON logs every 3 seconds so you can immediately see logs flowing.

Example 2: docker-compose.yml (Loki + Promtail + Grafana)

services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./promtail-config.yaml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - loki

  demo-app:
    image: alpine:3.19
    container_name: payment-api
    command: >
      sh -c 'while true; do
        echo "{\"level\":\"info\",\"service\":\"payment-api\",\"msg\":\"Order processed\",\"latency_ms\":42}";
        sleep 3;
        echo "{\"level\":\"error\",\"service\":\"payment-api\",\"msg\":\"Database timeout\",\"status\":503}" >&2;
        sleep 5;
      done'

Run the stack and verify

docker compose up -d
docker compose ps

Expected output

NAME          IMAGE                    STATUS
payment-api   alpine:3.19              Up 10 seconds
loki-1        grafana/loki:3.0.0       Up 10 seconds
promtail-1    grafana/promtail:3.0.0   Up 9 seconds
grafana-1     grafana/grafana:11.0.0   Up 9 seconds

Open your browser and go to http://localhost:3000. Add Loki as a data source using the URL http://loki:3100, then click Explore to start searching your container logs.

4. Alternative: All-in-One Logging with OpenObserve

If you prefer a single-container backend with a built-in UI and standard SQL querying, OpenObserve is an excellent choice. You do not need a separate Grafana container.

Example 3: Running OpenObserve with Docker Compose

services:
  openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    container_name: openobserve
    ports:
      - "5080:5080"
    environment:
      - ZO_ROOT_USER_EMAIL=admin@upgradix.local
      - ZO_ROOT_USER_PASSWORD=ComplexPass#123
      - ZO_DATA_DIR=/data
    volumes:
      - ./o2-data:/data

Once running, open http://localhost:5080 and log in. OpenObserve gives you a ready-to-copy Fluent Bit or Vector configuration under the Ingestion → Logs tab. Fluent Bit tails your container stdout/stderr and sends JSON batches directly to OpenObserve's HTTP API.

Pro Tip: OpenObserve automatically parses incoming JSON fields into searchable columns. You can query your logs using standard SQL like SELECT * FROM "default" WHERE level = 'error' AND status = 503.

5. Searching & Parsing Logs in LogQL

In Grafana Loki, you query logs using LogQL. Every query starts with a stream selector inside curly braces {}, followed by optional filters and parsers.

Example 4: Common LogQL Troubleshooting Queries

# 1. Get all logs from the payment-api container
{container="payment-api"}

# 2. Show only stderr (error stream) logs
{container="payment-api", stream="stderr"}

# 3. Parse JSON logs and filter where status is 503
{container="payment-api"} | json | status = 503

# 4. Count error logs per minute (great for alerts!)
sum(count_over_time({container="payment-api"} | json | level="error" [1m]))

Expected parsed log match

2026-11-18 20:51:12  {"level":"error","service":"payment-api","msg":"Database timeout","status":503}
  Extracted fields: level="error"  service="payment-api"  status="503"  stream="stderr"

6. Step-by-Step Execution Breakdown

Here is exactly what happens under the hood from the moment your code prints an error to the moment you see it in your browser.

Step 1: Your application (Python, C#, Node.js) writes a JSON log line to stdout or stderr.
Step 2: Docker intercepts the stream and writes it to a local log file on the host at /var/lib/docker/containers/<id>/<id>-json.log.
Step 3: Promtail (or Fluent Bit) detects the running container via the Docker socket, reads the new line, and attaches metadata labels like container="payment-api" and stream="stderr".
Step 4: The shipper groups multiple log lines into a small compressed batch and sends it over HTTP to Loki or OpenObserve.
Step 5: The logging engine compresses the raw log chunks and saves them to disk or S3 object storage, making them instantly searchable in the UI.
Watch out for High Cardinality in Loki! Never put dynamic values like user_id, request_id, or ip_address as Loki labels. Labels create separate index streams. High cardinality will crash Loki or make queries painfully slow. Keep those values inside the JSON log body and parse them at query time using | json.

7. Hands-On Practice

Beginner Exercise 1: Spin Up and Filter stderr

Run the Docker Compose stack from Section 3. Open Grafana Explore and write a LogQL query that shows only the stderr logs from the payment-api container.

Hint: Use the label selector {container="payment-api", stream="stderr"}.

Intermediate Exercise 2: Parse JSON and Filter by Latency

Modify the demo-app command to log random latency values, then write a LogQL query to find only requests where latency_ms is greater than 40.

Expected result: Grafana returns only the JSON log lines matching | json | latency_ms > 40.

Advanced Exercise 3: Configure Host Log Rotation

Even with centralized logging, Docker's local JSON log files can fill up your server's hard drive. Add a logging block to your demo-app service in docker-compose.yml to limit local log files to 10MB and keep a maximum of 3 files.

Hint: Use the json-file driver with max-size: "10m" and max-file: "3".

8. Interview Questions

Q1. Why should containerized applications log to stdout/stderr instead of writing to local log files?

Answer: Containers are ephemeral (temporary). Writing to local files inside a container couples the app to the filesystem, risks filling up container disk space, and loses logs when the container restarts. Writing to stdout and stderr lets the container runtime and external log shippers handle rotation, collection, and storage uniformly across all services.

Q2. How does Grafana Loki save so much memory and storage compared to Elasticsearch?

Answer: Elasticsearch builds a full-text inverted index for every word in every log line, which requires massive RAM and disk space. Grafana Loki only indexes a small set of metadata labels (like app, env, container) and stores the actual log lines as heavily compressed chunks in cheap object storage like AWS S3.

Q3. What is "high cardinality" in logging, and why is it dangerous in Loki?

Answer: Cardinality refers to the number of unique values a label can have. A label like env="prod" has low cardinality (maybe 3 values). A label like trace_id or user_id has high cardinality (millions of unique values). In Loki, every unique combination of labels creates a new log stream in the index. High cardinality explodes index size and degrades query performance.

Q4. When would you choose OpenObserve over Grafana Loki?

Answer: OpenObserve is a great choice when a team wants a single-binary, all-in-one platform (logs, metrics, and traces) with a built-in UI, prefers standard SQL over learning LogQL, and wants fast full-text search backed by compressed Parquet files on S3 with very low operational overhead.

Conclusion

Consolidating container logs is one of the highest-impact upgrades you can make to any cluster. By printing structured JSON to stdout and stderr and shipping it to Grafana Loki or OpenObserve, you turn stressful 3:00 AM outages into fast, searchable investigations.

Start small with the Docker Compose stack above, practice writing a few LogQL or SQL queries, and always remember to keep your labels clean and your host log rotation enabled.

Official Documentation

Upgradix by Shrash Studio – Learn Technology. Build Your Future.

Comments & Ratings (0)

Log in or create a student account to comment and rate this article.

Be the first to comment on this article.