LATEST 🚀 Welcome to Upgradix by Shrash Studio – Your destination for free technology learning and career growth.
🕘 Updated:
Kubernetes

Cluster-Wide Visibility with Prometheus & Grafana

Running applications in Kubernetes without monitoring is like driving a car at night with the headlights off. Everything might feel fine until you hit a wall. When a pod crashes, CPU spikes, or memory runs out, you need to know about it immediately—ideally before your users notice.

Cluster-Wide Visibility with Prometheus & Grafana

Running applications in Kubernetes without monitoring is like driving a car at night with the headlights off. Everything might feel fine until you hit a wall. When a pod crashes, CPU spikes, or memory runs out, you need to know about it immediately—ideally before your users notice.

In this guide, we will set up complete, cluster-wide monitoring using the open-source kube-prometheus-stack Helm chart. In just a few minutes, you will have Prometheus scraping metrics from every corner of your cluster and Grafana displaying clean, pre-built dashboards.

What you will learn
  • Why kube-prometheus-stack is the industry standard for Kubernetes monitoring.
  • How to install Prometheus, Grafana, and Alertmanager in one shot using Helm.
  • How metrics flow from your nodes and pods into live Grafana dashboards.
  • How to access Grafana securely, retrieve your admin password, and explore pre-built dashboards.
  • Production best practices, common mistakes, and real DevOps interview questions.

1. What Is the kube-prometheus-stack?

Setting up Prometheus, Grafana, and all the required Kubernetes exporters manually takes dozens of YAML files and hours of configuration. The kube-prometheus-stack solves this by bundling everything you need into a single, community-maintained Helm chart.

Instead of wiring everything together by hand, this stack automatically discovers your nodes, pods, and Kubernetes API server, starts collecting metrics right away, and loads dozens of ready-to-use Grafana dashboards.

Component What It Does Why You Need It
Prometheus Time-series database and scraper Pulls and stores numeric metrics (CPU, memory, request rates) over time.
Grafana Visualization and dashboard engine Turns raw Prometheus data into readable charts, graphs, and health panels.
Alertmanager Alert routing and deduplication Sends notifications to Slack, Email, or PagerDuty when a threshold is crossed.
Node Exporter Hardware and OS metrics agent Runs on every node to expose disk space, CPU load, and network traffic.
kube-state-metrics Kubernetes object state listener Tracks pod restarts, deployment replicas, and failed jobs.

2. Installing kube-prometheus-stack Using Helm

Before we begin, make sure you have a running Kubernetes cluster (such as Minikube, Kind, EKS, AKS, or GKE), along with kubectl and helm installed on your machine.

Step 1: Add the Prometheus Community Helm Repo

First, register the official Prometheus community repository so Helm knows where to download the chart.

Example 1: Add and update the Helm repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Expected output

"prometheus-community" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈

Step 2: Install the Stack into a Dedicated Namespace

Always keep monitoring tools isolated in their own namespace. The command below creates a monitoring namespace and installs the entire stack under the release name prometheus.

Example 2: Deploy kube-prometheus-stack

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Expected output

NAME: prometheus
LAST DEPLOYED: Wed Nov 18 20:45:12 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace monitoring get pods -l "release=prometheus"

Step 3: Verify All Pods Are Running

Give the cluster about 60 seconds to pull the container images and start all components, then check the pod status.

Example 3: Check running monitoring pods

kubectl get pods -n monitoring

Expected output

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          62s
prometheus-grafana-7c8d9f6b4c-x9k2p                      3/3     Running   0          62s
prometheus-kube-prometheus-operator-6f8b5d7c4-m2n8q      1/1     Running   0          62s
prometheus-kube-state-metrics-5b9c8d7f6-p4r7t            1/1     Running   0          62s
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          62s
prometheus-prometheus-node-exporter-9k4m2                1/1     Running   0          62s

Step 4: Get the Grafana Password and Open the Dashboard

By default, Grafana is only accessible inside the cluster. Let's grab the auto-generated admin password from Kubernetes Secrets and use port-forwarding to open Grafana in your local browser.

Example 4: Retrieve Grafana password and port-forward

# 1. Decode the default admin password (username is: admin)
kubectl get secret --namespace monitoring prometheus-grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

# 2. Forward local port 3000 to the Grafana service
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

Expected output

prom-operator
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Now open your browser and go to http://localhost:3000. Log in with username admin and the password printed in your terminal. Click Dashboards on the left menu to see over 20 pre-built dashboards covering Cluster Compute Resources, Node Health, Networking, and CoreDNS.

Important: On Windows PowerShell, the base64 --decode command will not work directly. Instead, use [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String((kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}"))).

3. Architecture & How Metrics Flow

Understanding how data moves through this stack makes troubleshooting much easier. Prometheus uses a pull model. Rather than waiting for your apps to send data, Prometheus actively reaches out on a schedule (every 30 seconds by default) and scrapes the /metrics endpoint of each target.

Live Kubernetes Observability Pipeline

Node Exporter CPU, RAM, Disk I/O kube-state-metrics Pods, Deployments, Jobs App Pods (/metrics) HTTP Latency, Errors Scrape Prometheus TSDB Storage PromQL Engine PromQL Alerts Grafana Pre-built Dashboards Alertmanager Slack, Email, PagerDuty

Prometheus continuously scrapes metrics from exporters and pods, stores them in its time-series database (TSDB), and serves them to Grafana for visualization and Alertmanager for incident alerts.

4. Step-by-Step Execution Breakdown

When you ran that single helm install command, a lot happened behind the scenes. Here is how the platform wires itself together:

Step 1: Custom Resource Definitions (CRDs) are installed. The Prometheus Operator registers new Kubernetes objects like ServiceMonitor, PodMonitor, and PrometheusRule. This lets you configure monitoring using standard Kubernetes YAML instead of editing raw Prometheus config files.
Step 2: Exporters start collecting host and cluster data. A DaemonSet deploys node-exporter onto every worker node to read Linux kernel stats, while kube-state-metrics connects to the Kubernetes API to track pod phases and replica counts.
Step 3: Prometheus auto-discovers targets. The Prometheus Operator watches for any ServiceMonitor resources in the cluster and automatically updates Prometheus's scrape configuration without restarting the server.
Step 4: Grafana loads dashboards via ConfigMaps. Grafana runs a sidecar container that watches for ConfigMaps labeled with grafana_dashboard: "1". The Helm chart ships with over 20 of these ConfigMaps, so your dashboards appear automatically on first login.
Watch out for ephemeral storage! By default, the Helm chart stores Prometheus metrics inside the pod's temporary filesystem. If the Prometheus pod restarts, you lose all historical metrics. In production, always enable a PersistentVolumeClaim (PVC) in your values.yaml file and set a clear retention period (such as 15 or 30 days).

5. Hands-On Practice

Put your new monitoring stack to the test with these practical exercises.

Beginner Exercise 1: Inspect Cluster CPU Usage

Log into your local Grafana instance, open the Kubernetes / Compute Resources / Cluster dashboard, and identify which namespace is currently consuming the most memory.

Hint: Look for the "Memory Usage by Namespace" table at the bottom of the dashboard.

Intermediate Exercise 2: Run Your First PromQL Query

In Grafana, click Explore on the left sidebar, select Prometheus as the data source, and write a query to count how many pods are currently running in the monitoring namespace.

Expected result: Running count(kube_pod_info{namespace="monitoring"}) should return the exact number of pods you saw in kubectl get pods -n monitoring.

Advanced Exercise 3: Scrape Your Own Application

Deploy a sample microservice that exposes a /metrics endpoint (for example, a .NET Core API using prometheus-net or a Python FastAPI app using prometheus-fastapi-instrumentator) and create a ServiceMonitor YAML so Prometheus automatically scrapes it.

Hint: Make sure your ServiceMonitor includes the label release: prometheus so the Prometheus Operator picks it up.

6. Interview Questions

Observability is a core topic in Cloud, DevOps, and Senior Backend interviews. Here are the questions hiring managers ask most often.

Q1. Why does Prometheus use a "pull" model instead of a "push" model?

Answer: In a pull model, Prometheus controls the scrape interval and immediately knows if a target is down (because the scrape fails). It also prevents a sudden burst of traffic from flooding the monitoring server, since Prometheus only pulls data at its own configured pace.

Q2. What is the difference between a ServiceMonitor and a PodMonitor?

Answer: A ServiceMonitor discovers targets by matching labels on a Kubernetes Service and scraping its backing endpoints. A PodMonitor skips the Service layer and directly matches labels on Pods, which is useful for background workers or batch jobs that do not have a Kubernetes Service.

Q3. What is "High Cardinality" in Prometheus, and why is it dangerous?

Answer: Cardinality is the number of unique label combinations stored in Prometheus. If you add an unbounded label like user_id or request_id to a metric, Prometheus creates a brand-new time series for every single user or request. This rapidly exhausts RAM and can crash your Prometheus server.

Conclusion

Installing kube-prometheus-stack with Helm is the fastest way to get production-grade visibility into your Kubernetes platform. With Prometheus handling metric collection, Alertmanager watching for failures, and Grafana visualizing your cluster's health, you can spot bottlenecks early and scale with confidence.

As your cluster grows, your next steps should be configuring persistent storage for Prometheus, setting up Slack or Email receivers in Alertmanager, and adding custom business metrics to your own applications.

Official Documentation

Upgradix by Shrash Studio – Learn Technology. Build Your Future.

Comments & Ratings (0)

Log in or create a student account to comment and rate this article.

Be the first to comment on this article.