Cluster-Wide Visibility with Prometheus & Grafana
Running applications in Kubernetes without monitoring is like driving a car at night with the headlights off. Everything might feel fine until you hit a wall. When a pod crashes, CPU spikes, or memory runs out, you need to know about it immediately—ideally before your users notice.
In this guide, we will set up complete, cluster-wide monitoring using the open-source kube-prometheus-stack Helm chart. In just a few minutes, you will have Prometheus scraping metrics from every corner of your cluster and Grafana displaying clean, pre-built dashboards.
- Why
kube-prometheus-stackis the industry standard for Kubernetes monitoring. - How to install Prometheus, Grafana, and Alertmanager in one shot using Helm.
- How metrics flow from your nodes and pods into live Grafana dashboards.
- How to access Grafana securely, retrieve your admin password, and explore pre-built dashboards.
- Production best practices, common mistakes, and real DevOps interview questions.
1. What Is the kube-prometheus-stack?
Setting up Prometheus, Grafana, and all the required Kubernetes exporters manually takes dozens of YAML files and hours of configuration. The kube-prometheus-stack solves this by bundling everything you need into a single, community-maintained Helm chart.
Instead of wiring everything together by hand, this stack automatically discovers your nodes, pods, and Kubernetes API server, starts collecting metrics right away, and loads dozens of ready-to-use Grafana dashboards.
| Component | What It Does | Why You Need It |
|---|---|---|
| Prometheus | Time-series database and scraper | Pulls and stores numeric metrics (CPU, memory, request rates) over time. |
| Grafana | Visualization and dashboard engine | Turns raw Prometheus data into readable charts, graphs, and health panels. |
| Alertmanager | Alert routing and deduplication | Sends notifications to Slack, Email, or PagerDuty when a threshold is crossed. |
| Node Exporter | Hardware and OS metrics agent | Runs on every node to expose disk space, CPU load, and network traffic. |
| kube-state-metrics | Kubernetes object state listener | Tracks pod restarts, deployment replicas, and failed jobs. |
2. Installing kube-prometheus-stack Using Helm
Before we begin, make sure you have a running Kubernetes cluster (such as Minikube, Kind, EKS, AKS, or GKE), along with kubectl and helm installed on your machine.
Step 1: Add the Prometheus Community Helm Repo
First, register the official Prometheus community repository so Helm knows where to download the chart.
Example 1: Add and update the Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
Expected output
"prometheus-community" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "prometheus-community" chart repository Update Complete. ⎈Happy Helming!⎈
Step 2: Install the Stack into a Dedicated Namespace
Always keep monitoring tools isolated in their own namespace. The command below creates a monitoring namespace and installs the entire stack under the release name prometheus.
Example 2: Deploy kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace
Expected output
NAME: prometheus LAST DEPLOYED: Wed Nov 18 20:45:12 2026 NAMESPACE: monitoring STATUS: deployed REVISION: 1 NOTES: kube-prometheus-stack has been installed. Check its status by running: kubectl --namespace monitoring get pods -l "release=prometheus"
Step 3: Verify All Pods Are Running
Give the cluster about 60 seconds to pull the container images and start all components, then check the pod status.
Example 3: Check running monitoring pods
kubectl get pods -n monitoring
Expected output
NAME READY STATUS RESTARTS AGE alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 62s prometheus-grafana-7c8d9f6b4c-x9k2p 3/3 Running 0 62s prometheus-kube-prometheus-operator-6f8b5d7c4-m2n8q 1/1 Running 0 62s prometheus-kube-state-metrics-5b9c8d7f6-p4r7t 1/1 Running 0 62s prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 62s prometheus-prometheus-node-exporter-9k4m2 1/1 Running 0 62s
Step 4: Get the Grafana Password and Open the Dashboard
By default, Grafana is only accessible inside the cluster. Let's grab the auto-generated admin password from Kubernetes Secrets and use port-forwarding to open Grafana in your local browser.
Example 4: Retrieve Grafana password and port-forward
# 1. Decode the default admin password (username is: admin)
kubectl get secret --namespace monitoring prometheus-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
# 2. Forward local port 3000 to the Grafana service
kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
Expected output
prom-operator Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000
Now open your browser and go to http://localhost:3000. Log in with username admin and the password printed in your terminal. Click Dashboards on the left menu to see over 20 pre-built dashboards covering Cluster Compute Resources, Node Health, Networking, and CoreDNS.
base64 --decode command will not work directly. Instead, use [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String((kubectl get secret -n monitoring prometheus-grafana -o jsonpath="{.data.admin-password}"))).3. Architecture & How Metrics Flow
Understanding how data moves through this stack makes troubleshooting much easier. Prometheus uses a pull model. Rather than waiting for your apps to send data, Prometheus actively reaches out on a schedule (every 30 seconds by default) and scrapes the /metrics endpoint of each target.
Live Kubernetes Observability Pipeline
Prometheus continuously scrapes metrics from exporters and pods, stores them in its time-series database (TSDB), and serves them to Grafana for visualization and Alertmanager for incident alerts.
4. Step-by-Step Execution Breakdown
When you ran that single helm install command, a lot happened behind the scenes. Here is how the platform wires itself together:
ServiceMonitor, PodMonitor, and PrometheusRule. This lets you configure monitoring using standard Kubernetes YAML instead of editing raw Prometheus config files.node-exporter onto every worker node to read Linux kernel stats, while kube-state-metrics connects to the Kubernetes API to track pod phases and replica counts.ServiceMonitor resources in the cluster and automatically updates Prometheus's scrape configuration without restarting the server.grafana_dashboard: "1". The Helm chart ships with over 20 of these ConfigMaps, so your dashboards appear automatically on first login.PersistentVolumeClaim (PVC) in your values.yaml file and set a clear retention period (such as 15 or 30 days).5. Hands-On Practice
Put your new monitoring stack to the test with these practical exercises.
Log into your local Grafana instance, open the Kubernetes / Compute Resources / Cluster dashboard, and identify which namespace is currently consuming the most memory.
Hint: Look for the "Memory Usage by Namespace" table at the bottom of the dashboard.
In Grafana, click Explore on the left sidebar, select Prometheus as the data source, and write a query to count how many pods are currently running in the monitoring namespace.
Expected result: Running count(kube_pod_info{namespace="monitoring"}) should return the exact number of pods you saw in kubectl get pods -n monitoring.
Deploy a sample microservice that exposes a /metrics endpoint (for example, a .NET Core API using prometheus-net or a Python FastAPI app using prometheus-fastapi-instrumentator) and create a ServiceMonitor YAML so Prometheus automatically scrapes it.
Hint: Make sure your ServiceMonitor includes the label release: prometheus so the Prometheus Operator picks it up.
6. Interview Questions
Observability is a core topic in Cloud, DevOps, and Senior Backend interviews. Here are the questions hiring managers ask most often.
Answer: In a pull model, Prometheus controls the scrape interval and immediately knows if a target is down (because the scrape fails). It also prevents a sudden burst of traffic from flooding the monitoring server, since Prometheus only pulls data at its own configured pace.
Answer: A ServiceMonitor discovers targets by matching labels on a Kubernetes Service and scraping its backing endpoints. A PodMonitor skips the Service layer and directly matches labels on Pods, which is useful for background workers or batch jobs that do not have a Kubernetes Service.
Answer: Cardinality is the number of unique label combinations stored in Prometheus. If you add an unbounded label like user_id or request_id to a metric, Prometheus creates a brand-new time series for every single user or request. This rapidly exhausts RAM and can crash your Prometheus server.
Conclusion
Installing kube-prometheus-stack with Helm is the fastest way to get production-grade visibility into your Kubernetes platform. With Prometheus handling metric collection, Alertmanager watching for failures, and Grafana visualizing your cluster's health, you can spot bottlenecks early and scale with confidence.
As your cluster grows, your next steps should be configuring persistent storage for Prometheus, setting up Slack or Email receivers in Alertmanager, and adding custom business metrics to your own applications.
Official Documentation
- Prometheus Community – kube-prometheus-stack Helm Chart
- Prometheus Documentation – Overview & PromQL
- Grafana Documentation – Dashboards & Data Sources
Upgradix by Shrash Studio – Learn Technology. Build Your Future.