How to Install kube-prometheus-stack for Kubernetes Cluster Monitoring (2026 Guide)

Running a cluster without metrics is like driving with the dashboard covered in tape. It works fine until it doesn’t. Then you’re guessing. This guide installs kube-prometheus-stack via Helm, the standard way to get Prometheus and Grafana running on Kubernetes. It also covers the part most tutorials skip: which metrics, dashboards, and alerts actually matter once the pods turn green.

By the end, Prometheus will scrape every node and pod. Grafana will show real cluster health. Alertmanager thresholds will be tuned enough that they won’t spam your on-call channel the first night. That last part matters more than people think; a monitoring stack that pages you for nothing gets muted within a week.

What is kube-prometheus-stack?

kube-prometheus-stack is a Helm chart maintained by the Prometheus community. It bundles the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics into one installable unit. Hand-wiring each of those components used to eat a full afternoon. Now you run one helm install and get a working, Kubernetes-aware monitoring pipeline with sane defaults for scrape intervals, dashboards, and alert rules.

The chart also installs the Prometheus Operator’s custom resources: ServiceMonitor, PodMonitor, and PrometheusRule. These let you add scrape targets or alert rules as regular Kubernetes YAML. You manage them the same way you manage deployments, with no more hand-editing a monolithic prometheus.yml and reloading it. That’s the real reason this chart became the default starting point. It turns monitoring config into GitOps-friendly manifests, not a snowflake config file that only Dave knows how to touch.

Before You Begin

Make sure you have:

• A running Kubernetes cluster with kubectl access (managed cloud cluster, k3s, kind, minikube, or bare-metal)
• Cluster admin permissions (the chart creates CRDs and cluster-scoped RBAC resources)
• At least 2 vCPUs and 4 GB RAM of spare capacity across your nodes for Prometheus, Grafana, and Alertmanager pods
• A storage class available for PersistentVolumeClaims if you want metrics to survive pod restarts (recommended for anything beyond a throwaway test cluster)
• Helm 3.x and kubectl installed locally
• A terminal (macOS Terminal, iTerm2) and a modern web browser

Requirement	Details
Kubernetes version	1.27+ (tested through 1.30)
Helm version	3.14.x or later
kubectl version	Matched within one minor version of your cluster
Chart version	kube-prometheus-stack 62.x (2026 release line)
Minimum cluster resources	2 vCPU / 4 GB RAM free (small cluster); scale up for production

> This tutorial assumes you already have a cluster to point at. If you need one first, spin up a local k3s or kind cluster. The install steps below work the same once kubectl get nodes returns results.

Step-by-Step Guide

Step 1: Install kubectl and Helm on macOS

Open Terminal and install both tools with Homebrew.

# Install the Kubernetes CLI
brew install kubectl

# Install the Helm package manager for Kubernetes
brew install helm

Verify both installed correctly:

kubectl version --client
helm version

Expected output looks like:

Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
version.BuildInfo{Version:”v3.15.2″, GitCommit:”…”, GitTreeState:”clean”, GoVersion:”go1.22.4″}

Step 2: Confirm cluster access

Point kubectl at your cluster’s kubeconfig (already set up from your cloud provider, k3s, kind, or Rancher Desktop). Confirm it responds.

kubectl get nodes

NAME STATUS ROLES AGE VERSION
node-01 Ready control-plane 14d v1.30.2
node-02 Ready worker 14d v1.30.2
node-03 Ready worker 14d v1.30.2

If this hangs or returns a connection refused error, fix that before continuing. Helm can’t install anything if kubectl can’t reach the API server.

Step 3: Add the Prometheus community Helm repo

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

“prometheus-community” has been added to your repositories
Hang tight while we grab the latest from your chart repositories…
…Successfully got an update from the “prometheus-community” chart repository
Update Complete. ⎈Happy Helming!⎈

Step 4: Create a values.yaml with sane defaults before installing

Don’t install with defaults and tune later. Set retention, storage, and the Grafana admin password up front. If you go back to fix storage after Prometheus already wrote data to an ephemeral volume, you lose your metrics history and start over. Create a file called values.yaml:

# values.yaml
grafana:
  adminPassword: "YOUR_GRAFANA_ADMIN_PASSWORD"
  persistence:
    enabled: true
    size: 5Gi

prometheus:
  prometheusSpec:
    retention: 15d
    scrapeInterval: 30s
    resources:
      requests:
        cpu: 250m
        memory: 1Gi
      limits:
        memory: 2Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: "" # WARNING: an empty string can disable dynamic provisioning; omit this field to use the cluster default, or set an explicit class like "gp3" or "local-path"
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

An empty storageClassName: "" does not mean “use the cluster’s default storage class.” Depending on your cluster’s configuration, an explicit empty string can actually disable dynamic provisioning instead. If you want the default storage class to be used, omit the storageClassName field entirely (or leave it unset) so Kubernetes can apply defaulting. If you have more than one storage class, set storageClassName explicitly to the one you want (kubectl get storageclass lists what’s available). Skip storageSpec entirely and Prometheus stores data in ephemeral pod storage. Every restart wipes your history, which defeats the whole point of having history.

Step 5: Install the chart

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

--create-namespace creates the monitoring namespace if it doesn’t exist. -f values.yaml applies your overrides on top of the chart defaults.

NAME: monitoring
LAST DEPLOYED: Fri Aug 7 10:14:22 2026
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl –namespace monitoring get pods -l “release=monitoring”

Get Grafana ‘admin’ user password by running:
kubectl –namespace monitoring get secrets monitoring-grafana -o jsonpath=”{.data.admin-password}” | base64 -d ; echo

Step 6: Verify all pods are running

Give it 60–90 seconds for images to pull, then check:

kubectl get pods -n monitoring

NAME READY STATUS RESTARTS AGE
monitoring-kube-prometheus-operator-7d4f8b6c8f-xk2lp 1/1 Running 0 2m
monitoring-kube-state-metrics-6b8f9d5c4-p9nzt 1/1 Running 0 2m
monitoring-grafana-5f7c9b8d6-vqm2j 3/3 Running 0 2m
monitoring-prometheus-node-exporter-4jz6n 1/1 Running 0 2m
monitoring-prometheus-node-exporter-8x2wr 1/1 Running 0 2m
prometheus-monitoring-kube-prometheus-prometheus-0 2/2 Running 0 90s
alertmanager-monitoring-kube-prometheus-alertmanager-0 2/2 Running 0 90s

Seven pods, all Running, zero restarts. That’s what you want to see. If any pod is stuck in Pending, it’s usually a resource or PVC issue. Check with kubectl describe pod POD_NAME -n monitoring.

Step 7: Access Grafana and confirm login

Retrieve the admin password (skip this if you set grafana.adminPassword explicitly in values.yaml):

kubectl get secret monitoring-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode
echo

Port-forward the Grafana service to your local machine:

kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring

Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

Open http://localhost:3000 in your browser. Log in with username admin and the password from above.

macOS

Run the port-forward command above in a Terminal tab and leave it running. It stays in the foreground until you press Ctrl+C. Open Safari or Chrome to http://localhost:3000 in a second tab.

Web

The Grafana UI is identical no matter which OS you’re forwarding from. Everything from here happens in the browser.

Step 8: Verify Prometheus is actually scraping targets

This is the step people skip, and it matters. A green pod doesn’t mean Prometheus is collecting data. Port-forward the Prometheus service in a new terminal tab:

kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring

Open http://localhost:9090/targets in your browser.

Every target should show State: UP with a recent Last Scrape time. Targets stuck in DOWN? Jump to the troubleshooting section below. It’s almost always an RBAC or network policy issue blocking Prometheus from reaching the target.

Step 9: Open the default Grafana dashboards

Back in Grafana, go to Dashboards in the left sidebar. The chart pre-loads several dashboards under folders like Kubernetes / Compute Resources / Cluster and Kubernetes / Networking. Open Kubernetes / Compute Resources / Cluster first. It’s the closest thing to a single pane of glass for overall cluster health.

Then open the Node Exporter / Nodes dashboard for per-node detail: CPU, memory, disk, and network broken out per host.

Step 10: Import additional dashboards beyond the defaults

The bundled dashboards cover the basics. Two community dashboards earn a permanent spot in your bookmarks: Node Exporter Full (dashboard ID 1860) for deeper hardware-level detail, and Kubernetes Cluster (Prometheus) (dashboard ID 7249) for a workload-centric view that’s easier to read than the chart’s default.

In Grafana, go to Dashboards → New → Import. Enter the dashboard ID, click Load, select your Prometheus data source (usually named Prometheus) when prompted, and click Import.

Step 11: Check Alertmanager and configure a real notification receiver

Port-forward Alertmanager:

kubectl port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 -n monitoring

Open http://localhost:9093 to see any currently firing alerts.

Out of the box, Alertmanager has no external receiver configured. Alerts fire but go nowhere. That’s about as useful as a smoke alarm with no battery. Add a Slack (or webhook) receiver by extending your values.yaml:

# values.yaml (append)
alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      receiver: "slack-notifications"
      group_by: ["alertname", "namespace"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    receivers:
      - name: "slack-notifications"
        slack_configs:
          - api_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
            channel: "#cluster-alerts"
            send_resolved: true

Apply the change:

helm upgrade monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Which Metrics Actually Matter

The default dashboards surface dozens of panels. Most of them are useful context, not signal. If you’re triaging cluster health day to day, these four categories are where the real answers live.

Node resource pressure. Watch node_memory_MemAvailable_bytes relative to total, and node_filesystem_avail_bytes for disk. Kubernetes evicts pods under memory pressure well before a node hits 0% free. A node above 85–90% memory use for more than a few minutes is worth alerting on.

Pod restart loops. kube_pod_container_status_restarts_total from kube-state-metrics is the single most useful metric for catching CrashLoopBackOff and OOMKilled pods. It flags problems before someone files a ticket. A pod that restarted once during a deploy is normal. A pod restarting five times in ten minutes is not.

PVC capacity. kubelet_volume_stats_available_bytes versus kubelet_volume_stats_capacity_bytes tells you how full a PersistentVolumeClaim is. This metric prevents a 2 a.m. page from a stateful workload, like databases or log stores, that silently filled its disk. Alert at 80% used with room to act. Don’t wait for 95%; by then it’s already too late.

API server latency. apiserver_request_duration_seconds (via its histogram buckets) tells you if the control plane is struggling. Rising p99 latency on LIST/WATCH requests is often the earliest sign of etcd or control-plane resource exhaustion. It shows up long before pods start failing to schedule.

Everything else, including total request counts, container CPU throttling on non-critical batch jobs, and per-pod network byte counts, is useful for deep debugging. Treat it as reference data, not something worth an alert or a dashboard headline panel.

Configuration

Sizing retention and storage

Retention is a direct trade-off between historical visibility and disk cost. Prometheus‘s on-disk size scales roughly linearly with retention period, scrape interval, and the number of active time series.

Cluster size	Suggested retention	Suggested Prometheus PVC
Homelab / small (1–10 nodes)	7–15d	20–50 GB
Mid-size (10–50 nodes)	15–30d	100–200 GB
Large (50+ nodes)	15d local + long-term via Thanos/Mimir	200 GB+ per Prometheus shard

For anything beyond a homelab, don’t extend local Prometheus retention past 30 days. Pair it with Thanos or Grafana Mimir for long-term storage instead. Local Prometheus disk grows fast, and long retention on a single instance makes queries slower without much benefit.

Scrape interval trade-offs

The chart defaults to a 30-second scrape interval. Dropping to 15s doubles your sample count. It also roughly doubles disk usage and Prometheus CPU load, for marginally better graph resolution. Going up to 60s halves storage but can hide short-lived spikes. A pod that OOMKills and restarts inside a 60-second window might never show up clearly on a memory graph. For most clusters, 30s is the right default. Drop to 15s only for specific high-value ServiceMonitors, like your API gateway, not globally.

Avoiding cardinality explosions

Cardinality is the number of unique label combinations for a metric. A metric with a pod label is fine at typical scale. A metric with a user_id or full url_path label is not, because every unique value creates a new time series. This is the single most common way teams accidentally take down their own Prometheus. It’s usually self-inflicted, not a Prometheus bug.

Check current series count:

kubectl exec -n monitoring prometheus-monitoring-kube-prometheus-prometheus-0 -c prometheus -- \
  wget -qO- http://localhost:9090/api/v1/status/tsdb

Look at seriesCountByMetricName in the response. Anything with an unexpectedly high count is worth investigating. Fix it by dropping the offending label at scrape time with metric_relabel_configs on the relevant ServiceMonitor:

# service-monitor-example.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      metricRelabelings:
        - sourceLabels: [__name__]
          regex: "http_requests_total"
          targetLabel: url_path
          replacement: ""
          action: replace

Prometheus itself exposes prometheus_tsdb_symbol_table_size_bytes and scrape_samples_scraped. Watch these on your own Prometheus instance to catch cardinality growth before it OOMKills the pod monitoring everything else. That’s a genuinely annoying way to lose visibility right when you need it most.

Setting alert thresholds that don’t spam the on-call channel

The chart’s default PrometheusRule set is tuned for general use, not your cluster’s baseline. Two changes fix most alert fatigue: add a for: duration so alerts only fire after a condition persists, and group related alerts in Alertmanager instead of sending one notification per pod.

# prometheus-rule-example.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-memory-pressure
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: node-resources
      rules:
        - alert: NodeMemoryHighUtilization
          expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Node {{ $labels.instance }} memory usage above 90%"
            description: "Memory has been above 90% for 10 minutes. Investigate before pods start getting evicted."

for: 10m means a brief CPU or memory spike during a deploy won’t page anyone. Only sustained pressure will. Combine that with the group_by and group_interval settings from Step 11 to batch related alerts instead of firing one notification per affected pod.

Tips and Troubleshooting

Prometheus pod gets OOMKilled

Why it happens: Default resource limits are sized for small clusters. On larger clusters, or with high-cardinality metrics, Prometheus‘s memory footprint outgrows the chart defaults.

Fix: Raise prometheus.prometheusSpec.resources.limits.memory in values.yaml (start at 2Gi, go to 4Gi+ for larger clusters) and run helm upgrade. Watch prometheus_tsdb_head_series to correlate memory growth with series count.

All monitoring data disappears after a pod restart

Why it happens: No PersistentVolumeClaim was configured, so Prometheus and Grafana were writing to ephemeral pod storage. That storage gets wiped on restart or rescheduling.

Fix: Set prometheus.prometheusSpec.storageSpec and grafana.persistence.enabled: true as shown in Step 4, then helm upgrade. Confirm the PVC bound correctly with kubectl get pvc -n monitoring.

Prometheus targets show DOWN on the /targets page

Why it happens: Usually an RBAC issue (the Prometheus Operator’s ClusterRole doesn’t have permission to scrape a namespace) or a NetworkPolicy blocking traffic from the monitoring namespace to the target pod’s metrics port.

Fix: Check kubectl describe servicemonitor SERVICE_MONITOR_NAME -n monitoring for selector mismatches. Confirm any NetworkPolicy in the target namespace allows ingress from the monitoring namespace on the metrics port.

New application metrics don’t show up in Prometheus

Why it happens: No ServiceMonitor exists for the app, or its labels don’t match what prometheus.prometheusSpec.serviceMonitorSelector expects.

Fix: Create a ServiceMonitor like the example in the cardinality section above. Make sure it carries the release: monitoring label (or whatever your Helm release name is). The chart’s default selector filters on that label.

Can’t log into Grafana

Why it happens: The admin password was auto-generated at install and never retrieved, or grafana.adminPassword wasn’t set explicitly.

Fix: Pull the current password from the secret:

kubectl get secret monitoring-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode
echo

Or set grafana.adminPassword explicitly in values.yaml and run helm upgrade to reset it.

Wrapping Up

You now have a full Kubernetes monitoring pipeline. Prometheus scrapes every node and pod, Grafana shows dashboards for cluster and node health, and Alertmanager is wired to a real notification channel with thresholds that won’t page you for a two-minute CPU blip. That’s the point of using the Helm chart instead of wiring components by hand, so you spend your time tuning thresholds and dashboards, not writing scrape configs from scratch.

The defaults are a solid starting point, not a finished product. Budget an afternoon after the initial install to right-size retention, add PVC and API-latency alerts, and trim the rules you don’t need. That tuning pass separates a dashboard nobody looks at from a monitoring stack that actually catches problems before your users do.

Step	Action	Applies To
1–3	Install kubectl, Helm, verify cluster access	macOS
4–6	Configure values.yaml, install chart, verify pods	macOS
7–8	Access Grafana, verify Prometheus targets	Web
9–10	Review default dashboards, import Node Exporter Full and Kubernetes Cluster dashboards	Web
11	Configure Alertmanager receiver and thresholds	macOS + Web

Resources

• kube-prometheus-stack Helm chart source
• kube-prometheus-stack on Artifact Hub
• Prometheus official documentation
• Grafana official documentation
• Prometheus metric and label naming best practices