Observability in Kubernetes: Installing, Configuring, and Using Prometheus & Grafana

Introduction

Observability is a key aspect of modern infrastructure. Prometheus and Grafana are widely used for monitoring and visualization in Kubernetes environments. This guide walks you through installing, configuring, and using Prometheus and Grafana for observability.

Prerequisites

Before proceeding, ensure you have:

• A running Kubernetes cluster (EKS, AKS, GKE, or Minikube)
• kubectl configured for your cluster
• Helm installed

Step 1: Install Prometheus & Grafana using Helm

Add the Helm repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install Prometheus

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

This installs Prometheus, Alertmanager, and required exporters.

Install Grafana

Grafana is included in kube-prometheus-stack, but if needed separately:

helm install grafana grafana/grafana --namespace monitoring

Step 2: Access Prometheus & Grafana

Get Prometheus service URL

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090

Access Prometheus at: http://localhost:9090

Get Grafana credentials & URL

kubectl get secret --namespace monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80

Access Grafana at: http://localhost:3000 (Default username: admin, password: from above command)

Step 3: Configure Dashboards

1. Log into Grafana
2. Add Prometheus as a Data Source

• Navigate to Configuration → Data Sources
• Select Prometheus
• Set URL as http://prometheus-kube-prometheus-prometheus.monitoring:9090
• Click Save & Test

3. Import prebuilt dashboards from Grafana Dashboard Repository

Step 4: Setup Alerting with Alertmanager

Prometheus uses Alertmanager for notifications.

Create Alerting Rules

Example: Alert if CPU usage exceeds 80%

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: high-cpu-usage
  namespace: monitoring
spec:
  groups:
    - name: CPUAlerts
      rules:
        - alert: HighCPUUsage
          expr: avg(rate(container_cpu_usage_seconds_total[5m])) * 100 > 80
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "High CPU Usage Detected"

kubectl apply -f high-cpu-alert.yaml

Configure Alertmanager for Slack

Modify values.yaml and set Slack webhook:

alertmanager:
  config:
    receivers:
      - name: "slack"
        slack_configs:
          - send_resolved: true
            channel: "#alerts"
            api_url: "https://hooks.slack.com/services/XXXX/XXXX/XXXX"

Apply the changes:

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f values.yaml

Advanced Commands

List all running Prometheus pods

kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus

Query metrics in Prometheus

kubectl exec -n monitoring -it prometheus-kube-prometheus-prometheus-0 -- sh -c 'promql "up"'

Restart Prometheus

kubectl rollout restart deployment prometheus-kube-prometheus-prometheus -n monitoring

Restart Grafana

kubectl rollout restart deployment prometheus-grafana -n monitoring

FAQs

1. What is Prometheus used for?

Prometheus is a monitoring system that collects and stores metrics data from applications, enabling real-time analysis and alerting.

2. What is Grafana used for?

Grafana is an open-source visualization tool that allows users to create interactive dashboards for monitoring data from Prometheus and other sources.

3. How do I customize Prometheus alerts?

Modify the PrometheusRule configurations and apply them using kubectl apply -f <file>.yaml.

4. How do I scale Prometheus?

Increase replica count in Helm values or deploy Thanos for high availability.

5. How do I persist data in Prometheus?

Use persistent storage:

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

Conclusion

Prometheus and Grafana provide powerful observability in Kubernetes environments. By following this guide, you have successfully installed, configured, and started monitoring your infrastructure. 🚀

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!

Originally published on Medium.