How to Enable Prometheus Monitoring for Kubernetes Cluster

Prometheus, an open-source monitoring, and alerting toolkit is becoming the most popular solution for monitoring Kubernetes cluster for Spinnaker users or otherwise.   To perform manual or automated Red/Black or Canary analysis, having a useful monitoring solution is essential.

In this blog,  we will provide instructions on how to enable Prometheus monitoring for your Kubernetes cluster

Background:

Prometheus is by default a “pull-based” metrics system.  The monitoring node needs to expose the metrics on “/metrics” endpoint (http://IP:9090/metrics), and the Prometheus server accesses the node’s “/metrics” endpoint on regular intervals (jobs) to collect the metrics. Node’s endpoint URLs should be configured on the Prometheus server for the server to access the nodes.

Enabling Prometheus Exporter on the Worker Nodes

To enable Prometheus exporter, launch cAdvisor container on every worker node in the cluster. For info information on cAdvisor, check out  https://github.com/google/cadvisor

sudo docker run \

  --volume=/:/rootfs:ro \

  --volume=/var/run:/var/run:rw \

  --volume=/sys:/sys:ro \

  --volume=/var/lib/docker/:/var/lib/docker:ro \

  --volume=/dev/disk/:/dev/disk:ro \

  --publish=9090:8080 \

  --detach=true \

  --name=cadvisor \

  --restart always \

  google/cadvisor:latest

The above command will launch cAdvisor container in the node and exposes metrics on http:<NODEIP>:9090/metrics.  Run the above command on all worker nodes.

Next step is to configure the node IPs in the Prometheus server to allow it to pull the POD metrics off the nodes.

Enabling Prometheus Server

Step 1: Download Prometheus server from https://prometheus.io/download/

Step 2: Extract the .tar file and configure “prometheus.yml”, refer and add below configuration with the node IPs (where the cAdvisor containers were enabled in the previous section) under “scrape_configs:” section.

scrape_configs: 

   - job_name: 'K8sCluster'

     honor_labels: true

     static_configs:

       - targets: ['NODE1 IP:9090']

       - targets: ['NODE2 IP:9090']

       - targets: ['NODE3 IP:9090']

Step 3: Start Prometheus server

nohup ./prometheus config.file=prometheus.yml &

Step 4:  Check status of exporters(Monitoring nodes) in “9090/target” endpoint

Prometheus Targets

Step 5:  Check relevant metrics data

Prometheus Metric Graph

That’s it, you are done.  For more advanced configurations, check out the complete documentation at https://prometheus.io/docs/introduction/overview/

You can connect this Prometheus server to your Automated Canary Analysis system to able to perform automated release analysis.  If you need further assistance, email info@opsmx.com