Basic Authentication for Prometheus and Alertmanager on a Spinnaker CD pipeline

After deploying applications on the Kubernetes platform using Spinnaker we need to use APM (application performance monitoring) tools in a secured way using Authentication. APM tools are used to monitor and track the overall health, performance, and behavior of the deployed app and the environment where it runs. For applications and microservices running in Kubernetes, Prometheus is a popular APM tool that provides detailed actionable metrics for the DevOps teams on the performance of containers, pods and services, and underlying Kubernetes cluster infrastructure. Prometheus server can be configured to trigger alerts to the Prometheus Alertmanager service instance, which then notifies the end-users through Email, Slack, or other communication channels. 

Although Prometheus and Alertmanager are effective monitoring tools to track threshold breaches, yet they do not support any type of authentication mechanism for connections to browser and HTTP API calls. This means anybody with the URL can access the monitoring information. However,  enterprises need to have authentication to prevent unauthorized access to the monitoring information about their application deployments.

So if you are already using Prometheus or are considering one, then you can apply an authentication mechanism at the proxy layer, using any reverse proxy techniques. For a reverse proxy to enforce basic authentication, you can use NGINX Ingress Controller , which is a traffic management solution for cloud‑native apps in Kubernetes containerized environments. 

This blog explains how to implement the Basic Authentication for both Prometheus and Alertmanager running on the Kubernetes using NGINX as the reverse proxy Ingress Controller.

Pre Requisites:

This blog assumes that you have the Prometheus and Alertmanager installed on your Kubernetes cluster and both of them are accessible in the web browser without any authentication mechanism. If not installed, please refer to Prometheus Installation using the steps mentioned here. 

1. Creating Basic Authentication credentials.
2. Creating Kubernetes Secret from the Basic Auth File.
3. Creating/ Updating the Ingress rule with the Basic Authentication file.
4. Testing your Ingress.

Create Basic Authentication credentials for securely accessing APM tools used in a Spinnaker CD pipeline:

Basic Authentication credentials can be created using the htpasswd tool. The htpasswd tool creates and updates the flat-files used to store usernames and passwords for basic authentication of the HTTP users. Given below is the example to create the basic auth file:

$ htpasswd -c auth-details <user>

New password: <passwd>

Retype new password: 

Adding password for user <user>

After you enter a username and password, a new auth file will be created with the name “auth-details”.

Creating Kubernetes Secret from the Basic Auth File:

Once you have the basic auth file created, now Kubernetes Secret can be created using the below command,

$ kubectl create secret generic basic-auth --from-file=auth-details

As the Secret gets created successfully, you can see the Secret using the below command: 

$ kubectl get secret basic-auth -o yaml

The sample output is shown as:

apiVersion: v1
data:
  auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK
kind: Secret
metadata:
  name: basic-auth
  namespace: default
type: Opaque

Creating/ Updating the Ingress rule with the Basic Authentication file:

To enable the Basic Authentication at the Kubernetes Ingress layer we need to add the below annotations to the Kubernetes Nginx Ingress resource: 

1. nginx.ingress.kubernetes.io/auth-type: basic
(This indicates the type of Authentication.)

2. nginx.ingress.kubernetes.io/auth-secret: basic-auth
 (Name of the secret that contains the user/password definitions.)
3. nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - <user>'
(Message to display with an appropriate context of why the authentication is required.)

Now set the configuration of the following Ingress config files enabling Basic Authentication:  

Prometheus-ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
  creationTimestamp: "2020-11-23T13:32:25Z"
  generation: 1
  name: prometheus-ingress
  namespace: oes
  resourceVersion: "33407088"
  selfLink: /apis/extensions/v1beta1/namespaces/oes/ingresses/prometheus-ingress
  uid: faeb0983-382a-4172-8a30-a0cd9dfc5b13
spec:
  rules:
  - host: demoprom.opsmx.com
    http:
      paths:
      - backend:
          serviceName: prom-prometheus-server
          servicePort: 80
        path: /
  tls:
  - hosts:
    - demoprom.opsmx.com
    secretName: promtls
status:
  loadBalancer:
    ingress:
    - ip: 10.240.0.6

Alertmanager-ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-alertmanager
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-realm: Authentication Required - admin
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-type: basic
  creationTimestamp: "2020-11-24T04:32:56Z"
  generation: 1
  name: alertmanager-ingress
  namespace: oes
  resourceVersion: "33561571"
  selfLink: /apis/extensions/v1beta1/namespaces/oes/ingresses/alertmanager-ingress
  uid: 4e1150dc-34cf-4854-8b7c-e85a31737283
spec:
  rules:
  - host: demoalerts.opsmx.com
    http:
      paths:
      - backend:
          serviceName: prom-prometheus-alertmanager
          servicePort: 80
        path: /
  tls:
  - hosts:
    - demoalerts.opsmx.com
    secretName: alerttls
status:
  loadBalancer:
    ingress:
    - ip: 10.240.0.6

Testing your Ingress:

Use cURL to interact with your NGINX/ Prometheus setup. Try this request:

$ curl https://demoprom.opsmx.com

If you try to reach the Prometheus it will show the following 401 authentication error:

<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.17.8</center>
</body>
</html>

For successful authentication, you need to pass the username with the -u flag and enter the password when prompted. 

$ curl https://demoprom.opsmx.com -u admin

Enter host password for user ‘admin’: 

After you enter the password, the following output is shown signaling that the authentication has been successful.

<a href=”/graph”>Found</a>

Summary

In this blog we have shown you how to store a username and password in an auth file and create Kubernetes secret from it, configured the NGINX Ingress layer for reverse proxying and use the auth file credentials to authenticate users accessing Prometheus and Alertmanager HTTP endpoints. Now you should be able to enhance security in monitoring your new releases.

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us