Securely deploy Kubernetes manifests using Spinnaker pipelines and Vault

Our applications have secrets. These are credentials to remote systems like a database, or tokens to infrastructure like Kubernetes systems or keys used within the application for CI/CD process. And the question always is, how can we secure this data during the process of deployment using Spinnaker? How do we manage different secrets for different environments? How can we securely manage this data? One of the solutions to this is provided by Hashicorp’s Vault. 

HashiCorp’s Vault is a secrets management tool specifically designed to control access to sensitive credentials. It can be used to store sensitive values, and at the same time, dynamically generate access for specific services/applications on lease.

Here we walk through how secrets managed in Vault can be injected into Kubernetes workloads. This works as a sidecar in Kubernetes that pulls in secret and makes it available to the pods. There are three high-level tasks to be performed to achieve this:

1. Setup authentication to Kubernetes cluster in Vault
2. Define policy in Vault to grant read-access to secrets to Kubernetes service accounts
3. Add annotations to Kubernetes deployments to make the secrets available to the pods

Set up Vault

For our demo, we will setup a developer instance of Vault. If you already have a Vault setup and ready to use, you can skip this step. This is setup as a developer instance and hence does not require to unseal the vault before usage. 

helm repo add vault https://helm.releases.hashicorp.com

helm install vault vault/vault --set "server.dev.enabled=true"

Create some secrets in Vault 

In the Vault pod, run the following commands to create a secret that we will use in our pod.

vault secrets enable -path=internal kv-v2

vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"

This creates a key-value secret of username and password in Vault under the path internal/database/config.

Setup authentication to Kubernetes cluster 

In this setup, we are assuming that Vault is running in the same Kubernetes cluster as the application that will use the secrets stored in Vault. If they are in different Kubernetes clusters, you  need the following information to connect the Kubernetes cluster to Vault:

1. Kubernetes cluster API host
2. Kubernetes cluster CA certificate
3. Token to authenticate with the Kubernetes Cluster

Assuming that Vault is running in the same Kubernetes cluster, in the Vault pod, run:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Define policy in Vault to grant access to secrets

From the Vault pod, create a policy to allow access to secrets stored in Vault and then grant access to the service account and namespaces in Kubernetes to that policy.

vault policy write internal-app - <<EOF
path "internal/data/database/config"
{  capabilities = ["read"]
}
EOF
vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=* \
    policies=internal-app \
    ttl=24h

The important thing to note in the above policy grant is the service_account and the namespace granted access. Both cannot be wild cards. In this case, any pod running as a service account internal-app will be able to extract the secrets into the pod. 

And the role, internal-app that you created, will be used in annotations in your pod. 

Use Vault secrets in your Kubernetes pods

First, you need to create a service account that will be running the pod:

apiVersion: v1
kind: ServiceAccount
metadata:
name: internal-app

Make sure to set the service in the pod manifest:

spec: 
  serviceAccountName: internal-app

Now, for the pod to be able to access the secret, you need to assign the following annotations to the pod:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

 Note, the value of vault.hashicorp.com/role should be the value of the role created in the previous step. And the pod must run in the context of the service account specified during the role creation. 

And the last annotation line will read the secret in the path internal/data/database/config and make it available in the container in the file /vault/secrets/database-config.txt in the pod! Your application can now read the file and use the content! If the content gets updated in Vault, the file will also be updated! It is the responsibility of the application to re-read the content!

Using annotations, you can have the content formatted and written to file or even available as environment variables:

• Format the content

vault.hashicorp.com/agent-inject-template-database-config.txt: |
  {{- with secret "internal/data/database/config" -}}
  postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
  {{- end -}}

• Set environment variables

vault.hashicorp.com/agent-inject-template-config: |
  {{- with secret "sinternal/data/database/config" -}}
  export db_passwd="{{ .Data.data.password }}"
  {{- end -}}

This would create a file /vault/secrets/config in your pod. So, you need to source that file before starting your application executable to make the db_passwd available as an environment variable to your application.

Role of Spinnaker 

The injection of secrets into a pod to be used by an application is the responsibility of the application manager. So, what role can Spinnaker play in this process? 

Spinnaker can help you deploy the same application manifest in different environments but using various Vault secrets. You can have Spinnaker set the annotations in your manifest depending on the environment your application is being deployed to. You can also use Spinnaker to control the service account in which the pod is running or the namespace where the pod gets deployed. So, the application developer does not need to worry about the actual secrets being used during the deployment of the application.

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us

---

References:

https://www.vaultproject.io/docs/platform/k8s

https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar?in=vault/kubernetes

https://www.vaultproject.io/docs/platform/k8s/injector/examples