Update expired security certificates used for mTLS authentication in Spinnaker

Background 

The self-signed certs (certificates) for mutual TLS, used as a means of security and authentication for communication between Spinnaker microservices, are usually short-lived (maybe for three months) and expire after the specified period of time. If the mTLS certs are approaching expiry, these certs need to be updated to extend their expiry time. The cert-manager is a native Kubernetes certificate management controller that helps with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self-signed. It ensures certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry on its own. To know more, about the certs, cert issuers – CA (certificate authorities), how to create them and use them as means of authentication, please refer to the following articles: 

• “How to set up Mutual TLS (mTLS) Authentication for Spinnaker Services”
• “Authentication for Spinnaker Services using x509 client certificate”

However, when there is no external certificate management solution such as cert-manager to rely upon, you will need to update the certs yourself. While updating you can extend their validity to two years or more. But you must first update them in advance without incurring downtime. However, if your certs have already expired, then they are already in an invalid state, and avoiding downtime might not be possible. When the certificate fails the Spinnaker microservices, say Redis, Clouddriver, Gate services fail to talk to each other, and Spinnaker itself is not accessible more. So even the pre-scheduled jobs, like Cron jobs fail to run.

In the high paced world of Continuous Delivery, if Spinnaker microservices fail to communicate with each other due to expired certificates resulting in authentication failure, will result in a service outage for CI/CD pipelines and bring catastrophic losses to the business. Therefore it is best to proceed with replacing the expiring certificates with valid ones in advance. 

Here in this blog, we will show you how to update the expired certs used for mTLS authentication in Spinnaker. 

Here we will create two certificates:

1. Mtlsca-new cert – this is the certificate of CA signed by the self-signed issuer
2. Mtlspkcs12-new cert – this is the certificate for mTLS, signed by the CA issuer.

Steps to extend the validity of expired CA certificate of self-signed issuer:

• Recreate the mTLS certificate issued by CA with a new secret 

Here we create “mtlsca-new” as the new CA (Certificate Authority) authorized certificate signed by a self-signed issuer with a new secret “cacert-new” using the YAML code, as shown below:

kubectl create -f 

apiVersion: cert-manager.io/v1beta1

kind: Certificate

metadata:

  name: mtlsca-new

  namespace: spinnaker

spec:

  commonName: mtlsca

  dnsNames:

  - spin-clouddriver.spinnaker.svc

  - spin-echo.spinnaker.svc

  - spin-front50.spinnaker.svc

  - spin-gate.spinnaker.svc

  - spin-igor.spinnaker.svc

  - spin-orca.spinnaker.svc

  - spin-rosco.spinnaker.svc

  - spin-fiat.spinnaker.svc

  - '*.spinnaker.svc'

  - localhost

  duration: 21600h0m0s

  isCA: true

  issuerRef:

    kind: ClusterIssuer

    name: selfsigned-issuer

  secretName: cacert-new

• Verify the status of the newly created certificate and the secret:

kubectl -n spinnaker get certs

kubectl -n spinnaker get secrets

• Create a new issuer “caissuer-new” and the related secret name “cacert-new” as used above while recreating the new mTLS certificate. Make sure the issuer names and secret are different from the original issuer and it’s secret using the below YAML code as shown below:

kubectl create -f 

apiVersion: cert-manager.io/v1beta1

kind: Issuer

metadata:

  name: caissuer-new

  namespace: spinnaker

spec:

  ca:

    secretName: cacert-new

• Verify the status of the new issuer by running the following command by checking:

kubectl -n spinnaker get issuers

• Create a new “mtlspkcs12” certificate which is a certificate for mTLS, signed by the new CA issuer using the below YAML code as shown below: 

kubectl create -f 

apiVersion: cert-manager.io/v1beta1

kind: Certificate

metadata:

  name: mtlscerts-pkcs12-new

  namespace: spinnaker

spec:

  commonName: spinnaker.svc

  dnsNames:

  - spin-clouddriver.spinnaker.svc

  - spin-echo.spinnaker.svc

  - spin-front50.spinnaker.svc

  - spin-gate.spinnaker.svc

  - spin-igor.spinnaker.svc

  - spin-orca.spinnnaker.svc

  - spin-rosco.spinnaker.svc

  - spin-fiat.spinnaker.svc

  - '*.spinnaker.svc'

  - localhost

  duration: 21600h0m0s

  issuerRef:

    kind: Issuer

    name: caissuer-new

  keystores:

    pkcs12:

      create: true

      passwordSecretRef:

        key: passphrase

        name: passphrasesecret

  renewBefore: 3600h0m0s

  secretName: mtlscerts-pkcs12-new

  usages:

  - digital signature

  - key encipherment

  - server auth

  - client auth

• Update the cacerts using the existing secret, and delete old certs using their aliases:

kubectl -n spinnaker get secret cacerts -o jsonpath='{.data.cacerts}'| base64 -d > cacerts

keytool -delete -alias ca -keystore cacerts

keytool -delete -alias caprod -keystore cacerts

keytool -list -keystore cacerts -v -storepass changeit -noprompt > myfile

1. • Get the ca.crt from the new secret:

kubectl -n spinnaker get secret mtlscerts-pkcs12-new -o jsonpath='{.data.ca\.crt}'| base64 -d > ca.crt

• Verify expiration date again:

openssl x509 -in ca.crt -text -noout

• Import ca.crt to cacerts:

keytool -importcert -keystore cacerts -alias caprod -file ca.crt -storepass changeit -noprompt

• Verify if the new alias exists:

keytool -list -keystore cacerts -v -storepass changeit -noprompt > myfile

vi myfile

Check the alias caprod and the expiry date.

• Create a new secret for cacerts:

kubectl -n spinnaker create secret generic cacerts-new --from-file cacerts

• Copy tls.crt and tls.key to halyard:

kubectl -n spinnaker cp tls.crt spinnaker-spinnaker-halyard-0:/home/spinnaker/.hal/default/tls-new.crt

kubectl -n spinnaker cp tls.key spinnaker-spinnaker-halyard-0:/home/spinnaker/.hal/default/tls-new.key

1. • Change in redis sts (statefulset ):

Replace mtlscerts-pkcs12 with mtlscerts-pkcs12-new

• In Hal config, replace old TLS key and cert names with new TLS key and cert names 

tls.crt and tls.key to tls-new.crt and tls-new.key

• In the cd, service-setting replace cacerts with cacerts-new and secret mtlscerts-pkcs12 with mtlscerts-pkcs12-new in all service setting files.

find . -name '*.yml'| xargs grep cacerts

find . -name '*.yml'| xargs sed  's/cacerts/cacerts-new/g'

find . -name '*.yml'| xargs sed  -i 's/cacerts/cacerts-new/g'

find . -name '*.yml'| xargs grep cacerts-new

find . -name '*.yml'| xargs grep mtlscerts-pkcs12

find . -name '*.yml'| xargs sed  's/mtlscerts-pkcs12/mtlscerts-pkcs12-new/g'

find . -name '*.yml'| xargs sed -i  's/mtlscerts-pkcs12/mtlscerts-pkcs12-new/g'

find . -name '*.yml'| xargs grep mtlscerts-pkcs12-new

hal deploy apply

• Verify if the spinnaker is working fine after the certs update: 

1. Check if Spinnaker is accessible via the browser
2. Check logs of all the spinnaker services

• If the above verification is okay and Spinnaker is working fine

1. Run one sample pipeline in the spinnaker
2. Delete tls.crt and tls.key from default (old cert and key) in halyard pod
3. Delete old secret, certs, and issuers
   1. Secret : mtlscerts-pkcs12 , cacert and cacerts
   2. Certs: mtlscerts-pkcs12 and mtlsca
   3. Issuer: issuer

Finally, check once again to see that everything is working fine once all the changes have been implemented.

Conclusion:

After applying the above configuration changes to your Spinnaker deployment, the Mutual TLS (mTLS) Authentication certs for Spinnaker Services are updated thereby making it secure to communicate securely over the network with other services.

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us at OpsMx Support.