Readiness probe issue with custom image for Spinnaker microservices

Spinnaker has multiple components ( or microservices) like Orca, Igor, Clouddriver, and so on. Each component has a specific role. While deploying or installing Spinnaker in Kubernetes, each component is installed separately. Enterprises often need to use custom Docker container images to specify their project-specific dependencies and incorporate additional functionality, and features that are not part of the open-source Spinnaker. But while using custom images to install Spinnaker, often a common issue faced is that the pods of that installed Spinnaker service fail to execute and run, as the “readiness probe” (Kubernetes health check probe for pods) fails. 

Here in this blog, we will discuss how OpsMx has found a way to resolve this issue and get you going with your Spinnaker deployments.

Issue

When we use a custom image for microservice in Spinnaker, the pods of that service will not come up as the readiness probe fails since the custom image may not have the tools like netstat, wget, or curl.

Here, we have a custom docker container image for the Spinnaker’s Orca microservice for which the readiness probe fails. 

For e.g. : 

Custom image used:

artifactId: docker.io/opsmxdev/orca:1.22.2-pr3989-40

kubectl describe po spin-orca-fff44f5f4-8j6zs

Output:

Readiness probe failed

Steps for resolution

The following steps will help you in resolving the issue as discussed above you can use and implement this resolution very easily. Execute the steps as given below:

1. Manually remove readiness probe from that service’s deployment as below:

kubectl edit deploy spin-orca
Remove below:
readinessProbe:
exec:
command:
- wget
- --no-check-certificate
- --spider
--q
- https://localhost:8083/health 
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1

2. Verify the new pod that comes up with the custom image.

kubectl describe po spin-orca-fff44f5f4-8j6ab
orca:
Container ID:   docker://
05bdc0f4122ff67ef928497cfdebc4ac78de86763fc36ba6855a64b9be0e55f2
Image: docker.io/opsmxdev/orca:1.22.2-pr3989-40

3. Now execute the custom image in the new pod.

kubectl
-n spinnaker exec -it spin-orca-86df845bf6-c7sk6 -- bash

4. Verify the ‘wget’ tool used for the health check probe. 

wget
Output:
wget: missing URL

5. Do the health check manually in the pod.

wget
--no-check-certificate --spider -q --private-key /pkcs12/tls.key --certificate /pkcs12/tls.crt https://localhost:8083/health

6. Change the health check command accordingly in the service.yaml

artifactId:
docker.io/opsmxdev/orca:1.22.2-pr3989-40
scheme: https
kubernetes:
useExecHealthCheck: true
customHealthCheckExecCommands: ["wget", "--no-check-certificate", "--spider", "-q", "--private-key", "/pkcs12/tls.key", "--certificate", "/pkcs12/tls.crt", "https://localhost:8083/health"] 
overrideBaseUrl: https://spin-orca.devmt.isym.io:8083
healthEndpoint:

7. Deploy Halyard

hal deploy apply

Note: Do not use –service-names=orca, as the monitoring daemon. It is causing issues with individual updates.

Conclusion 

OpsMx provides the solution to the readiness probe failure issue with a custom image used for installing Spinnaker microservices. Now, by following the above steps you can easily go ahead and install Spinnaker using custom Docker container images. If you are facing similar issues with your Spinnaker installation, please feel free to contact OpsMx support. 

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us