How to Set Up Automated Release Analysis in Spinnaker Deployments

Updated 04/12/2018

Spinnaker, a continuous delivery platform is becoming popular among software deployment teams with its superior ability to orchestrate software delivery faster, safer and across multiple clouds.  Spinnaker currently supports Red/Black for most cloud providers and Canary deploy options for the AWS cloud provider.  Canary deploy can also be set up for any provider through a multi-stage setup. Rolling Red/Black is currently experimental and will be available in a non-experimental mode in a future Spinnaker release.

Starting with 1.7 version,  Spinnaker has officially released the automated canary analysis feature (aka Kayenta).  Spinnaker offers a default judge called the NetflixACAJudge-v1.0 and offers modularity to plugin 3rd party or custom judges. It is critical to enable automated canary analysis for the pipelines to truly benefit from these safe deployment strategies.

In this blog, we will review how to set up automated analysis for Canary and Red/Black deployment strategies instead of manual judgment or verification.

Safe Deployment Options with Spinnaker

Figure 1 above highlights the differences between these three deployment strategies.

• Canary:   Spinnaker deploys a cluster pair consisting of the current baseline/production release and the new release (Canary).   A small % of production traffic is directed to this cluster pair allowing real-time analysis of the versions.
• Red/Black:  Spinnaker deploys an equal number of instances of the new version as the production version, and the traffic is directed 100% using the load balancer to these new server groups away from the old or the baseline/production server group. The old version instances will sit idle until it is destroyed if the new version is acceptable upon analysis.
• Rolling Red/Black:  This is a variation of Red/Black and Canary where the new version is deployed alongside the baseline/production server groups.   The analysis is continuously performed as the new version is rolled forward and the older version instances are decommissioned in parallel.  Eventually, 100% of the instances run the new version, if the analysis finds no issues with the new version.

For releases 1.7 and above, please follow the below instructions:

Step 1: Enable Canary Feature

To enable the automated canary analysis feature, use the following hal command

hal config canary enable

hal config canary aws enable

hal config canary edit --default-judge NetflixACAJudge-v1.0

hal config canary prometheus enable

hal config canary prometheus account add ACCOUNT --base-url

Currently supported metric store include catalog, Prometheus and Google Stackdriver.

If you are using, OpsMx-Judge, it supports additional providers of logs (Elastic Search, Splunk, Sumo Logic)  and APM (App Dynamics, New Relic) as an additional data sources for comprehensive analysis.  If interested, e-mail info@opsmx.com for a free trial.

Please refer to the official Spinnaker Canary documentation for more configuration options.

Pre 1.6, follow these instructions

You have to enable Canary feature and configure the ACA judge before you can use the Canary or ACA stage in Spinnaker.

Follow the steps 1-3  below depending on your Spinnaker deployment model

• Spinnaker Deployment based  on Halyard
• Spinnaker Deployment using  .deb Files

How to Enable Canary/ACA  for Halyard based Spinnaker installation

Step 1:  Enable Canary Feature

To enable Canary feature starting Spinnaker 1.5 version, use the following hal command

ubuntu$ hal config features edit --mine-canary=true

To enable the canary feature before 1.5, you need to edit the profile in Halyard

HAL configuration files for default profile are placed in the folder – $SPINNAKER_HOME/.hal/default/config/profiles

Note:  If HAL is using a different profile then the changes must be made in that profile.

ubuntu$ cd $SPINNAKER_HOME/.hal/default/config/profiles

ubuntu$  cp ../staging/settings.js .

Edit the  settings.js file,  to add the canary feature like below.

  feature: {

    canary: true,

    entityTags: entityTagsEnabled,

…

}

Step 2:  Configure provider of automated Canary analysis service

To enable automated Canary analysis, you need to configure a provider of the Canary analysis.

Create  orca-local.yml in the $SPINNAKER_HOME/.hal/default/config/profiles

Add the line below and save the file.

mine: 

      baseUrl: https://<ACA service end-point  address>

Example:

mine: 

    baseUrl: https://analyzer.opsmx.com

In the above example,  OpsMx is serving as the provider of automated Canary analysis service. OpsMx performs in-depth analysis of new releases and provides the score and a detailed Canary report for diagnostics.

Step 3:  Restart Spinnaker

Restart spinnaker using

sudo hal deploy apply

How to Enable Canary/ACA for .Deb Filed based Spinnaker installation

Step 1: Enable Canary Feature

To enable the canary feature, you need to edit /opt/deck/html/settings.js

Add the canary feature like below.

  feature: {

    canary: true,

    entityTags: entityTagsEnabled,

…

}

Step 2:  Configure provider of automated Canary analysis service

To enable automated Canary analysis, you need to configure a provider of the Canary analysis.

Create  orca-local.yml in the /opt/spinnaker/config directory.

Add the line below and save the file.

mine: 

      baseUrl: https://<ACA service end-point  address>

Example:

mine: 

    baseUrl: https://analyzer.opsmx.com

In the above example,  OpsMx is serving as the provider of automated Canary analysis service. OpsMx performs in-depth analysis of new releases and provides the score and a detailed Canary report for diagnostics.

Step 3: 

Restart spinnaker using

sudo stop spinnaker

sudo start spinnaker

If you are having difficulty,  please reach out to Spinnaker slack channel or info@opsmx.com for help.

At this point, you have enabled Canary stage in Spinnaker and configured the judge.   The next step is to enable monitoring data access

Step 4:  Enable monitoring data access to the automated judge service provider.  The monitoring data source could be cloud providers monitoring sources such as GCP Stackdriver or AWS Cloudwatch.  The monitoring source could also include APM tools like Datadog, New Relic, AppDynamics, etc. or log management tools like Splunk, SumoLogic, etc.

Add Canary Stage with Automated Analysis to Spinnaker

Now, you can configure Spinnaker to deploy the new version in the Canary cluster pair.    Add and configure the Canary stage in the pipeline before production deploy stage.

Note: The Canary stage is only supported for AWS cloud provider (current as of Spinnaker 1.5 release).  The ACA task stage can be configured for other cloud providers  (in addition to AWS) starting in Spinnaker 1.5 release.

Add Canary Stage in Spinnaker Pipeline

Now, you can run the pipeline to get the automated Canary score.

Automated Canary Score in Spinnaker Pipeline

If you have configured the automated Canary analysis correctly, you will get the Canary score as part of your every pipeline run.   You can set up the score threshold to automatically deploy the new version or terminate the pipeline.  You can also look at the Canary report for further diagnosing the issues with the new version.

Add Automated Red/Black Deployment Analysis:

If you are like most and using Red/Black deployment, you can ACA task stage to do an automated analysis of that deployment.

Add and configure ACA task stage in Spinnaker pipeline after your current deploy stage.

Note: The ACA task stage can be configured for other cloud providers (in addition to AWS) starting in Spinnaker 1.5 release.

Add ACA Task Stage to Spinnaker Pipeline

When you run the pipeline, the ACA stage is performed on your red/black deployment

Automated Red/Black Analysis in Spinnaker Pipeline

If you have configured the ACA task stage correctly, you will get the automated Canary score as part of your pipeline run.   You can set up the score threshold to automatically trigger roll-back of the new release if needed.

In future blogs, I will provide more details about best practices for configuring the Canary and ACA task stage to ensure optimal automated analysis.

If you are interested in OpsMx automated release analysis for Canary or Red/Black deployment,  check out the interactive product walk-through. For a free trial account, fill out the form below or email us at info@opsmx.com.