Deploying to Azure Cloud using Spinnaker

Introduction

Spinnaker as a renowned Continuous Delivery tool supports app deployment into all major cloud environments including Azure cloud. In this blog, we will guide you on how to configure a Spinnaker CI/CD pipeline to deploy an app to the Azure Cloud using Spinnaker.

Prerequisites

• Valid Azure Subscription/Account Spinnaker 1.16.2
• Working Jenkins integrated with Spinnaker. Jenkins does your CI build
• Our app is a Java springboot application, to be deployed into the Azure cloud.
• These inputs are required for configuring Load Balancer and Firewall:
  • The app is listening on port 8080
  • Healthcheck URL is http://localhost:8080/health
  •  
    App URL is http://localhost:8080/greeting

Preparing Azure for Spinnaker Integration

• Azure CLI for creating Azure resources for Spinnaker
• Azure Portal (portal.azure.com) for Virtual network setup and Action monitoring

Installing Azure CLI

Azure CLI is required to set up pre-requirements on the Azure cloud. Install Azure CLI on your Laptop or any VM which has connectivity to the Azure cloud. Simply run this command

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

For detailed manual instruction, you can follow the guide from the link https://docs.microsoft.com/en-gb/cli/azure/install-azure-cli-apt?view=azure-cli-latest

Create Azure resources for Spinnaker Integration

In your Azure account, essentially you will need a few items created…

• Service Principle (Halyard to communicate with Azure),
• Resource Group (All of the Azure resources like temp VMs, machine image, etc to be grouped under this name)
• Azure Vault (Default Username and password/key of Azure VMs created Spinnaker are stored here) – SSH Username (Default username of VM) – SSH Password (Default password of VM

Throughout the procedure here, I choose to configure and deploy Azure resources in the eastern US region. But you can change it as per your choice.

Configure Azure CLI to create Azure resources

1. Login to Azure by Azure CLI

saga@sagamc:~$ az login Note: We have launched a browser for you to login. For old experience with device code, use "az login --use-device-code" Opening in existing browser session. You have logged in. Now let us find all the subscriptions to which you have access:
[
  {
    "cloudName": "AzureCloud",
    "id": "959c83bf-PKljgfr0f16-4a56-a54267be765432e", #Subscription ID 
     "isDefault": true,
    "name": "Microsoft Azure Sponsorship",
    "state": "Enabled",
    "tenantId": "d1ea76543-64d0-48u4ed-a093-pk8041071ee6",
    "user": {
      "name": "sagayaraj.d@opsmx.io",
      "type": "user"
    }
  }
]

2. Get your Subscription details.

Note: You can also get your subscription from Azure portal: Home -> Subscriptions

saga@sagamc:~$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "959c83bf-PKljgfr0f16-4a56-a54267be765432e", #Subscription ID
    "isDefault": true,
    "name": "Microsoft Azure Sponsorship", #Subscription Name
    "state": "Enabled",
    "tenantId": "d1eay7473-64d0-44poed-a093-ad8041071ee6",
    "user": {
      "name": "sagayaraj.d@opsmx.io",
      "type": "user"
    }
  }
]

3. Make your Azure CLI to work with your Subscription.

saga@sagamc:~$ SUBSCRIPTION_ID=959c83bf-PKljgfr0f16-4a56-a54267be765432e
saga@sagamc:~$ az account set --subscription $SUBSCRIPTION_ID

Create Azure resources

1. Create a Service Principal. This is the account by which Spinnaker’s Halyard will communicate with Azure cloud.

Syntax: az ad sp create-for-rbac --name "<ServicePrincipalName>"

saga@sagamc:~$ az ad sp create-for-rbac --name "Spinnaker-SP"
Changing "Spinnaker-SP" to a valid URI of "http://Spinnaker-SP", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
  Retrying role assignment creation: 3/36
{
  "appId": "fc061b7e-d5k8c5-474d-a888-f34fd69bd85b",
  "displayName": "Spinnaker-SP",
  "name": "http://Spinnaker-SP",
  "password": "1fd416e8-15e2-478995-8a6e-aea4eabca3be",
  "tenant": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
}

2. Make a note of AppId and TenantId

saga@sagamc:~$ APP_ID=fc0ki61b7e-d5c5-474d-a888-f34fd69bd85b
saga@sagamc:~$ TENANT_ID=d1ea7473-64hygd0-44ed-a093-ad8041071ee6

3. Create a resource group in eastus region.

az account list-locations --query [].name
Syntax: az group create --name <resourcegroup-name> --location <location>

saga@sagamc:~$ RESOURCE_GROUP=SpinHal-RG
saga@sagamc:~$ az group create --name $RESOURCE_GROUP --location eastus
{
  "id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG",
  "location": "eastus",
  "managedBy": null,
  "name": "SpinHal-RG",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

4. Create an Azure vault. The vault will store the default username and password of VMs.

Note: Vault names are used in a DNS record (like spinnakervault.vault.azure.net), so the name must be unique across all customers.

saga@sagamc:~$ VAULT_NAME=SpinHal-Vault
saga@sagamc:~$ az keyvault create --enabled-for-template-deployment true --resource-group $RESOURCE_GROUP --name $VAULT_NAME
{
  "id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
  "location": "eastus",
  "name": "SpinHal-Vault",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "9aeada99-4d00y6-4f53-8567-956c8645e905",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "d1ea7473-647ytd0-44ed-a093-ad8041071ee6"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": true,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "d1ea7473-6874d0-44ed-a093-ad8041071ee6",
    "vaultUri": "https://spinhal-vault.vault.azure.net/"
  },
  "resourceGroup": "SpinHal-RG",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

5. Set Vault permissions

saga@sagamc:~$ az keyvault set-policy --secret-permissions get --name $VAULT_NAME --spn $APP_ID
{
  "id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
  "location": "eastus",
  "name": "SpinHal-Vault",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "9aeada99-4d980-4f53-8567-956c8645e905",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "d1ea7473-hgtfrd0-44ed-a093-ad8041071ee6"
      },
      {
        "applicationId": null,
        "objectId": "05676264-65njhgtc8-48c7-925d-00daba4bc5d6",
        "permissions": {
          "certificates": null,
          "keys": null,
          "secrets": [
            "get"
          ],
          "storage": null
        },
        "tenantId": "d1ea7473-64nhbgvfd0-44ed-a093-ad8041071ee6"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": true,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "d1eahygtr473-64d0-49874ed-a093-ad8041071ee6",
    "vaultUri": "https://spinhal-vault.vault.azure.net/"
  },
  "resourceGroup": "SpinHal-RG",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

6. Insert the default SSH username of VMs into your Azure vault

saga@sagamc:~$ VMUSER=ddsaga
saga@sagamc:~$ az keyvault secret set --name VMUsername --value $VMUSER --vault-name $VAULT_NAME 
{
  "attributes": {
    "created": "2019-11-14T03:12:18+00:00",
    "enabled": true,
    "expires": null,
    "notBefore": null,
    "recoveryLevel": "Purgeable",
    "updated": "2019-11-14T03:12:18+00:00"
  },
  "contentType": null,
  "id": "https://spinhal-vault.vault.azure.net/secrets/VMUsername/a7d987a593552874f76afef83cfd11d8232",
  "kid": null,
  "managed": null,
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "ddsaga"
}

7. Set the public key in Vault

a. Generate SSH pubkey to insert into Vault

saga@sagamc:~$ ssh-keygen -f ddsaga -N ''
#Username is ddsaga (Change it as per your choice)
#cat ddsaga.pub and copy the content without the last part <user>@<host> [e.g. opsmxuser@spinnaker-saga]

saga@sagamc:~$ VMUSER_KEY="ssh-rsa AAAAB3NzaC1iuytgtgggyc2EAAAADAQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"

b. Insert the default SSH pubkey of VMs into your Azure vault

saga@sagamc:~$ az keyvault secret set --name VMSshPublicKey --vault-name $VAULT_NAME --value "$VMUSER_KEY"
{
  "attributes": {
    "created": "2019-11-14T03:21:02+00:00",
    "enabled": true,
    "expires": null,
    "notBefore": null,
    "recoveryLevel": "Purgeable",
    "updated": "2019-11-14T03:21:02+00:00"
  },
  "contentType": null,
  "id": "https://spinhal-vault.vault.azure.net/secrets/VMSshPublicKey/38ab80bc184d4beab03e35dff30c9c8f",
  "kid": null,
  "managed": null,
  "tags": {
    "file-encoding": "utf-8"
  },
  "value": "ssh-rsa AAAAB3NzaC1yc2gfrEAAAADAKKKKQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"
}

That brings up the end of activities on Azure CLI.

Integrating Azure provider in Spinnaker

Use Halyard command-line program to enable the integration of Azure in Spinnaker.

Note: The Procedure below should be done on hal machine (VM/Pod/Container)

1. During the above procedure “Create Azure resources for Spinnaker Integration“, you would have created resources like Service Principle, Resource Group and Azure Vault. Make a note of those values and set them as env variables here:

spinnaker@hal-xyz:~$ cat > azureapp.vars <<-EOF
SUBSCRIPTION_ID=959c83bf-PKljgfr0f16-4a56-a54267be765432e
APP_ID=fc061b7e-d5c5-4decde74d-a888-fdsws34fd69bd85b
TENANT_ID=d1ea74xscde73-64d0-44ed-a04de393-ad8041071ee6
RESOURCE_GROUP="SpinHal-RG"
VAULT_NAME=SpinHal-Vault 
EOF

spinnaker@hal-xyz:~$ source azureapp.vars

Optionally, you can verify the variables are set correctly

echo $APP_ID
echo $TENANT_ID
echo $SUBSCRIPTION_ID
echo $VAULT_NAME
echo $RESOURCE_GROUP

2. Execute the ‘hal’ command to configure hal with Azure. This step will prompt you for password. The password was outputted for you when Service principle was created via Azure CLI.

spinnaker@hal-xyz:~$ hal config provider azure account add saga-azure-account \
--client-id $APP_ID \
--tenant-id $TENANT_ID \
--subscription-id $SUBSCRIPTION_ID \
--default-key-vault $VAULT_NAME \
--default-resource-group $RESOURCE_GROUP \
--packer-resource-group $RESOURCE_GROUP \
--app-key

3. Enable Azure provider in Spinnaker

spinnaker@hal-xyz:~$ hal config provider azure enable

4. You may review the Azure provider configuration

spinnaker@hal-xyz:~$ hal config provider azure account list
saga-azure-account #Output would contain this value. Use this value in the next command

spinnaker@hal-xyz:~$ hal config provider azure account get saga-azure-account

#Enable eastus region. The region should match the region configured in the earlier
Azure CLI steps

spinnaker@hal-xyz:~$ hal config provider azure account edit saga-azure-account - -
regions=eastus

5. Complete the hal configuration with “hal deploy apply” command. Note: Now if you create a new application in Spinnaker, the application should have the provider ‘azure’ in the list.

Azure Virtual Network

Spinnaker will deploy applications on a virtual network in the Azure cloud. Hence, ensure to have a Virtual Network and a minimum one Subnet created in eastus region (or another region, it should match hal configuration). You may create the virtual network on portal.azure.com My Virtual network details are…

Network name: Spin_Vnet
CIDR: 172.16.0.0/22
Subnet: default

The Virtual Network and Subnet will be used by Firewall, Load Balancer, and Server group configurations.

Azure and Spinnaker Resource Mapping

Spinnaker generalizes the Cluster terms across multiple clouds. The resource in Spinnaker could mean a different things in the Azure cloud.

• Spinnaker Firewall means Network Security Group in Azure cloud.
• Spinnaker Load Balancer is referred to as Load Balancer only in  Azure.
• Spinnaker Server Group is referred to as Virtual Machine ScaleSet in Azure.

Configuring Azure application in Spinnaker UI

Note: Perform these steps in Spinnaker UI.

1. Create an Application

Create an Application in Spinnaker for deployments to Azure

Under Applications, Create an application ‘appxinazure’. Ensure to select ‘azure‘ in the provider list.

2. Configure Application’s Infrastructure

Prior to configuring a CD pipeline, we need to configure the Infrastructure of the application. A separate resource group by name <appname>-<region> will be created in Azure cloud automatically. In our case, ‘appxinazure-eastus’ is the new resource group. The resources created in the Spinnaker UI like Firewall, Load Balancer, VM ScaleSet are stored on this resource group; however, the baked images are stored on the resource group configured in Halyard. ** To successfully configure the Firewall, Load Balancer and Server group – Virtual network and Subnet are one of the pre-requirement. You should have them created. If not, refer to the section Azure Virtual Network to set up one.

a. Create Firewall

Create Firewall in Spinnaker for the app deployment to Azure

b. Create a Load Balancer

You should know the application listening port, health check page, app URL so that the load balancer can be configured correctly.

Create LoadBalancer in Spinnaker for the app deployment to Azure

c. Create Server Group

Do not create a server group using the Infrastructure tab, because you may be unsuccessful – while it will look for an image that will not available until we perform a Bake stage. Hence, the configuring server group is part of the pipeline setup process. You will have to create a server group in the Deploy stage.

Configuring the Spinnaker CI/CD pipeline for deployments to Azure

Under the Pipeline section, click ‘Create’ to create a New Pipeline with the type being selected as Pipeline. Input a valid name for your pipeline.

1. Configure Stage

Create Stage for triggering Spinnaker pipeline for Jenkins CI build of the app for deployment to Azure

Here, select your Jenkins CI account to take Build information. Prior to this step, you should have your Jenkins configured in Halyard under CI systems.

2. Bake Stage

Create Bake Stage for baking the Jenkins CI build of the app for deployment to Azure

We are targeting the package to Ubuntu server (16+), while the base image is baked with our app installed on it. The deb package file name should have been created with the naming convention: name_version-release_arch. Here, select your package (.deb) name to create a baked image. The package is essentially the output of Jenkins Build. This stage will look for a valid package within the Jenkins project directory. Select your region ‘eastus‘ to deploy the application into. Remember the available regions here are activated by hal command. Then, type your package name without any version information

3. Deploy Stage

The deployment server group in Spinnaker will create a VMScaleSet on Azure, the LoadBalancer will route the traffic to the VM ScaleSet.

Create Deploy Stage for deploying the app to Azure

Add a Server group and configure every section in ‘Configure Deployment Cluster‘. Choose the account, select the region as ‘eastus‘, valid stack that may denote environment like dev/qa/stg. Detail may contain any text, but I chose to have ‘svrgroup’. Then choose the load balancer you had created in one of the above steps.

Configure Azure deployment Cluster

Choose the Virtual network and Subnet

Choose the Virtual network and Subnet, where the VM ScaleSets will be created during CD pipeline execution.

Choose the Firewall

Choose the Instance type that best fits for your application. Here, we have gone with General purpose computer.

Choose the Azure Instance type that best fits for your application

You can associate some tags that may be used as custom meta-data for your VMs, which can be used for filtering and other operations.

Associate custom tags

Advance Settings

Save the configuration. And, you are done with your Pipeline configuration. You are now ready to execute your Pipeline and see how it goes.

Troubleshooting Known Issues

1. Problem:

In the Bake stage, the package name was mentioned as restapp_ (since our build output package was restapp_1.0_all.deb). The Baking process throw the error “Unable to find deployable artifact starting with [RestApp ] and ending with .deb in [] and [restapp_1.0_all.deb]”

Solution:

Make sure your deb package file name complies with the naming convention: name_version- release_arch. In the Bake stage, just mention the name part alone (Do not include the underscore).

2. Problem:

The Baking process throws the error “==> azure-arm: E: Unable to locate package restapp”

Solution:

This is possible when you have not configured your Deb path in Spinnaker. Update the rosco-local file with deb repo and redeploy with hal deploy apply command.

On halyard Pod/VM,

$ cat >> /home/spinnaker/.hal/default/profiles/rosco-local.yml <<-EOF
debianRepository: http://spindd.s3-website-us-west-2.amazonaws.com trusty main
azure:
  enabled: true
EOF

03. Problem:

The Baking process throws the error “==> azure-arm: E: Failed to fetch http://spindd.s3-website-us-west-2.amazonaws.com/dists/trusty/main/binary-amd64/Packages  404 Not Found ==> azure-arm: E: Some index files failed to download. They have been ignored, or old ones used instead.”

Solution:

If Deb repository is an S3 bucket, you should enable the web-hosting option and use the Endpoint URL in the rosco-local.yml file. The endpoint URL becomes available once you enable Web-hosting.

04. Problem:

The Baking process throws the error “==> azure-arm: -> Additional Disk: skipping, managed disk was used… Build ‘azure-arm’ errored: unexpected EOF”

Solution:

The problem occurs due to a defect in Packer version 1.4.2 as stated here https://github.com/has hicorp/packer/issues/7816. The fix is to update the Packer version to 1.4.4 which is packaged with Spinnaker 1.17. This is confirmed here https://github.com/spinnaker/rosco/pull/452. Either update the Spinnaker to 1.17 or you can update the rosco.yml file with overridden docker image version which has the Packer 1.4.4 updated. See below how to update the docker version for Rosco.

$ cat > /home/spinnaker/.hal/default/service-settings <<-EOF
artifactId: gcr.io/spinnaker-marketplace/rosco:0.15.0-20191023172815
EOF

05. Problem:

Got an error in the deployment stage… “Exception ( Monitor Deploy ) Multiple errors occurred: NotFound, NotFound. Please see details., Unexpected exception: null! Please login into Azure Portal and manually delete any resource associated with the appxinazure-dev-fe-v000 server group such as storage accounts, internal load balancer, public IP, and subnets”

Solution:

Spinnaker did not provide enough log information as to which resource is NotFound. In tracing the resources everything we created, found that the items – VMUsername, and VMSshPublicKey were missing in Azure vault SpinHal-vault

After adding the SSH key and public-key, the deployment stages went successfully.

Refer to the Azure CLI procedure again to create the Username and Public-key. In particular the commands,

saga@sagamc:~$ az keyvault secret set --name VMUsername --vault-name $VAULT_NAME --value $VMUSER
saga@sagamc:~$ az keyvault secret set --name VMSshPublicKey --vault-name $VAULT_NAME --value
"$VMUSER_KEY"

06. Problem:

References:

https://www.spinnaker.io/setup/install/providers/azure/ https://www.spinnaker.io/guides/tutorials/codelabs/azure-vmss-source-to-prod/  

If you want to know more about Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us