Baking AWS AMI Image in Spinnaker Continuous Delivery Pipeline

To create an AWS EC2 instance, or maintain the high-availability of the application with auto-scaling-group (ASG) requires a base image (AMI) in AWS. This base image should contain the application that you want to install. In this article, we will be covering the procedure to bake an AWS AMI image in the Spinnaker Continuous Delivery (CD) pipeline.

Pre-Requisites for setting up the baking stage (for AWS AMI) in Spinnaker

For setting up a successful baking stage in Spinnaker, the requirements are as follows:

• A working Spinnaker instance.
• Access to the AWS console with a valid account.
• Working knowledge of AWS services like IAM and EC2 including AMI, EC2 instance, AutoScaling Group.
• Working knowledge of the Linux environment.

Spinnaker pipeline requires the following:

• A Jenkins job that publishes Debian packages (for Ubuntu image build), so that Spinnaker can read the package name.
• A Debian repository to install the application using the .deb packages.

Understanding the Spinnaker Baking Process in AWS

The process of baking an AMI image refers to creating an immutable base image that becomes the basis for creating application EC2 instances or server-groups. This process of installing an application on the existing OS, and generating an AMI from the current state is known as Baking AMI.

For Ubuntu operating systems, you may use the `apt` command-line program to install from a Debian repository. In Spinnaker, you need to mention the repository from where to fetch the application, and the installation process is abstracted implicitly.

Spinnaker’s Rosco service is responsible for governing the baking process. So, you need to provide the package repository location and the IAM credentials to connect with your AWS account. Under the hood, the Rosco is using a `packer` to perform AMI baking.

Typically the baking process takes place after a successful CI build process which would generate a .deb package and publish it to a repository. Hence Baking AMI process depends on a CI system like Jenkins.

Creating IAM User with Spinnaker Baking permissions in AWS

In order for Spinnaker to connect with AWS, it should have a proper credential. This credential is provided by the AWS admin. 

The AWS admin sets up IAM credentials as below:

1. Go to IAM in AWS Console. Users > Add User > Name ‘SpinBakeryUser’. Select ‘Programmatic access’.
2. Next: Tags > Have optional tags as required
3. Review and Create. Make a note of the ACCESS_KEY and SECRET_KEY
4. Next, Add a permission policy with a dedicated policy ‘AMIBakeryPolicy’ or with an Inline Policy to the user. The policy JSON as below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                  "ec2:*"
            ],
            "Resource": [
                    "*"
            ],
            "Effect": "Allow"
        }
]
}

Configuring Spinnaker Rosco Service to access Debian Packages

Remember, Custom profiles are configuration files that override the default configuration files for each Spinnaker Service. These files are created in the Halyard path: ~/.hal/default/profiles/

So, to let Rosco know from where to read the Debian packages, you need to update the Rosco profile with a custom rosco-local.yml file. The content is mentioned below:

debianRepository: http://demojenkins.net:8181/ trusty main

It is important that Halyard and Rosco are able to connect with the repository. You may check the connectivity from Halyard by using the following command :

nc -zvw5 demojenkins.net:8181

Applying the Configurations via Halyard

Once an IAM user is created with sufficient EC2 permissions and Rosco is configured with a Debian repository, you can safely enable the AWS bakery flags. Run the following commands in Halyard:

export ACCESS_KEY_ID=xxxxxxxxxx
hal config provider aws bakery edit –aws-access-key ${ACCESS_KEY_ID} –aws-secret-key #You will be prompted for a secret
# Optionally, you can specify a default subnet to bake
# hal config provider aws bakery edit –aws-subnet subnet-abc012345
hal deploy apply

Note: In case you did not mention the target subnet here, you can still configure the VPC and Subnet in the Pipeline Bake stage.

AWS AMI Bake Stage in Spinnaker Continuous Delivery Pipeline

The Bake stage depends on a successful build on Jenkins. Jenkins should have a post-build Archive step for Spinnaker to read the packages from it. You should have integrated Jenkins beforehand using Halyard. 

Spinnaker Pipeline Configuration

In the Pipeline configuration stage, under Trigger Type, choose ‘Jenkins’ and the Controller to ‘<Jenkins master-name>’. The Job name should be the one that generates the Debian package.

AWS AMI Bake stage in Spinnaker pipeline

Once the pipeline is configured, follow the steps given below:

1. In the Pipeline, add a stage after the Configuration stage. Let the type be ‘Bake’. 
2. In the ‘Base Configuration’ select a region to bake the AMI and type the package name without the version information. 
3. Choose a Base OS from the available list.

Here, the Bake stage configuration screen for your reference.

4. After setting up the Bake Configuration, go back to the Pipeline Execution page and trigger a manual build. Wait for the pipeline execution to be completed. 
5. Once it is done, you can review the Baking log to identify the AMI id the pipeline has created.

Common Errors and Fixes

1. Error: Pipeline exits with the log : No permission to access AWS EC2 instance

packer_5f4ee5e0-c7d8-6524-7f28-bcfc6a516c69
Build ‘amazon-ebs’ errored: Error creating temporary keypair: retry count exhausted. Last err: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: SKHQSzSTAfx- *********************_-0GSM0gCHuTt
status code: 403, request id: 713d9451-6674-47b9-9ed4-99f986a912f9

• Cause: The bakery user does not have sufficient permission to create Keypair and EC2 instances.
• Fix:  Go to the AWS console, and edit the User’s policy with “EC2:*” to Actions.

2. Error: Fail to fetch

W: Failed to fetch http://demojenkins:8181/dists/trusty/InRelease

    amazon-ebs:
    amazon-ebs: W: Failed to fetch 
http://demojenkins:8181/dists/trusty/Release.gpg  Could not resolve 'demojenkins'
    amazon-ebs:
    amazon-ebs: W: Some index files failed to download. They have been ignored, or old ones used instead.

• Cause: The Debian repository server is not reachable.
• Fix:  Check the connectivity using `ping <server>`, or else try using the server’s IP address. Also, make sure the Debian repository service (Aptly is running) on the remote server.

Summary

We have explained the procedure to perform a successful AMI baking through the Spinnaker Continuous Delivery CD pipeline by creating an IAM user with sufficient permissions, configuring Halyard with Debian-repository, and enabling AWS bakery configuration. Now that you know the process go ahead and give it a try yourself!

---

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us