External Account Configuration in Spinnaker

Kiran G January 7, 2020

Introduction

As the Spinnaker is evolving, new features are being incorporated and one of them is External Account Configuration which allows the configurations to be externalized via Git, Hashicorp Vault, databases, CredHub, etc. You may be asked if we can keep the cloud provider account details in a remote repository? If so, what if an account change happens in that remote configuration while the Spinnaker service is running? Will it take effect without service restart? What are the steps involved? What other use cases are possible with remote configuration?

This blog provides answers to the above questions. In Spinnaker world, the credentials configured through halyard using which Spinnaker deploys the applications into various Cloud Platforms are known as Accounts. Typically complete account configuration resides in hal config. ‘hal deploy apply’ is a Spinnaker mantra for any account change to take effect. Broadly account can be of any cloud provider, artifactory, CI system, storage system, etc. When such an account is added or updated or deleted, ‘hal deploy apply’ makes Spinnaker understand the change. But this also means a restart of one or more affected Spinnaker services. Spinnaker is built on the Spring framework and uses various Spring modules. One of such modules is Spring Cloud Configuration which allows dynamic external configuration using Config Server and Client implementation. A client application can be refreshed for reloading any account changes that the Config server is designed to capture. Spinnaker service need not be restarted for the account changes in external configuration. This refresh can be automatic or manual. As of Spinnaker 1.17.5, only Clouddriver service supports automatic refreshing of external accounts and only for Kubernetes and Cloud Foundry cloud providers. For manual triggering, /refresh endpoint on the service be http POSTed.

External Account Configuration for Spinnaker

A file named spinnakerconfig.yml is a bootstrap configuration file that provides the location of the remote configuration so that Spinnaker services like clouddriver, igor fetch configuration from the remote location at startup. The following examples show the configuration with Git as a remote repository. Spinnaker uses profiles like <service>-local.yml for additional configuration and profile is loaded from the remote configuration, if present. Remote locations can be one or a mix of the following backends.

Git
Hashicorp Vault
JDBC
CredHub
Local File System

The next sections provide step by step instructions for the following dynamic configurations:

Account Configuration
Secrets Configuration
Secrets Encryption
Configuration Files

Using this dynamic account configuration feature, one or more cloud provider accounts(Kubernetes, Azure, AWS, Cloud Foundry, etc) and/or CI accounts and/or notification services can be configured in the external sources mentioned above. Keeping all the secrets in plain text in hal config may not always align with organizational policies so using the dynamic configuration feature, it is possible to ship the secrets to external repositories. The secrets in the external repositories can also be encrypted and Spinnaker services decrypt them at runtime. Configuration files like Kubernetes config file, Google Cloud configuration file can also be kept in Git or other backends and it can be configured in Spinnaker to retrieve them at runtime.

External Account Configuration for Spinnaker

This section explains how to configure the following accounts with Git as an external source. The same configuration can be extended to any other cloud provider accounts.

Kubernetes Account
AWS Account
Azure Account
Jenkins Master Account

Create a Git repository for storing the external accounts

Let the repository be private with the name spinnaker-config and the github account username be opsmx, so the git repository URL is https://github.com/opsmx/spinnaker-config.git.
Place spinnakerconfig.yml file under ~/.hal/default/profiles/

The file spinnakerconfig.yml (or <service-name>config.yml, i.e., clouddriverconfig.yml, igorconfig.yml, etc) contains the external repository details using which Spinnaker services fetch the account configurations. Following spinnakerconfig.yml must be placed under ~/.hal/default/profiles/ in halyard pod or the machine in which halyard service is running. If a spinnaker service (say clouddriver pod) is configured to run without halyard, then the file is pushed under /opt/spinnaker/config/ in the respective spinnaker service pod at startup.

spinnakerconfig.yml
```
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/spinnaker-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
```

Create a file ‘clouddriver-local.yml’ in the Git repository

The file clouddriver-local.yml contains all the cloud provider accounts that are to be externally configured. The following configurations are contained in this file:

Kubernetes Account

    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: /home/opsmx/.hal/default/staging/dependencies/1031763043-config
        oAuthScopes: []
        onlySpinnakerManaged: false

Note: The kubeconfigFile is not configured in Git. External config file configuration is explained in the following sections.

AWS Account

    aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '123456789012'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
        awsAccessKey: AKIAXXXXXXXXXXXZHKU5 
        awsSecretKey: YqfRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
      accessKeyId: ALNMXXXXXXXXXXXZHKT8
      secretAccessKey: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl4dK
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole

Azure Account

   azure:
      enabled: true
      accounts:
      - name: my-azure-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        clientId: 921f7add-XXXX-XXXX-XXXX-c1a695s08755
        appKey: 23373f24-XXXX-XXXX-XXXX-6b0b5113f5a0
        tenantId: d1ea7473-XXXX-XXXX-XXXX-ad8041071ee6
        subscriptionId: 959c83bf-XXXX-XXXX-XXXX-2067be19692e
        defaultResourceGroup: RG-OpsMx
        defaultKeyVault: SpinHal-OpsMx-Vault
        packerResourceGroup: RG-OpsMx
        regions:
        - westus
        - eastus
        useSshPublicKey: 'true'
      primaryAccount: my-azure-account
      bakeryDefaults:
        templateFile: azure-linux.json
        baseImages: []

For simplicity, only one account per cloud provider is configured but they can be more than one. Note that any accounts of the above three cloud provide configured in hal config are overridden by the above configuration.

Create a file ‘igor-local.yml’ in the Git repository

We configure the Jenkins Master account in igor-local.yml and place it directly under the Git repository created in step 1. The following configuration is contained in this file:
```
 ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: http://jenkins.opsmx.com
        username: opsmx
        password: mypassword
```
Run ‘hal deploy apply’

At this point, there are two files – clouddriver-local.yml and igor-local.yml – in the Git repository, and Git repository details are configured in ~/.hal/default/profiles/spinnakerconfig.yml. Upon running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into all spinnaker service pods under /opt/spinnaker/config/ and the pods fetch the respective <service>-local.yml files into their /tmp/config-repo/ directory and load the accounts. In our scenario, clouddriver and igor pods load the accounts from clouddriver-local.yml and igor-local.yml files respectively.

Secrets Configuration for External Accounts

The following procedure details the secrets configuration using Git. The rest of the configuration lies with the halyard. Two scenarios are explained here: 1. AWS access key id and secret key and 2. Jenkins Master address, username, and password.

Configure hal config with Spring property placeholders

AWS Account Configuration in hal config

Configure AWS Account in hal config with spring property placeholders for AWS Access Key Id and Secret Key which get replaced with the actual values at run time by the clouddriver service. After configuring, the AWS part of the hal config looks like the following:

   aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '732813442182'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
      accessKeyId: ${aws.access-key}
      secretAccessKey: ${aws.secret-key}
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole

Jenkins Master Account Configuration in hal config

Similarly configure hal config for Jenkins Master Account with spring property placeholders for Jenkins Master address, username, and password as shown below:

  ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: ${jenkins.address}
        username: ${jenkins.username}
        password: ${jenkins.password}

Create a Git repository for storing the secrets

Let the repository be private with name secrets-config and the GitHub account username be opsmx, so the git repository URL is https://github.com/opsmx/secrets-config.git.
Create a clouddriver-local.yml file in the Git repository

The file clouddriver-local.yml contains the actual values for the AWS Access Key and Secret Key:
```
aws:
    access-key: ALIAXXXXXXXXXXXZHKU5
    secret-key: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
```
Create an igor-local.yml file in the Git repository

The file igor-local.yml contains the actual values for the Jenkins Master Account’s address, username, and password:
```
jenkins:
    address: http://jenkins.opsmx.com
    username: opsmx
    password: mypassword
```
Enable external configuration through spinnakerconfig.yml

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which the halyard service is running with the following configuration.

spinnakerconfig.yml
```
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/secrets-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
```
Run ‘hal deploy apply’

Now all the configuration is ready and by running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into clouddriver and igor pods which fetch the secrets from Git and replace the spring property placeholders at runtime.

Secrets Encryption in Spinnaker

Please refer to this blog to know how to externalize encrypted secrets: Managing Secrets in Spinnaker – Encryption using Symmetric Key

Dynamic External Account Configuration File for Spinnaker

The following steps explain how to configure a kubeconfig file that is checked-in a git repository.

Create a Git repository for storing the config files

Let the repository be private with name config-files and the github account username be opsmx, so the git repository URL is https://github.com/opsmx/config-files.git
Create a file kubeconfig.yml in the Git repository
Create kubeconfig.yml which provides cluster access for organizing Kubernetes accounts, in the Git repository created in step 1.

Refer kubeconfig.yml in clouddriver-local.yml

Create a clouddriver-local.yml file in the Git repository file which contains the Kubernetes account details including the path to the kubeconfig file. Since the clouddriver service is backed by Spring Cloud’s config server, by configuring appropriately, it fetches the kubeconfig file from the Git repository.

`clouddriver-local.yml`

    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: configserver:kubeconfig.yml
        oAuthScopes: []
        onlySpinnakerManaged: false

Place spinnakerconfig.yml file under ~/.hal/default/profiles/

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which the halyard service is running with the following configuration.

spinnakerconfig.yml
```
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/config-files.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo
```
Run ‘hal deploy apply’

By running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into the clouddriver pod which loads the Kubernetes account and kubeconfig.yml from Git.

Note – Currently, dynamic accounts only work with Kubernetes and not AWS.

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us

29th July 2021

15th Jul 2021

01st Jul 2021

17th Jun 2021

29th July 2021

15th Jul 2021

01st Jul 2021

17th Jun 2021

External Account Configuration in Spinnaker

Introduction

External Account Configuration for Spinnaker

External Account Configuration for Spinnaker

Create a Git repository for storing the external accounts

Place spinnakerconfig.yml file under ~/.hal/default/profiles/

spinnakerconfig.yml

Create a file ‘clouddriver-local.yml’ in the Git repository

Kubernetes Account

AWS Account

Azure Account

Create a file ‘igor-local.yml’ in the Git repository

Run ‘hal deploy apply’

Secrets Configuration for External Accounts

Configure hal config with Spring property placeholders

AWS Account Configuration in hal config

Jenkins Master Account Configuration in hal config

Create a Git repository for storing the secrets

Create a clouddriver-local.yml file in the Git repository

Create an igor-local.yml file in the Git repository

Enable external configuration through spinnakerconfig.yml

spinnakerconfig.yml

Run ‘hal deploy apply’

Secrets Encryption in Spinnaker

Dynamic External Account Configuration File for Spinnaker

Create a Git repository for storing the config files

Refer kubeconfig.yml in clouddriver-local.yml

clouddriver-local.yml

Place spinnakerconfig.yml file under ~/.hal/default/profiles/

spinnakerconfig.yml

Run ‘hal deploy apply’

You May Like

OpsMx Enterprise for Spinnaker 3.2 is Generally Available

Interested in Contributing to Spinnaker? Attend Spinnaker Code Night

Readiness probe issue with custom image for Spinnaker microservices

Submit a Comment Cancel reply

`spinnakerconfig.yml`

`spinnakerconfig.yml`

`clouddriver-local.yml`

`spinnakerconfig.yml`