Make CI/CD pipeline Compliant and Auditable using OpsMx Enterprise for Spinnaker

CI/CD pipelines form the backbone of the modern software lifecycle. They help the DevOps team in developing and deploying cloud-native applications at velocity. While software delivery speed is good, it is equally critical to ensure the software delivery pipeline is compliant with governance policies and industry standards. Non-adherence will always lead to hefty fines and can severely affect profitability; for reference, please read WSJ’s article: Regulators reprimanded Citi Bank $400M for falling short of risk management, data management, and regulatory reporting.

Ensuring CI/CD Compliance is Eminent with Speed

Compliance problems can arise from disorderly development workflows. For example, a new software release introduced to production will be of high risk if it has not gone through proper functional and performance testing. Or perhaps an unauthorized person deployed a defunct release into production. The losses can involve reputational and business damage.

Due to the current speed of releasing software, it is highly likely that a new release with some potential threats will go undetected. In a few cases, policy or the compliance managers enforce policies at the end of the deployment process; well, the cost of an issue at the end of software delivery becomes exponentially high.

Hence with growing deployment frequency, there is a new trend among enterprise to:

• shift compliance to the left of the CI/CD process
• perform more audit trails, i.e., tracking necessary approvals and versions of applications that are getting deployed into production
• use more guards and guardrails to inspect any unwanted release and avoid unauthorized rollouts
• seek higher collaboration between DevOps and compliance & audit team

Challenges in CI/CD Compliance and Audit

The Compliance and Audit function spins around all enterprise departments- Finance reporting/ regulatory controls/ data privacy/ vendor management/IT. Compliance managers and auditors work tirelessly to ensure that all the processes adhere to SOX, HIPAA, GDPR, etc. Although compliance managers and auditors work behind the veil, a small situation related to non-adherence can become a news headline. They have to be attentive round the clock.

Compliance and Audit is a corporate plan, and the team usually works manually. With rising microservices to hundreds and thousands, manually handling compliance & risk exposure is not sufficient and scalable. Below are some of the challenges for the team to cope with the speed of CD:

1. Day-in-a-life of compliance team

   Every time a developer merges code in Github, it needs to be built, tested, and deployed with proper policy checks. DevOps team notifies compliance or policy managers through JIRA or ServiceNow to perform their inspections. After the manual reviews and policy checks, a compliance manager would approve the ticket, following which the DevOps team would trigger the deployment. Using policy and governance controls to achieve safety is carried out at the end of the deployment and is arduous.

   Fig 1A shows detailed lists of controls a policy manager would typically make in the non-prod and prod deployment stages to ensure secured and risk-free deployments.

   

   Manual controls enforced at the end of the CI/CD process

2. Day-in-a-life of audit team

   For audit and reporting, auditors would require information such as time-taken for deploying a release or patch, when the deployment activities carried out, regions where deployment happened, success/failure of deployment pipelines, the person who deployed, etc.. These data-gathering activities, documentation, communication, and reporting are all done through spreadsheets or over email. Auditors would agree that this is an inefficient process and, to some extent, boring and annoying work ( annoying because developers, testers, and the Ops team treat uncovering of information related to deployment as an ad hoc activity and not a part of their Job description).

3. Changing dynamics of policies in a CI/CD process

   Not only the miasma of documentation but also the implementing of any change in policies is one of the challenges compliance and policy managers have to face. One of the common reasons is companies operate the whole process in a non-integrated and siloed way. So whenever there is a change in control, policies, or regulations, compliance managers fail to propagate (or enforce) policy changes into software delivery in real-time.

Given the pace of software development and delivery, automation for handling an array of compliance and policy enforcement, and real-time visibility for audit & self-service must be brought to the mix. 

Introducing OpsMx Enterprise for Spinnaker (OES) for Making CI/CD 100% Compliant

To overcome the above challenges, OpsMx Enterprise for Spinnaker offers Continuous Compliance and Audit. You can ensure compliance of your CI/CD process with industry standards and organizational policies while shipping your code, upgrades, application to production quickly. Continuous Compliance and Audit lets you track compliance requirements during the following stages of the software lifecycle- Build, Test, and Deploy.

Moreover, you can now quickly identify the who, what, when, where, and how of CI/CD pipelines and application deployment through audit reports and traces.

Continuous Compliance and Audit comes with the following compelling capabilities:

1. Configuring gates and guardrails through OES Policy Management

   The policy management feature of OES Continuous Compliance allows the security and compliance team to automatically express policy (in a declarative language) that can promote safe and fine-grained controls on Spinnaker deployment pipelines. The compliance team can better control the software development lifecycle with the automated implementation of restrictions and best practices.

   Fig 2A: OES Policy Management page where compliance managers can quickly declare policies and integrate with 3rd party policy managers for validations.

   

   OES Policy Management for CI/CD

2. Defining static and dynamic policies

   With OpsMx Enterprise for Spinnaker, compliance and policy managers will have the flexibility to declare static and dynamic policies. Policies that need to be adhered to when creating a CD pipeline in Spinnaker are called Static policies, e.g., a testing stage should be configured before the deployment stage. And policies that must be adhered to while executing a Spinnaker pipeline are called Dynamic pipelines, e.g. deployment into all European Geo should happen over weekends. In the runtime, policies are validated through 3rd party policy engines ( like Open Policy Agent) using API. Moreover, compliance managers get the flexibility to quickly add, modify, delete policies in tune with business policy changes. (Watch the 3 mins video to learn how to enforce policies into CD pipeline and ensure compliance)

    

   OES policy enforcement

3. Compliance standards

    OES allows DevOps managers to mitigate risks by enforcing compliance policies like SOX, HIPAA, at any stage of the software delivery pipeline.

   

   Compliance ensured in Spinnaker CD Pipeline

4. Auditability and Traceability

   OpsMx Enterprise for Spinnaker (OES) acts as a single source of truth for finding pipeline information by integrating with various sources- Spinnaker, Autopilot, monitoring tools, service management tools, and policy management tools (the complete list of tools here). Auditor using OES can obtain information at their fingertips, e.g.

• List of pipelines created and executed for various software delivery along with the dates.
• Status of pipelines run in a given time interval.
• List of terminated pipelines that were accountable for deploying business-critical applications in a time frame.
• Any unauthorized person behind failed pipeline execution.
• Lists of deployments happened into specific clusters during peak business hours violating policies.

Fig 3A: Examples of finding the status of the Spinnaker pipeline execution in an environment using Audit filters.

Audit filters for CD Pipeline

Spinnaker pipeline execution status

OpsMx Enterprise for Spinnaker(OES) Integrations with 3rd party tools

1. Spinnaker integrations with 3rd Party CI/CD tools

   OES offers out-of-the-box integrations with 3rd Party tools like Gitlab, New Relic, App Dynamic, ServiceNow, Sonarqube (click here to see all the integrations) to consolidate and compile information and automatically impose compliance rules and governance.
   

   

   OES integration with 3rd Party CI/CD tools

    

2. Alerts and Notifications in CI/CD process
   OES also provides support for sending alerts and notifications of any policy violation or non-compliance event through various channels such as email, Slack, HipChat, or SMS (via Twilio).
   
   

   Alerts and Notifications of software delivery

Benefits of using OES for Compliance and Audit in CI/CD pipeline

Operational Benefits: 

1. 24*7 assurance of continuous delivery process being compliant
2. Improved Efficiency:
   1. At least 50% effort reduction to enforce, change and modify the CI/CD process policies.
   2. At least 80% effort reduction to gather documents, track status for auditing and investigation purposes.
3. Boost Developer’s Confidence: Automated policy controls during the deployment process increase engineers’ confidence about not introducing any problems into production.
4. Better Visibility and Insights: Real-time visibility into compliance activities gives compliance managers and auditors instant access to the status. Auditors also get to identify and prioritize non-compliant activities before they can impact the business.

Business Values: 

1. Deliver Software Faster and Safer: DevOps team can release software as per the business schedule while 100% complying with governance and standards.
2. Cost Saving: Save at least $200,000 annually with automated policies enforcement in the software delivery process, proactive notification for policy violations, and real-time audit dashboard.
3. Enhanced DevOps experience: Self-service workflows to enforce static and dynamic policies in CD pipelines make compliance considerably easy. Moreover, timely alerts and notifications make them more proactive rather than reactive to non-compliance. Similarly, a real-time dashboard harmonizes the document gathering process, freeing Auditors up for high-value audit activities and decision-making rather than annoying admin tasks.

References:

1. State of DevOps 2019 by Google Cloud and DORA
2. State of DevOps 2019 by Puppet, CircleCI and Splunk
3. IT Audit Manual by Grant Thornton
4. DevOps Automated Governance Reference Architecture
5. Building Continuous Compliance into DevOps by Twain Taylor

OpsMx is a leading provider of Continuous Delivery solutions that help Fortune 500 companies safely deliver software at scale and without human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us