Validate Software Releases Through Application Behavior Analysis

Introduction

Enterprises are moving to DevOps model and also adopting agile software development methodologies to accelerate the software deliveries.  As the velocity of software development increases,  the need to speed up the testing and validation also arises.  Enterprises usually do some combination of the following to validate new releases –  functional testing, integration testing,  end-to-end system testing,  user acceptance testing, accessibility testing, load or stress testing, security test and staging environment validation. Another need of these organizations is to find the issues earlier in the testing  cycle as the cost of finding issues later or in production is much higher (Figure 1)

Figure 1: Cost Rises Exponentially for Defects Found in Later Stages/Production

In this blog, we will look at ways enterprises can leverage automatic in-depth application behavioral based analysis to validate releases quicker, find defects early and spend less time diagnosing the issues.

Need for Analyzing Application Behavior in Testing Stages

Typical enterprise Jenkins/CI pipeline has multiple stages (Figure 2) of testing for quality and validating the new releases to ensure that release has no user impacting critical issues and that the business SLAs could be met.

Figure 2: Typical Software Release Pipeline

The goal of the team performing each stage of testing is to ensure adequate correctness of the release to pass on to the next stage and to find issues that could have an impact in future stages.

In modern organizations,  most of the functional test is automated and are usually adequate even though the test coverage may not be 100%.   The most glaring gap is in the application run-time performance testing.  Typical current validations are performed by spot checking few metrics that are inadequate in reliably finding all the issues on a consistent basis over many releases.  Below is an actual scenario in an enterprise that reflects the challenges of identifying issues in testing stages.

“We perform load tests with 3 times the load that we see in production but we still don’t catch the issues that crops up in our production deployment”

– QA Manager at a Large Retailer, San Francisco

The solution to such scenarios is to perform a systematic analysis of the application behavior and identify degradations on all relevant metrics for any given applications.  Every service/application will have its unique fingerprint even though many services share the same/similar baseline software components. Also, the baseline behavior of the applications will change over multiple updates, and the analysis needs to adjust to those changes.  Monitoring only the metrics that identified the issues in the past may not be enough to capture future issues.  This is the crux of the problem on why current manual or static scripting techniques will fail over time to detect newer issues.

Automatic Multi-Dimensional Application Behavior Analysis

OpsMx is a machine learning release analytics bot that is designed to automatically perform multi-dimensional application behavior analysis for releases.  It validates releases through in-depth analysis based on 1000s of metrics and logs during various testing stages. OpsMx directly integrates into Jenkins through a plugin allowing it to provide validation input directly as part of each stage of the pipeline (Figure 3).

Figure 3: OpsMx Analytics Integration Into Jenkins Pipeline

OpsMx provides the risk assessment and diagnostics report for the build. The risk assessment report includes the safety score and the detailed diagnostics about the build. The safety scores for a given build signifies the readiness and suitability of that new build for the next stage.  The build is evaluated for architectural regression, performance, scalability, configuration and security violations. The detailed diagnostics part of the report clearly identifies safety divergences of the various sub-components of the build and flag any significant deviations as shown in Figure 4.  OpsMx analyzes interactions between various services and transactions to narrow down the problematic service in multi-service complex applications. OpsMx report also allows users to investigate trends on any specific component or metric for regression over time.

Figure 4: OpsMx Risk Assessment and Diagnostics for Testing Stages

Benefits of OpsMx Solution for Testing Stages

The OpsMx solution brings the following benefits to any organization leveraging automated application behavior based analysis and diagnostics:

• Reliably validate and approve good releases:  OpsMx provides the ability to reliably validate a release beyond existing automated functional testing and unreliable manual or script-based mechanisms.
• Identify issues with the releases early:  By enabling OpsMx in all testing stages,  enterprises can find issues early in their testing cycles.  OpsMx saves time and cost since the finding defects at later stages are very high.
• Diagnose root cause of issues quickly: OpsMx can not only identify issues but also provide detailed diagnostics including offending code commit responsible for the issue.  OpsMx can narrow down the problematic service in a complex multi-service application. The test engineer can collect all relevant logs/metrics identifying the issue and collaborate with the developer to fix the issue quickly.

For a free trial of the OpsMx solution for testing stages with Jenkins or other CI framework, fill out the below form or email us at info@opsmx.com.