Let’s face it: Agile & DevOps is a fact of life. It is the super glue that binds everything together and Non-Functional Testing (NFT) is right at the centre when we talk about scalability, availability, security, accessibility and predictability. Without having an automated NFT strategy, we cannot catch up with DevOps.
With the increased adoption of Agile + DevOps, the real question is what level of automation should a team achieve before they position themselves as Mature DevOps Organization. In reality, despite performing tasks manually, teams do declare that 100% DevOps principles are followed by their teams.
As quality assurance is being integrated into the sprints, continuous testing is the only way forward. Alternatively, QA will be considered a roadblock to the success of Agile DevOps adoption. To achieve the continuous testing objective, QA organizations have mostly focused their efforts on scaling up functional test design automation, test execution automation, and integrating these with the CI-CD pipeline.
One key area that continues to be executed out of this pipeline is Non-Functional Testing. Now, the main question that arises is - how can Non-functional Testing be aligned in DevOps pipeline?
As a matter of fact, teams should analyze and answer few basic questions before they can even embark or think of integrating QA into Agile DevOps journey:
- How much process and DevOps test automation can be done?
- What is to be done with manual tasks?
- Where to start and what to automate in continuous testing towards non-functional aspect?
Teams need to ideate, strategize, create, and implement solutions that will help implement zero-touch automation to integrate non-functional testing in DevOps pipeline.
Envision the following steps that gets triggered when new code for an existing application gets checked in:
- Relevant code gets checked in
- Automated code scans are performed for security and performance analysis
- Build gets deployed to DEV for automated unit tests
- Automated scans get triggered to identify code changes and to flag impacted test cases which is then moved to regression test suite
- Upon success, build gets deployed to QA1 for automated functional regression and early performance checks
- Auto alerts and notifications on each process
- Functional scripts are auto migrated to load scripts for performance testing
- Upon success, build gets deployed to QA2 for running performance tests
- Automated dynamic application security scans and accessibility evaluations are performed and flagged for PROD deployment upon success
- Provides eligibility to pull out historic test results for analysis
- Code gets deployed to PROD and monitoring and alerts get triggered for continuous monitoring
Some of these activities are typically being performed manually. To facilitate a real end-to-end automation including non-functional testing in a DevOps world, teams need to focus on developing specific point solutions to avoid manual interventions. These developments could be even in the form of bots. (e.g.) having a bot to do result trend analysis of performance testing in a continuous testing cycle.
When building the solutions, it is essential to consider the below elements:
- Auto discovery: Integration to requirement changes, code changes, test needs etc.
- Single asset for multiple needs: When the asset is executed once, it captures all NFT metrics
- Reusability of test assets: Build flexible and scalable assets for reuse across sprints and releases
- Measurability, Tracking and Optimization: In terms of metrics, feedbacks, monitoring etc.
- Tool, Technology, and Resource agnostic
HCL has been focusing on building relevant solutions to address white spaces where current market tools don’t support in meeting the end objective.
DevOps is a journey. There will always be ways to improve and ideate new techniques to adopt and better collaboration techniques to implement, and so on. Getting organizations on-board with such an endeavour can be a whole journey in itself.