How to maintain software quality when scaling DevOps

When there is success in these endeavors, CTOs and other technology leaders start to think, Hey, why not expand DevOps even into business-critical areas where we need rapid response and the work of our best team members? On balance, this can be a great journey, yet it all depends upon the application’s user requirements. If DevOps is applied to a revenue-generating application and results in downtime or critical errors affecting users for a few hours, you can probably expect a visit from the CEO.

How to ensure quality in DevOps remains a hairy problem. Regardless, once you begin to scale DevOps throughout your organization, you will need to invest more energy (time and staff) into the quality equation.

Unless you work for Google, Netflix, Amazon, Facebook, Microsoft, or a few others, you’re not likely to have a huge staff, with all the right skills, and an unlimited budget to spend on perfecting DevOps at scale. These companies are on the cutting edge of cloud and software development and are inventing some of the technologies and best practices being used everywhere today.

For the rest of us, it’s time to get scrappy and work smart.

Here’s one pathway:

1. Controlled rollouts when a little failure is okay

If your company is able and willing to accept some risk of breakage, do what companies such as Facebook are trying: roll out new updates gradually to a small percentage of users. Then you monitor production and support calls to watch for issues. Once those are cleared up, you can begin releasing the update to larger sets of your users. What’s important here, though, is that you have a large user base so that the sample size can effectively weed out any issues in a timely manner. In other words, if you are a company such as Facebook with tens of millions of users and you do your first rollout to 2% of users, you’ll have a healthy sample for testing the update.

When the stakes are higher, here’s an alternative pathway:

2. Your app needs strong QA, so learn to do it at the speed of DevOps

It’s hard to imagine a bank, insurance company, healthcare organization, or government agency being okay with the controlled rollout strategy for certain applications. A trading application that is down for a minute can result in millions of dollars of losses and lawsuits. A glitchy remote monitoring application for a chronic disease patient could be giving the doctor’s office bad information, leading to a bad decision for the patient. That’s not workable. So instead, you invest more up front in testing and, specifically, test automation. That enables an organization with limited staff to conduct comprehensive testing quickly and efficiently. Yet this all depends on whether the testing can be accurately and quickly tracked back to the business requirements and the code that has changed.

QA planning and tracking are key

In DevOps, test automation should be the norm, with manual testing done sporadically and only when the use case dictates. Young companies or startup divisions in large companies that jump into DevOps will find that there are many test automation tools and frameworks available, from open source to commercial. Open-source automation test frameworksare easy to use and, of course, free, and many teams will start there. As your needs mature, commercial tools offer features to make it faster to set up and run the tests, as well as better features for collaboration.

Yet having a framework for test automation is not everything. There’s also the need to test quality assurance: Did we test the feature well enough to achieve the goal or fix the problem? Developers might see a 99% pass rate and think all is good. Yet if the 1% of tests that didn’t pass relate to a core user function: not good. Whether you do this in-house or with third-party technologies, make sure you have the means to correlate test coverage with business requirements and outcomes, not just with code.

QA takes a team

This is also where having a solid QA team in place makes a big difference. In DevOps, with so much focus on automation through continuous integration tools, some might think that there’s less of a need for qualityspecialists. Yet that couldn’t be further from the truth, particularly given the challenges of maintaining high-quality releases within fast development environments. The role of the QA team is to ensure that the application is receiving comprehensive testing in the areas that matter most, identifying all opportunities for automated tests and determining how to handle manual testing needs.

When your organization decides to make DevOps a mainstream versus an experimental approach to software development, consider that the basics of good development still apply: a clear definition of requirements and a process to ensure that the software has been developed to meet the specs, that tests have been written to cover the requirements, and that tools or processes are in place to measure the outcomes before and after release.

While each project or application has a different risk profile and unique needs for QA, this kind of discipline will help your organization go far in achieving consistent quality as you scale your DevOps practice.