From Measurement to Insight: Put DevOps Metrics to Work

Source – informationweek.com

When you can measure the results of your DevOps program, that’s where the magic happens.

Working in the software delivery industry today is not for the faint of heart. Blink and your competition have already passed you by with a product delivered that much faster and better. However, it is possible to stay on pace with even the best in the business by adopting the best practices and principles associated with DevOps.

While there are many keys to a successful transformation journey, arguably one of the most critical is creating feedback loops through measurement. Unfortunately, measurement and data collection methods easily become time-consuming – in a world where DevOps is pushing speed – and, as a result, cycle and response times may suffer. The good news is that there are ways to turn metrics into action that will optimize the deployment pipeline to produce real-time results.

I recently sat down with Dr. Nicole Forsgren, CEO and Chief Scientist at DORA and co-author of the book “Accelerate.” Together we discussed the importance of DevOps and Agile, metrics that matter to the business, and how data-driven decisions can guide continuous improvement efforts. Here is an overview of our key conversation points:

Why measure?

Let’s break down the importance of metrics. From my perspective, DevOps is really the art and science of continuous improvement. Organizations are all trying to deliver better software but dealing with high costs in terms of risk and time to release. Today, many of these processes are operated manually, using non-integrated tool chains or islands of automation that are separated by wide slots of manual hand-offs.

This is where metrics come in. In order to understand where you want to go, you need to understand where you are at right now, and the only way to do that is to measure. It is critical to start collecting baseline metrics off the bat – whether its release time or mean time to recovery (MTTR) – to statistically visualize where improvements are being made and where bottlenecks may still exist. The metrics that matter most might vary from company to company, but the right ones for your company will focus on what brings true business value. Nicole also shared her thoughts on this topic:

Nicole: I’ve been lead investigator on the State of DevOps report for the last four years. What I’ve noticed is that the industry continues to change, it continues to evolve. We’ve been getting better year, over year, over year, and what’s important is that we continuously improve. We capture metrics in an effort to strive to get better every year and not just set our sights on a destination. Because if we just try to point to a destination, once we arrive at that destination and we stop, the rest of the industry will continue to get better and they will pass us by.

It’s super important to embrace the right metrics and move forward because even a bad baseline can be powerful, because it can be used to communicate the importance of resource investment. Even if we suck at something, that’s fine. Embrace the bad…

Always keep metrics in mind as you’re fixing problems in order to learn how to improve and grow, even if those improvements are small or don’t work at first.

Challenges with metrics

The rapid rise of automation has certainly been a double-edged sword. While it’s helping organizations keep up with digital transformation and quickening software delivery paces, it also means that bugs and other errors are harder to catch and can be sent to production more easily. For example, you might fix a bug in one line of code, but that particular bug might appear in 14 other lines of code, add in automation and the end product can leave customers unsatisfied. Even something as deceptively simple on the surface as MTTR [Mean Time to Repair] actually turns out to be a fairly robust metric to measure when automation is thrown into the mix. This is where establishing metrics and goals upfront can play a critical role in catching potentially dangerous errors from being deployed into production. Metrics help teams along the software delivery pipeline understand what is important to the business, aligns them to the same goals and understand where bottlenecks exist. Nicole added her thoughts, citing challenges with automation.

Metrics that matter

We say “everything as code” often at Electric Cloud. Infrastructure as code. Pipeline as code. Testing as code. You don’t have snowflake environments. Everything is change managed. Self-service automation tends to be a big one. That’s a really big one for idle time. In our experience, a big chunk of idle time comes from waiting for environments.

I think, the idea that we’ll have lots of people doing manual testing, that time has come and passed. I think we all need to be more test writers and test runners these days because the test run will get automated pretty soon… it’s automate, automate, automate when it comes to all the things that I like to talk about. Nicole also shared her perspective on metrics:

Nicole: Implementing system-based metrics and survey-based metrics together can be a nice complement. They complement each other. So what you can do is use people to give you insight into how your systems are running.

People can tell us if we’re not using version control. People can tell us how often we deploy. People can tell us approximately what our MTTR is, right? People can give us reasonable insights to the particular health status of our deployment pipeline. So we can use people-based, survey-based measures. It can range from non-scientific to advanced scientific measures. DORA does have scientific survey-based measures, but then we can use those to help fill out and complement our system-based measures.

The heart of DevOps and Agile is fast feedback. Measuring your pipeline and collecting information about performance and quality is great, but is time consuming. This means longer cycle times and slower responses are inevitable. However, the magic happens when you can turn that telemetry into actionable insights in order to optimize end-to-end flow. Just remember: When embarking upon the journey to better measurement, you first have to have an idea of what questions you want to answer before you dive head first into the data sets.