Getting started with CI/CD: 6 pitfalls to avoid

Source- enterprisersproject.com

What shapes the long-term success of your CI/CD effort? A faster, more automated pipeline for software development? We recently outlined 4 success factors when getting started with CI/CD – and all led back to culture.

That’s helpful, but it’s also productive to look at the downsides – not of CI/CD itself, but of common mistakes organizations make, especially when they’re just starting out.

“If done right, proper continuous delivery and integration can lead to great customer experiences,” says Joy Scharmen, director of DevOps at Optimizely.

In the interest of doing CI/CD right, we asked Scharmen and other IT leaders to share some of the common ways teams go wrong. Here’s what they had to say.

1. Don’t follow the recipe for tool salad

I could walk into a Home Depot today and buy a bunch of expensive tools and materials; I’d walk out still not knowing how to build a new back deck on my house. Thankfully, I have just enough common sense to not spend money loading up my garage with a bunch of stuff that I don’t really understand.

IT teams need to apply the same common sense when determining the right tools and technologies for their CI/CD pipelines, or they risk ending up with what Nate Berent-Spillson, senior director of delivery at Nexient, calls “tool salad.” (Mmm, sounds delicious.)

“I’ve seen companies go on a buying spree, picking up this tool, that platform, and that system over there, yet in the end, they weren’t equipped to build an iOS app,” Berent-Spillson says. “The unspoken idea was that if they bought enough tools, the capabilities they wanted would have to be in there somewhere.”

Choosing the right platforms and tools is indeed an important part of your CI/CD plan, but your needs should dictate the tools, not vice versa.

“Instead of thinking about what tools you need, think first about the capabilities you want, like ‘build pipeline for iOS,’ ‘build pipeline for Android,’ ‘mobile app distribution for both platforms,’ or ‘signing certificate security and management,’” Berent-Spillson advises. “That way, you’re focusing on how to move value through the system. The choice of tools will follow from that.”

2. Don’t mess up on metrics and feedback

Continuous measurement is an important success factor for CI/CD, but it comes with a downside: Picking the wrong things to measure.

“One of the biggest pitfalls to watch out for is picking the wrong metrics to test,” Scharmen at Optimizely says. “Businesses can easily guide their products down the wrong direction if they choose the wrong metrics to track and measure.”

Because CI/CD and DevOps often go hand-in-hand, there can be some overlap in the metrics you use to evaluate their impacts. The key principle here: Track what actually matters to your organization. As Gordon Haff, technology evangelist, Red Hat, wrote previously about DevOps metrics: “A metric is properly thought of as a key performance indicator for data – a measurement that’s important to you in some fundamental way.”

The same idea applies to your CI/CD pipeline: Make sure you know what’s important, not just to you but to the organization.

“Different members of an IT organization or team have different priorities, so it’s important to assess which data everyone wants and needs from the start,” Scharmen says.

Scharmen notes another related pitfall: Getting duped by data via misinterpreting it, confirmation bias, or other reasons. This applies not only to your ongoing measurement but to testing itself in your pipeline.

“Misinterpreting a test can be worse than not running one at all and can quickly negate the value of an experimentation by injecting uncertainty and fear into a process that’s meant to provide clarity and confidence,” Scharmen says.

Finally, don’t forget about your actual production application or service. CI/CD doesn’t actually come with a defect-free guarantee as code moves into production. You still need a feedback loop there.

“Avoid the pitfall of having automated the delivery of software from source code to end user but not automating the collection of live, passive feedback, either from end users or automated tests,” says Maxwell Bloch, consultant at Applied Information Sciences (AIS). “For example, instrument your application to provide telemetry and error occurrences. Many bugs will be detected earlier in the CI/CD process, but some bugs will only occur with real usage.”

3. Don’t give people a way to bypass the CI/CD pipeline

Don’t enable team members to bypass the pipeline other than under the most extraordinary of circumstances. This can be particularly important if you’re dealing with skepticism or cultural stasis in the earlier stages of your shift to CI/CD.

“If your CI/CD platform is not the only way, excluding an emergency break-glass method, to deploy code to your production environment, it will become tempting for teams to work around it when your testing [finds] code issues that are not easily solvable,” says Jerry Gamblin, principal security engineer at Kenna Security.

A promise of CI/CD is that it will enable application teams to move faster than before, but there may still be real reasons to slow down, and that can cause some frustration or feelings of failure. (This is another reason why cultural health is as important as technical decisions and architecture: A healthy culture can handle the speed bumps.)

You might automate your pipeline iteratively or in small chunks, and that’s fine. Similarly, you might build a CI/CD pipeline for a specific application team – also fine. But no matter your approach, don’t effectively give people the chance to opt out of your CI/CD pipeline if they decide it doesn’t suit them for some reason.

4. Don’t treat automation as an infallible hero

Automation is intrinsic to CI/CD; a CI/CD pipeline laden with cumbersome manual processes isn’t much of a CI/CD pipeline.

“CI/CD is all about automation,” says Meera Rao, senior principal consultant at Synopsys Software Integrity Group.

That doesn’t mean “automate all the things” is literally the goal, though. Rao notes that while automated testing is an enormous part of CI/CD, it doesn’t actually eliminate the need for manual testing or other forms of human intervention.

“No single automated testing tool can find all the defects in an application. For example, static analysis tools cannot understand business logic,” Rao says. “Likewise, no automated security tool is effective in discovering architecture and design flaws in deployed applications. The only effective way for an organization to discover these types of flaws is to perform a manual architecture risk analysis.”

It’s a later-stage byproduct of Berent-Spillson’s tool salad: Assuming your technology will take care of everything almost certainly means there will be issues in your pipeline and, as a result, in your production code.

“On top of automation and continuous testing, you must make sure to have intelligence built within your CI/CD or DevOps pipeline to know when human intervention is required,” Rao says. “Recognize that tools need hand-holding.”

5. Don’t shortchange your CI environment

A truly robust CI/CD pipeline is the sum of its parts; if you skimp on the integration piece, that can have compounding effects later,  especially when code deploys to production.

“Ideally, the continuous integration environment for a software product mimics the production environment as closely as possible,” Bloch says. “Continuous integration will not prevent end users from encountering bugs if the software was not able to be tested under realistic conditions.”

Bloch shares a handful of examples where discrepancies might occur between integration and production environments, increasing the likelihood of missed issues.

• The allocation of process resources such as CPU and RAM.
• The number of distributed servers running a web application.
• The scale of web traffic encountered during testing.
• When applied to mobile apps, Bloch says making your CI environment as production-like as possible “means that the most accurate environment for testing and continuous feedback is a physical device and not an emulator.”

6. Don’t silo development and testing

Just as DevOps success required the breakdown of traditional walls between development and operations, CI/CD success depends upon bringing dev and testing into closer alignment.

“Watch out for persisting the separation between development and testing, even after automating the integration of code and the delivery of the latest version,” Bloch says. “If development reaches a checkpoint where a CI build occurs, the developers are not separate from the next steps in the process. If bugs are identified, developers must respond quickly and provide the fix into the CI process, or else the build may not be considered releasable.”

You don’t need to start doing trust falls with your developers and testing pros; rather, you need to start eliminating conflicting goals and incentives.

“The ability to deliver continuously is jeopardized if development and testing activities do not share priorities,” Bloch says.