Grow rapidly into a continuous delivery pipeline

“We see CI and CD as two different worlds,” said Steve Burton, DevOps evangelist at Harness. The pipeline engine, whether gated for CD or open for continuous deployment, must push tech code across infrastructures, verify deployments, roll back versions, manage secrets, provide auditability and handle other “unsexy stuff,” he said. Microservices migration is one of the prevalent customer use cases for Harness, because constant application updates on these independent pieces of code can expose brittle areas — such as poor connections that fail to deliver code in the proper format from one tool to the next — in a homegrown continuous delivery pipeline.

Create a safe deployment environment

To continuously deploy to live users, organizations must consider the quality of the code and visibility into each update’s effects.

Testing should be part of a CI/CD strategy, but test is never an exact replica of production. “You can’t replicate that scale, and you can’t put customer data into a [traditional] staging environment,” said James Freeman, head of professional services at Quru, a consultancy focused on open source technologies. Things test fine and pass to production, then they go live and fall over. “You’ve got to put good process behind deployments,” Freeman said in a presentation at AnsibleFest 2018 in Austin, Texas.

Ibotta uses blue/green deployment to handle the multitude of microservices updates per day. Blue and green setups mirror each other and trade off as staging and production environments. The team can quickly revert to a previous version of code without creating a bottleneck. The blue/green changeover currently serves as a gate between development/test and production. As the microservices count at Ibotta grows, Bassin plans to set up the CI pipeline to initiate deployment to production automatically via Harness.

Blue/green deployment is an effective method to quickly deploy and roll back production changes. But it is resource-intensive, and changes hit all users at once, Carroll said. It also requires tight configuration management so the two setups remain the same. “It’s reliable and great for back-end services,” he said. Canary releases are another option for a continuous deployment pipeline, well-suited for front-end UX features.

Canary deployment is so named for the early warning system of a canary in the coal mine. A small percentage of users experience the new code while the engineers monitor its performance and functionality before the entire deployment occurs. Ibotta plans to switch from blue/green deployment to canary releases eventually for its microservices. Canary deployment decreases the blast radius of problematic updates, Bassin explained.

In a project with U.K. retailer Sports Direct, Quru upgraded 60 nodes successfully on a Friday afternoon with automated Ansible playbooks, but it didn’t roll out all 60 nodes at once. Instead, it started with a single node to prove that the process was set up correctly and the configuration update worked as intended. If the canary continues to sing, deploy the remaining 59 nodes with confidence, and head out the door for the weekend, Freeman said.

Ibotta’s engineers deal with a cumbersome canary deployment process currently on the company’s monolithic application. Ibotta aims to change to canary deployments with microservices because it can make the changeover to new code faster than blue/green without staging and without the issues experienced on a monolithic architecture, Bassin said.

Carroll recommended that organizations model their operations toolchain and choose a deployment pattern that best fits the pipeline and objectives. Then, iterate on the CI/CD pipeline, with tests and post-deployment monitoring, until they achieve the target cycle time.

Hope for the best — expect the worst

Continuous delivery and deployment cannot succeed without a plan for failure. Even with rigorous control before code reaches production, things go wrong unexpectedly, and you cannot test for every possibility.