Did You Forget the Ops in DevOps?

Source:-infoq.com

The core premise of devops was for devs and ops to collaborate to speed up both parties
Using the same toolstack between dev and ops creates empathy and shared language
“Operations” is an overloaded concept, organizations need to understand what it means from both business and technical perspectives before embarking on a devops transformation
Although great technology, containers widened the gap between teams as well
Ignoring the Ops people in a devops transformation is a recipe for failure
Back in the early days, when we ran the first devopsdays events in Europe it felt like one part of the community was filled mostly with people from an operations background talking about how to improve their quality of life by reducing downtime, accelerating deployments, and improving platform stability.

What follows is a short story (with real war stories) of the evolution of devops in the last 10 years, achievements and shortcomings included.

Innovators Embraced Dev and Ops Collaboration
Back in 2009, many of us had been in situations where development teams had thrown unusable artifacts over the wall and expected the operations folks to just deal with it. We were tired of dealing with the side effects of development teams ignoring everything that needed to happen after they committed their code to version control.

RELATED VENDOR CONTENT
Reduce cycle time to remain competitive and meet customer needs
Mastering Remote Meetings – Download the InfoQ eMag
Free Scrum Master Learning Path
The Data that Engineering Leaders Need to Drive Continuous Delivery
The Engineering Leader’s Guide to Cycle Time: Accelerating Software Delivery with Data.
RELATED SPONSOR

GitLab is a single application for the entire software development lifecycle. Try GitLab for free.

At the same time, old school manual operations were not scaling anymore for our needs. Organizations were changing from having a couple of physical servers to managing hundreds of virtual machines thanks to virtualization. Topics we discussed at the early devopsdays included infrastructure-as-code, build automation, monitoring and metrics improvements and how to leverage the power of Agile, open source and cloud.

On the other hand, another part of the community was filled with developers that wanted to go faster and looked at the Ops folks as the bottleneck. They needed better platforms, faster response times, and ease of scalability.

Both groups realized that collaboration was the way forward. Developers would not get the platforms they wanted without discussing their needs with ops folks upfront. Simultaneously, ops would not get artifacts that were easy to operate in production if they didn’t discuss with the developers their requirements upfront. Therefore silos were burned, bridges were built, and people with different skills from all over the organization started to work together in cross functional teams. Software delivery became more continuous and smoother for everyone. devops had arrived.

Early Adopters Changed Their Ways of Working
Then people started to look at those early successful devops practitioners and wondered why can’t we work like them, why can’t we deliver faster, why can’t we build a more stable platform.

The first “devops transformations” started happening. Large organizations wanted to be like the “cool” kids. Some realized that agility and speed of delivery were now crucial in their line of business. Others just wanted to be able to hire all the cool kids that were already “doing devops”.

These large organizations had taken the first step by realizing they needed to evolve. But change is hard, and many times claims of successful “devops transformation” didn’t live up to the expectations. Nothing fundamentally changed for the teams on the ground doing the work.

Each organization has its own challenges, its own history. The path they need to take on their journey to happiness will be different for each of them. However, what always amazed me is to see many organizations – still today – claiming that they want to change but don’t include the operations people in their transformation plans. How can they ever “do devops” when leaving out operations folks from the conversations? Where is the Ops in their devops transformation?

One of the first engagements I took helping an organization to become “more devops” was a 150-people SaaS platform company that had serious issues deploying to production and keeping their stack up and running. They had grown at a high pace and had lost control over both their infrastructure and their deployments. It was chaos and everybody was pointing fingers at everybody else. The first thing we did in that journey was to teach the operations people about infrastructure-as-code and continuous delivery. We knew that later in the process the operations team would become responsible for supporting developers in achieving continuous delivery of the business applications.

Because operations folks now understood the basics of CI/CD and because they started using the same tools as developers, a shared understanding emerged. Operations now better understood what was required to deliver: how an artifact should look like, what stages were involved in a deployment, and how developers tooling of choice could help. The gap between developers and operations was closed, or at least seriously narrowed down. Both sides were now talking the same language, using the same tools, and taken significant steps towards a (better) culture of collaboration and understanding.

Early Majority Identified Constraints
One of the next large scale transformations that I witnessed did not include operations folks. In this organization it wasn’t even clear what operations meant from a devops perspective. Some people wrote code and other people “operated” the application from a business point of view (for example, adding new products in the catalog). Putting those two skill sets together in a single team was devops for them, but there was still no one looking after the highly technical operational aspects. It took eighteen months until I finally encountered an engineer whom I would qualify as an operations person from a technical point of view.

This person with deep operational knowledge was “too busy” fighting fires in production environments, and had not been included in the devops transformation conversations for this large organization. He worked for a different legal entity in a different building, despite being part of the same group, and he was about to leave due to lack of motivation. Yet the organization was claiming to do “devops”.

The action we took in this case was to take offline a number of experts who were effectively bottlenecks to the flow of work (if you’ve read the book “The Phoenix Project” you will recognize the “Brent” character here). We asked them to build the new components they needed with infrastructure-as-code under a Scrum approach. We even took them to a different city so they wouldn’t get disturbed by their regular coworkers. After a couple of months, they rejoined their previous teams but now had a totally new approach of working. Even the oldest Unix sysadmin had now become an agile evangelist that preached infrastructure as code rather than manually hot fixing production.

Late Majority Overly Focused on Tooling
A recent coaching engagement I took was for a large financial institution. I’d been pulled in to do a health check on their devops transformation, and I quickly heard a number of the devops coaches mentioning that they were not allowed to talk to the operations people. They told me operations was run by a different company that was not involved in the transition to devops.

I confronted senior management and their responded that their hands were tied, as this was an irreversible decision by Headquarters which is located in a different country. Their patchy solution was to build a “devops team” in the middle that didn’t have any operational or development experience, or even access to production environments. This new team was going to set up a continuous delivery pipeline for the whole organization to allow development teams to deploy all the way up to a test environment from where the operations people would take over and manually deploy everything.

The end result of this approach was a tooling stack that nobody used because it didn’t fit the needs of the developers. Plus they still needed to create manual tickets for the operations people to deploy their services. Many people decided to leave that organisation, including the local senior management who understood they would not be able to implement a real devops transformation.

The lessons I learned from these (and other) transitions is that you need to go beyond tooling and really include everybody in this process. If you don’t bring Ops along in your transformation, then you’ll never bridge that Dev and Ops gap where time is wasted and people become disengaged.

“Works on my tool, Ops problem now”
Issues around devops adoption are not always purely organizational, often there are also tooling issues. Not all tools are equally beneficial.

These days I see many organizations claiming they are doing devops because they’ve implemented tool X. When we take a closer look, tool X often will make life easy for developers but comes at the cost of hiding the operational complexity. Internal operations teams who typically have been left out of the discussion on how they would need to manage tool X get frustrated again. Once again we enter the “Dev vs Ops” mindset where development teams (and management) claim that Operations people are blocking them. Tool X might be extremely simple to use for them, but they ignore the monitoring, security, backups, scaling, metrics, networking and many other non-functional requirements that the Operations team now needs to address.

The number of tools that matches this painful pattern is sadly growing. In these 10 years of devops we’ve gone from “works on my machine, ops problem now” to “works in my tool, ops problem now”. I call that Dev-oops, not devops. Other folks call it Cloud Naive.

Giving operations people the same tool set as developers and letting them use the same patterns provides a common language, a way to better understand each other. The right use of a tool can be a fertile base to improve collaboration in an organisation. Yet, if tooling choices are made in isolation and operated in isolation by different teams, then it can drive a bigger gap between those teams.

Conclusion
Despite all the progress, devops in many organizations is still in its infancy. Sorting out how to improve development quality, introducing core practices like trunk-based development, test scenarios, and so on.

The fundamental devops premise of collaboration between Dev and Ops, of aligning their different skill sets to deliver and run quickly and safely is still often being ignored in many organizations.

So after 10 years of saying that everybody should be included, you’d think that any devops transformation would include not just Dev but also Ops in the discussion and roadmap. Sadly that still isn’t the truth.