DevSecOps: How to conquer 3 big culture challenges

Source – enterprisersproject.com

Just about any DevOps shop will hit speed bumps on the path toward continuous learning and improvement.

“Organizations are increasingly adopting DevOps environments in hopes of achieving transformative velocity and innovation,” says Elizabeth Lawler, VP of DevOps security at CyberArk. “But like any new business initiative, this comes with challenges – and in the case of DevOps, it’s often around culture and areas of responsibility.”

Even issues that seem technical in nature are often rooted in people. Take security: It’s as much a matter of culture and areas of responsibility as it is a technology problem, Lawler says. And even high-functioning DevOps teams are encountering challenges making security – and security teams – integral to development.

“One of the biggest areas of friction remains bringing together security and development operations,” Lawler notes.

This friction is driving interest in the DevSecOps approach – and, almost paradoxically, fueling resistance to DevSecOps practice. IT teams face several significant, common cultural challenges with DevSecOps, and while they might manifest in different ways, they’re closely related – often doing their damage in unison.

We’ll identify a trio of these cultural roadblocks below. Then we’ll share strategies from IT leaders and security experts for proactively breaking down these challenges, as a necessary step toward DevSecOps success.

Culture challenge #1: An entrenched view of security as something that happens “later”

“The biggest change in mindset necessary to [create] a mature DevSecOps practice is to understand that security cannot be done as an afterthought,” says Premand Chandrasekaran, VP of software engineering at Barclaycard, a division of the global bank Barclays. “Rather, it requires the attitude that it is built into the continuous delivery pipeline.”

It’s a major – and necessary – change, due to the reality of how systems get built and operated today. Tim Jefferson, VP of public cloud at Barracuda Networks, points out that environments can be spun up and down so quickly these days that their lifespan might be a matter of hours. Security as a final step or afterthought is a non-starter.

“With the traditional model, the infrastructure was built, and the security audit was a post-mortem process,” Jefferson says. “Today, DevSecOps professionals are tasked with baking security into architecture as they build, and into the system as it’s deployed.”

Culture challenge #2: An “us-versus-them” mindset

Some of the friction isn’t new: Developers and security pros have often been at loggerheads with one another in the past, notes Meera Rao, senior principal consultant at Synopsys Software Integrity Group.

“The primary cause for this friction is that each team often has its own roadmap, responsibilities, and priorities – and completely different incentives,” Rao says.

Most devs and security practitioners reading this are probably nodding their heads.

“Security teams have long thought that developers were not interested in security,” says Franklin Mosely, senior application security engineer at PagerDuty. “Along those same lines, security teams also thought developers believed that security wasn’t their responsibility.”

Neither is necessarily true, Mosely points out. But the divide can remain evident even in relatively mature DevOps shops – if security remains as a separate entity, it’s effectively siloed, reinforcing an us-versus-them mindset.

Culture challenge #3: The belief that security hinders innovation

This belief has helped fuel the traditional conflict between dev and security teams, but it also speaks to a broader antipathy toward IT security, bred from treating security as an afterthought in the software pipeline. (See? We told you these issues often wreak havoc in unison.) As the demand for faster, more frequent delivery continues to grow, there remains a deep-seated view of security as something that slows everything down – the bane of a modern, transformative IT shop.

“We are all very used to this romantic image of the resourceful developer who codes swiftly and slickly – but not necessarily securely, because security has been viewed as the antithesis of function,” says Robert Hawk, privacy and security lead at xMatters. “Thus, a big cultural challenge for DevSecOps is to instill order in the face of a legacy of chaos, and somehow accomplish this without hindering innovation.”

This is again both a driving force behind DevSecOps – part of its raison d’etre –  and a fundamental cultural challenge.

“Speed, velocity, and resiliency do not need to be sacrificed in order to be secure and have more stakeholders at the table,” Lawler says.

Five routes to success

The solutions to these cultural problems require equal importance in two areas: Tweaking technical strategy and enabling better collaboration and organizational alignment. Let’s delve into five strategies for doing just that:

1. Have people walk in each other’s shoes

Ignorance is hardly bliss: When it comes to security, ignorance greatly increases risk. It also breeds misunderstandings and resentment, and that’s what you’re trying to eliminate. Create opportunities for people in different roles to better understand each other’s jobs; many DevOps shops already have experience doing this to create better empathy and alignment between devs and ops. A recurring example is requiring developers to be in the on-call rotation. The same principle applies to security.

“To make security a priority, both security and development teams must learn from each other,” says PagerDuty’s Mosely. “The security professional needs to walk in the developer’s shoes and learn how software is made and the issues that are faced. It’s best to work with developers to come up with solutions that allow them to do the right thing when it comes to security.”

Developers and DevOps pros similarly have a responsibility to work more closely with their security colleagues toward understanding best practices and tools for securing modern infrastructure and software.

“Developers should embrace that security teams have something important to offer in terms of best practices and know-how,” Lawler says. “And security teams should embrace new ways of working and sometimes learn a new lingo to work in the fast flow of modern software engineering. Security needs to be built into development processes early on, and that can only happen when security and development teams collaborate.”

Lawler notes that simply asking people to “do more” on this front is likely just throwing gas on a smoldering fire. So…

2. Give people real opportunities to learn and train

Many of the cultural challenges that can hinder DevSecOps – or any other name you want to give to building a more secure organization – can be attributed to a lack of know-how.

A dev who has never had to be directly responsible for the security of their code in the past, for example, probably hasn’t had much opportunity or incentive to learn best practices for secure application development. Similarly, a siloed security team might have little to no idea how the software they’re charged with securing actually gets built. Change that.

“Most developers aren’t trained in secure coding best practices,” Mosely says. “This creates a situation where engineers are not implementing security thinking and awareness into design, coding, and testing.”

Robust training and learning opportunities are key. Mosely shares two training programs from used at PagerDuty as examples.

“Training can be very effective in creating a mindset where security is more top of mind for our engineers, rather than an afterthought,” he says.

Training and learning can take many forms. Chandrasekaran of Barclaycard says regularly scheduled and unannounced security audits can create good opportunities for learning. He also recommends “hack days” or similar programs, such as a capture-the-flag event where cross-functional teams are encouraged to actively find and exploit vulnerabilities.

3. Consider embedding security pros on development or DevOps teams

If you want to double down on the first two strategies, consider embedding security pros on your development or DevOps teams. Virtually every expert included here spoke of the necessity of breaking down the silo between security and the rest of the team; literally giving security pros a seat at the table can do the job faster.

“IT leaders should create avenues for security experts to be involved in designing, developing and testing code,” says Kiran Chitturi, CTO architect at Sungard Availability Services. “One way is to embed security team members directly into development teams.

Mosely says they’ve used this approach to great effect at PagerDuty, having previously embedded a security engineer on a development team.

“During that time he learned their tools and processes, and the development team gained insight into some security best practices,” Mosely says. “With this knowledge, the security team is better able to assess how they can add value and minimize risk to the business without inhibiting the development processes. At the same time, the product team has a better understanding of things to consider to ensure that they are protecting customer data.”

4. Explore a “security champion” program

The embedded approach has a close relative: The security champion. This person need not actually be a security pro, but rather someone who can help drive healthier security habits that spread throughout the organization. (Of course, there should be some incentive or reward for a person who embraces this role, too.)

“Establishing a training and security champions program allows members of development teams to learn and volunteer to build software security skills and awareness through mentoring, training, and working closely with the application security teams,” Rao of Synopsys says. “These security champions form the front line when it comes to guiding development teams on application security and bridge the gap that exists between DevOps and security teams.”

5. Embrace automation – and win over skeptics

Sometimes cultural challenges can, in fact, be overcome with technology. In the case of DevSecOps, most experts agree that automation has myriad benefits.

From the standpoint of cultural resistance, increasing automation is crucial to enabling security to indeed “shift left” in the software pipeline without hindering speed or innovation – while also getting rid of the perception of security as a final step or afterthought.

Let’s address an issue that sometimes gets brushed under the rug: One reason cultural resistance to a change like DevSecOps exists is that at first blush, people see it as translating into net-new work that they don’t necessarily want to do. Asking developers, for example, to simply to “do more,” as Lawler noted above, could be perceived, often rightfully so, as simply adding tasks to their already overloaded plate.

Automation cuts down on the manual toil.

“A lot of verification needs to be automated through the use of a combination of static, runtime and chaos engineering tools to reduce the amount of grunt effort involved,” Chandrasekaran says.

Chitturi of Sungard AS likewise says that “bringing in an automation mindset to teams by introducing tools, techniques, and frameworks that aid in automatic code scanning for security-related checks and dynamic scanning and policing” has been a key strategy in his first-hand experience with overcoming cultural roadblocks to DevSecOps.

Automation also soothes individual fears that you’re going to have to learn all the aspects of security that may occur throughout the DevOps cycle – a daunting thought even for security pros.

Instead, IT leaders should think about how to derive and define security policies from the organization’s governance model, and then choose security experts with the relevant expertise to take responsibility for baking those policies into the various processes in the project or product lifecycle.

“It’s all about automation,” Mike Bursell, chief security architect at Red Hat, recently noted. “The developer, the tester, the designer, the operations person – [all] need to be able to have responsibility for following security-relevant processes, not defining them. Once you have the processes in place, you can manage by exception: Instead of trying to check that people do the right thing all of the time, spot when they do the wrong thing and sort it out from there.”