IT ops pros adopt iterative approach to security in DevOps

Source:- searchitoperations.techtarget.com

Baby steps that add security in DevOps environments are better than none at all, according to experienced IT ops practitioners.

Most organizations have dedicated IT security departments or personnel, but lack of a mature DevSecOps collaboration means IT ops ends up on the front lines to identify vulnerabilities and anomalies in production applications. Like it or not, security in DevOps environments is often left up to them.

Attacks are often unmasked because they show up as a problem affecting performance of systems, operating systems or browsers.

“This goes back to the old days, when people would call the help desk and complain that their machine was running slow and it was being weird,” said Chris Christiansen, senior consultant at Hurwitz & Associates, a consulting, market research and analyst firm in Needham, Mass.

Unlike the old days, however, the increased speed of DevOps can deepen the IT fog of war.

Security in DevOps is an evolution, not a revolution

IT ops teams must walk before they run with security in DevOps shops, as well as think creatively about how security and IT ops tools can fit together and benefit one another.

One healthcare data analytics company in its early years managed secrets with a homegrown app, which wasn’t ideal, but it effectively prevented security breaches.

“It was a golang [Go language] binary that was written around [Amazon Key Management Service (KMS)],” said William Bengtson, senior security program manager at Nuna Inc. in San Francisco. “Every service would have a unique KMS key, and we’d encrypt static secrets and store them in a KMS-specific YAML file.”

Nuna now uses HashiCorp’s Vault, but still runs the homegrown KMS app in the environment because Vault can’t secure itself.

Bengtson has also seen firsthand how IT operations personnel end up on the front lines of security in DevOps deployment pipelines.

When the firm first adopted Vault, it struggled with the need to refactor and redeploy applications to add new security features. IT ops led the charge to bake Vault’s service deployment feature into the process used to spin up new Amazon Web Services (AWS) Elastic Compute Cloud instances, which enabled the addition of new security services without having to refactor applications. This also paved the way for the company’s DevOps personnel to take on more security improvements. IT ops also found ways to use Vault’s infrastructure monitoring to proactively test application behaviors.

If Bengtson’s team wants to understand how a tool operates from a security perspective, they issue the tool a set of dynamic credentials and have it connect with MySQL, then revoke those credentials and see if the tool continues to operate, Bengtson said. “From this, we can understand: If we revoke a database credential in the event of a compromise, would we need to do anything else?”

Back to basics for security in DevOps

While tech companies like Nuna can script and program their way to better security before they deploy a complex DevOps security tool, not every IT shop has such resources at its disposal.

In these cases, progress is still possible, according to a presentation given this month in Boston at a DevOps Meetup by Tom McLaughlin, an engineer at Threat Stack Inc., a cloud security software company in Boston.

“I am not a ‘security person,'” McLaughlin said. “I’m an ops person that has had to deal with security throughout my career, because there either was no security team or security came to me saying, ‘You need to do this.'”

In some ways, this requires a mental shift to identify banal errors that more likely will have major security consequences for a company than the latest zero-day exploit with a catchy name, McLaughlin said. Such boring, but dangerous security errors include systems with weak authentication credentials exposed to the open internet or security keys stored into code repositories in cleartext.

McLaughlin also offered pointers on simple ways IT ops pros can improve security when they lack the resources to write a complex secrets management utility themselves and also aren’t ready for a tool like HashiCorp’s Vault or the identity and access management platform Conjur.

One simple approach to shore up security in DevOps is an open source utility called git-crypt, which encrypts secrets directly in development repositories through symmetric encryption — i.e., shared passwords. “At the very least, you’ve now just gone through your code and found all the dead bodies … you’ve gone from the worst possible [scenario] to at least a step forward,” McLaughlin said.

Beyond that, tools such as Puppet have utilities like Hiera-eyaml that can provide public key encryption without shared passwords in one centralized repository. However, such tools may require manual intervention when rolling new Puppet masters, and IT ops pros may need to clean up Puppet code if they haven’t already moved to Hiera.

Finally, utilities such as Sneaker and credstash can encrypt, store and retrieve secrets from Amazon Simple Storage Service (S3) buckets. With this approach, secrets no longer live in the app’s code repository and are encrypted in S3. Credstash has integration with KMS. However, this requires IT pros to move on to a tool such as AWS CloudFormation or HashiCorp Terraform to manage permissions for S3 buckets, McLaughlin said.

IT ops testing, monitoring and deployment tools can also be brought to bear on security in DevOps environments, Christiansen said, provided the IT ops team can recognize the security ramifications of anomalies found by those tools.

“IT operations and security are more strongly related than most people realize,” he said. “But when you have a security issue that directly relates to performance, throughput and uptime, there can be a meeting of the minds.”