Are DevOps and Application Security Compatible?

Source – simpleprogrammer.com

DevOps is the word of the year. The software industry is on fire with the DevOps craze and more and more companies are looking for DevOps skills.

DevOps practices are the key to delivering value quickly, scaling effectively, and enabling a fast feedback cycle of important information. They allow the true agility in software development that companies need to be successful.

Good application security practices help to prevent the misuse of software for nefarious means. It aims to protect against data loss and software exploitation. Unfortunately, security is often seen as the opposite of fast and agile.

What do you think about when you read the words “application security”? Is it quickness and agility? Do you think of fast feedback cycles?

Or, like most developers, do you think of slow, onerous processes that bring development work to a halt?

I challenge the assumption that application security does not mix with DevOps. In fact, the intersection of security and DevOps may actually result in your software becoming more secure than using traditional methods.

Let’s take a look at what DevOps really means by defining its core principles, and then showing how security practices fit right in with it.

The Three Ways

There are three central tenets to true DevOps, known as “The Three Ways.” In order to achieve business goals through the use of DevOps, companies must build expertise and maturity in applying The Three Ways.

If you want an in-depth discussion of The Three Ways, pick up the excellent book The DevOps Handbook. It is well worth the read.

Let’s take a closer look at what each of The Three Ways entails and how successful companies apply these tenets.

The First Way – Flow

Imagine the development of software in your organization as a river. A piece of functionality must make it safely from the source of the river (development) down to the ocean (production). How smooth is your river?

Rocks and other obstacles make it difficult to have a safe journey down a river. Too many rocks cause rapids to form, making a trip down the river hazardous. A boat floating down the river has a high risk of crashing.

Is your development process full of rocks and obstacles? Does it take several weeks or a maybe a month to create a test environment for your code? Do you have long, manual testing cycles before you can go to production?

The more obstacles in your river, the more likely you are to crash. The First Way explains that you must find ways to increase the flow of work from development to production.

In other words, you must eliminate obstacles so that code can go from the developer’s mind to production (and thus deliver value) as quickly as possible. You must eliminate the rocks in your river to do that.

What are some common ways to increase the flow of your “river”?

Infrastructure as code – Automate the creation of your environments by transforming your infrastructure into code. Use tools like Chef, Ansible, or Docker to create environment “recipes” and store them in source control. Give yourself the ability to create identical environments on demand. You should also be able to destroy and recreate environments when needed.

Create a deployment pipeline – With environments easily spun up, create a deployment pipeline to enable continuous integration and continuous deployment. Tools such as Bamboo and Jenkins can be used to constantly deploy code to your production-like environments after each code commit.

Enable fast automated tests – DevOps is impossible without automated tests that are run with every build and deployment. Your build should break if any tests fail. Your tests will give you a measure of confidence that your code works in a production-like environment. If the pipeline breaks, fix it immediately.

Small batch development – Code should not be developed during long stretches of time, incorporating multiple pieces of functionality that are later merged into trunk. Develop in small batches that can be committed to trunk and run through the deployment pipeline quickly and with low risk. Large batches of work that take weeks, months, or more to develop are more likely to have issues when deployed to production. Small change = small risk.

Make deployments to production painless – You want to get to the point where you can push a button and deploy to production. This style of deployment is a scary proposition to companies that do things in a traditional way. However, if you have chosen to implement small batch development, as explained above, each individual production deployment can be relatively simple and low risk.

These practices will enable you to reduce the time it takes to develop and deliver features. You’ll increase the flow of your river, reduce risk, and bring value much more quickly to your business.

The Second Way – Feedback

The next step to DevOps maturity is enabling fast feedback to the development team. Why is this important?

Think about a typical scenario in software development today. Developers work on a project that some decision makers hope is the right project to work on at the moment. After building this product or feature for a year, it is released to the end user.

What happens if the end user doesn’t like the product? What happens if the feature you built goes largely unused? How does that make you feel as a developer? How much does that cost the business on wasted effort?

What if you could find out the code or feature you’re working on isn’t useful after only a couple of weeks or a month? The decision makers could then pivot and try something else that’s a wiser use of resources. This flexibility is what fast feedback enables.

What DevOps mechanisms exist to enable this fast feedback?

When writing code, ensure that code reviews are happening so that developers get fast feedback on the code they write.

In terms of features, use A/B testing to determine early whether a new feature will actually be used by your customers.

Set up telemetry in production so that you will have real-time data you can use to react and make decisions.

Mature DevOps companies use telemetry that is tied to business goals to provide fast feedback on whether your code changes have “moved the dial” on business outcomes, which is ultimately what software is meant to do.

The Third Way – Continual Learning and Experimentation

The final way is all about learning. Organizations should strive to incorporate learning into their everyday work in order to improve services and increase resiliency.

A good example of this type of learning is Netflix. You likely have heard of Chaos Monkey, the program that randomly shuts server instances down in production.

Chaos Monkey reveals strange and unexpected behaviors that surface when services don’t respond or different pieces go missing.

Netflix uses lessons learned from the Chaos Monkey test to inform its development teams how to make those services more resilient and stable. This learning is part of the developers’ daily work.

Companies need to build a culture that accepts experimentation, new ideas, and failure. That isn’t a typo. A company with a culture of learning and experimentation knows that failure is a natural part of figuring out what works.

You won’t see all of the failed iPhone designs that Apple came up with before the winning design was created. That’s okay. What we see today is the end result of trying different things and finding out what didn’t work.

Similarly, companies and developers shouldn’t be afraid to try things. Finding something that really is successful with users is worth a few failed experiments.

When an experiment does work, make sure there is a way for employees to share their findings with the company so that everyone can benefit, thus creating culture of continual learning and improvement.

How Security Fits In

With an understanding of the basics of becoming a successful DevOps company, it’s time to see where security fits into the picture.

It is true that security reviews and penetration testing done in the traditional way can slow down development or provide security feedback too late to act upon it. The way forward is to change the way we do application security to fit DevOps principles.

Let’s take a look at how good security practices can follow the formula of The Three Ways and make your software even more secure than it was previously.

Establishing Security Flow

Flow is all about reducing the time it takes to deliver a product or feature to production.

Security flow is all about reducing the time it takes to find and fix security vulnerabilities before they reach production. Flow is not an excuse to ignore application security. Instead, you need to incorporate it into your flow.

First, security can be enhanced by transforming your infrastructure into code. If your infrastructure is code, then it can be easily reviewed. The proper and safe configuration can be captured and repeatedly executed.

This kind of security eliminates the risk that a human configuring a server forgets a key step that opens up a vulnerability. All of the necessary configuration is executed with each new instance without missing any steps.

Security needs to be built into software from the beginning. Design reusable components that provide key security controls, such as authentication, authorization, and session management. Where actual coded components can’t be made available, provide clear guidelines and requirements on how to build software in a secure way.

The build pipeline is a huge piece of incorporating security into a DevOps environment. The build pipeline executes automated tests to ensure functionality works as expected.

Functionality isn’t the only piece that should be tested during the build process. Automated security tests also need to be executed during every commit-build-deploy process.

My recent post on securing a Node.js REST API featured security tests for a small banking API. The Github repo can be found here for further review. For now, let’s take a look at one security test as an example.

This test is the first in a series of authentication tests cases that test to make sure the correct response comes back from the API if the request is not authenticated (in this case with a JSON web token). A “401 Unauthorized” response is returned when the user is not authenticated.

Just as important are authorization test cases.

This test logs in with a legitimate user account and passes the token with the request. However, the user is not authorized to act on this account and thus receives a “403 Forbidden” response.

This example illustrates that security test cases can be easily included in your automated integration tests and run with every execution of the deployment pipeline. When these tests pass, you know that security is being built in from the start.

Automated tests can be used to test for several vulnerabilities, including CSRF validation, authentication, authorization, and clickjacking (by inspecting headers for a valid Content Security Policy). Be creative and automate as much as you can.

Developing with small batch sizes is also a great tool for security. Small changes are easy to understand and thus are easy to code review for possible security vulnerabilities.

Use lists such as the OWASP Top 10 and SANS Top 25 during code reviews to ensure the highest risk vulnerabilities are covered.

The enhanced flow of your pipeline also means that any vulnerabilities found can be quickly fixed and the fixes can be placed in production with minimal lead time.

These practices incorporate flow into your secure development life cycle. Instead of stopping all development to review the entire application for security holes, build in the security controls from the beginning and continuously test them throughout the project.

Establishing Feedback on Your Security Posture

The next key to DevOps is regular and fast feedback. Can that be applied to security? Certainly it can. The goal is to provide the development team with fast feedback on security issues that need to be addressed.

Traditional (and slow) application security reviews may reveal important information. However, this information typically comes too late in the development cycle. To fix the problems would require redesigns of features, and so these fixes go undone in order to meet delivery deadlines.The result is vulnerabilities found by traditional methods can at times sit for quite a while before they are fixed.

Since the goal of DevOps is fast feedback, security should be incorporated from the beginning and continuously throughout the project.

The automated security test cases we discussed above provide the flow we need and are also a good source of feedback. Tests let you know with every commit whether or not you introduced a vulnerability with your change. That’s pretty fast feedback.

Other tools exist to help development teams get the fast feedback needed. These tools are automated and bring another valuable layer to your application security strategy.

Static Application Security Testing (SAST) is an automated tool that scans your source code or binary for patterns that display possible security vulnerabilities. These tools can usually be incorporated into the deployment pipeline so they can execute with every build.

It may seem strange to recommend you do code review and static code analysis. Isn’t it a waste of time to have humans review the code if you have SAST doing it for you?

Actually, is isn’t. The reality of SAST tools is that they are not perfect. Different SAST tools may give you different results. They also may give false positives. Ultimately, you cannot get away from human judgment.

SAST is still an important part of a layered approach to application security and can provide valuable feedback. Just make sure you go in with eyes open and understand that SAST tools can be expensive and introduce some overhead to the process.

Dynamic Application Security Testing (DAST) also scans your application for vulnerabilities. The difference is that it scans the application as it runs in a test environment.

DAST tries to exploit common vulnerabilities in the running application and, if it finds what could be an exploit that works, it lets you know.

This test opens up some nice sources of feedback. For instance, a fun experiment could be creating a ticket automatically in your bug tracking or project management software when the DAST system finds a vulnerability. Then it can be prioritized and fixed along with the rest of the work.

I tend to feel that DAST can generally be more effective than SAST. The reason is that throwing real attacks at a running system in a production-like environment can often find vulnerabilities that human code reviews and SAST scans might miss.

You can use DAST as a permanent penetration tester in your test environment that regularly hits your application and lets you know if any vulnerabilities pop up. Very useful indeed.

Adapting to Developments in Security

The security world is always changing. New vulnerabilities and exploits are always being discovered. However, this fact does not need to paralyze you with fear.

You can use these developments to help fuel continuous learning with respect to security. There are two sides of the learning story, one human and one technological.

The human side is simply to keep up to date on new advances as well as threats that are currently being discovered. Developers should try their best to stay informed about threats that are found against the technologies they are using.

As for technology, information obtained may be fed into a Real-time Application Self-Protection (RASP) tool in order to act upon that information. A RASP allows an application to detect an attack in progress and take action to stop it.

For instance, say a new vulnerability is found for a version of PHP that your website uses. A signature can be fed into a RASP system such that if the system sees an attack that is trying to exploit that vulnerability, it can work to prevent its success. It can also send out an alert so that corrective actions can be taken.

RASP tools can also do some learning of their own. While many are signature based (where known attack signatures are used to detect attacks), some have developed or are developing the ability to learn while in operation.

For example, a RASP could monitor and learn over time what “normal” traffic and function calls look like within an application. If it sees activity that falls outside that behavior, then it sounds the alarm.

Another way of baselining is to traverse all code paths to learn the various functions of an application. Once the RASP tool understands how the application is supposed to operate, it can prevent execution that aims to change that behavior.

A RASP is not a silver bullet. Don’t ignore good security practices and then expect a RASP system to keep you safe 100 percent of the time.

The idea is to create a “virtual patch” that can keep you safe until you can properly fix the security hole against which the RASP is defending.

A great example of a RASP system is OWASP’s AppSensor project. The guide created by OWASP will allow you to implement it in any technology you wish. It allows developers to choose where in the code the RASP technology should be used. A good choice is during the login function or other functions commonly subjected to attack.

The Path to DevSecOps

In today’s world, DevOps alone is not enough. Security must be made a part of the equation and built into DevOps practices. Otherwise, the only thing you may accomplish is more quickly putting a piece of vulnerable software into production.

If you are not currently working these practices into your development life cycle, do not try to do everything at once. Overwhelming your developers will only encourage you to give up if things don’t go well at first.

It is much better to have a clear plan that will end with a DevSecOps environment. Such a plan may look like this:

• Implement automated tests – Automated security tests will give you a quick bang for your buck. Implement tests that ensure the basics are covered (XSS, CSRF, authentication, authorization, etc.).
• Secure code reviews – Start doing secure code reviews for key pieces of the application that may impact the security of your application (database connections, login and session management, etc.).
• Automate security configuration within your infrastructure code – Make updates to your infrastructure code to fill holes in your environment configuration.
• Investigate the use of SAST and DAST tools – Once you build some application security acumen, investigate various automated tools to help find things in a more automated fashion. If budget is limited, DAST would likely be the best dollar for dollar value.
• Use RASP tools to protect your production assets – This step is for more advanced and mature application security programs. However, if done well, RASP can be a great help to quickly “patch” major vulnerabilities until a permanent code change can be deployed to production.

Incorporating security with a DevOps mindset will lead to more secure applications. Previously, it was all too easy to simply focus on functionality and deliver a product to production with little or no knowledge as to how secure it was.