Quick test: Are you using all the DevOps testing tactics in your toolkit?

Source:-https://www.itproportal.com/

Technology delivery has seen an outpouring of problem-solving processes that each come with a whole kitbag of tools: CI/CD systems, frameworks, monitoring tools, security audit – and testing.

As modern technology companies mature along their DevOps journey by adopting continuous integration practices, an increasing level of importance is being placed upon examining and assessing automation. Testing and validation provide huge amounts of value through your entire software delivery lifecycle. Here’s a little guide to a few tests that can help ensure builds are strong and deployment runs seamlessly as code goes live.

These are a handful adopted by some of today’s highest performing teams, but there are many more. The field is packed with testing types and terminology – and sometimes these get used incorrectly. Clearly a good understanding of these terms’ meaning helps a software delivery team understand what their people and partners are talking about and get to success faster.

Unit testing
A method focused around vetting individual units, or pieces of code. The goal is to determine logical integrity – i.e. that code does what it’s supposed to. Most often teams test individual methods or functions as units. Then depending on the size and complexity, also classes. These will be tested in isolation, and then any typical dependencies are stubbed or mocked. (Also beware that mocking/stubbing can mask changes in underlying systems at times, so use with caution).

For instance, if you had a function that massages data from a database you wouldn’t use a real database: You’d make a call to a stubbed endpoint, which returns the data you’d normally expect from a database. That way, the only functionality being tested is this piece of code, or the unit.

Most languages have at least one unit testing framework recommended, e.g. Java → JUnit, Python → PyUnit or PyTest, JavaScript → Mocha, Jest, etc.

Integration testing
A method focused on vetting multiple components together. The goal of integration testing is to ensure relationship integrity and flow of data between components or units. Usually, teams will run unit tests first to test logical integrity of individual units. Then they will run integration tests to ensure interaction between these units is behaving as expected.

Example: Continuing the previous example, an integration test in this case would be running the same test against a real database. With real databases, you have additional scenarios and behaviors to consider.

Integration testing is a broad term encompassing any tests where multiple components are involved. A large variety of technologies and frameworks can be used, including those used above in unit testing, or behavioral-based frameworks, such as Cucumber, Postman, SoapUI, etc.

End-to-end testing
System tests, or end-to-end (E2E) tests, are focused on vetting the behavior of a system from end-to-end. The goal is to ensure the entire application or system behaves how we expect it to, regardless of internal workings. Unit and integration tests are typically white box, meaning that the internals are known. E2E tests are typically black box, that is we only verify input and output combinations.

One example is an E2E test might be a generic user story like “fetch a user’s data.” The input could be a simple GET request to a specific path, and then we verify that the output returned is what we expect. How the system fetched that data underneath is irrelevant.

E2E tests can only check the overall behavior, so this is why unit and integration tests are necessary. It could be that although the output is correct, the way the result is obtained internally is incorrect, and an E2E test would not catch that.

For E2E tests, you typically use behavioral-based frameworks. You might use frameworks like Cucumber, Postman, SoapUI, etc. A lot of API-testing frameworks are used for E2E testing because an API is how you programmatically interact with an app.
Acceptance testing
Typically a phase of the development cycle, the goal is to verify that a given product or feature has been developed according to specifications set forth by a customer or an internal stakeholder, like a product manager. This could have multiple phases, such as alpha or beta testing.

While acceptance tests can verify that the application behaves how a user wants it to, it does not verify the integrity of the system. A caveat of user acceptance testing is there’s a limit to the scenarios a person can come up with: Which is why automated testing methods are important – since every single use case is codified.

White box testing (structural, clear box)
White box (which is also called structural or clear box) testing describes tests or methods in which the details and inner workings of the software being tested are known. Since you know the functions, the methods, the classes, how they all work, and how they tie together, you’re generally better equipped to vet the logical integrity of the code.

You might know a quirk with the way a language handles certain operations. You could write specific tests for that, which you otherwise would not know to write in a black-box scenario.

Unit testing and integration testing are often white box.

Black box testing (functional, behavioral, closed box)
Black box (functional, behavioral, or closed box) testing describes any tests or methods in which the details and inner workings of the software being tested are not known.

Since you don’t know any of the particulars, you can’t really create test cases that target specific niche scenarios or stress specific logic in the system. The only thing you do know is that for a request or given piece of input, a certain behavior or output is expected. Hence, black box testing primarily tests the behavior of a system. End-to-end tests are often black box.

Gray box testing
Gray box testing is a hybrid of black box and white box testing. It takes simplicity of black box testing (e.g. input → output) and targets specific code-related systems of white box testing. This exists because black and white box testing can miss important functionality.

Black box testing only tests that you get a certain output for a given input. It does not test the integrity of internal components  - you could be getting the correct output purely by chance.

White box testing focuses on the integrity of individual units and how they function together, but it is sometimes insufficient for finding system-wide or multi-component defects.

By combining the two, gray box testing can encompass more complicated scenarios to really validate that an application is sound in structure and logic.

The purpose of continuous integration is to deliver feedback as fast as possible, so run the fastest tests first. In some cases, you might run the tests most likely to fail (and thus provide signal) first. Generally you want the cheap and fast things to run first and gain confidence so that you don’t run the complex and long-durations tests unless you’re sure everything else is good to go.

These examples are just a few of a great number of possible tests the DevOps team should know – but there are so many more. It’s worth supercharging this part of the software engineering process to ensure a more successful delivery throughout.