Applying DevOps and agile to embedded testing

Source – devopsonline.co.uk

When people talk about testing, it generally applies to all types of software testing. However, if you look at a bit more in detail, there are some areas of testing that claim their own space and embedded testing is one of them. It is an area that deserves to be treated as a specific case due to the particularities of the requirements and of the skills needed.

With the evolution of the development and testing practices towards agile and DevOps, this column aims to propose a few ideas on how to try to apply the principles of agile and DevOps to embedded development and testing.

Embedded testing is the practice of testing the software stack developed to run on a device or hardware for which the software has been designed. For example, you can think of the software running on a sports watch (Fitbit, Garmin), a set-top box, medical equipment, automated production line devices, etc.

In order to understand a bit more let’s start by describing the system under test. It is safe to assume that we will have the following:

• Some custom hardware in which the software will have to run. Very often, as it is the case in consumer devices, the resources of the hardware will be reduced to the minimum in order to keep the cost as low as possible. There will be a certain amount of future proofing, but in the 3 to 4 years following the launch of the device, it is common for the hardware to be struggling to keep with the demands put on it.
• It is likely that there will be some kind of firmware or driver layer that will do the interface with the hardware and a middleware layer to implement some of the functionality of the device.
• There will be a user interface to drive the device; to send commands to the hardware, to configure it, or to make it perform the device purpose.
• Finally, there will probably some kind of connectivity, either with another device or with some back-end services.

This is a very diverse ecosystem and the task of testing the code running in the device is to ensure that the software and by extension the device and its connected services are able to perform their purpose functionally and with quality.

Customer value

Historically, the main device manufacturers have got release cadences of about 2 to 3 years for a new consumer device. They will produce software updates more often, sometimes quite regularly (up to a release every 1 to 2 months) but more likely it is 3 to 4 releases a year at the beginning, and then after a few releases, the devices become obsolete or the cost of developing and testing is too high to carry on the specific device and it is left to stagnate and to become stale.

So what can be done to try to ensure a regular and consistent delivery of value to customers?

First, the description of the system already provides some part of the answer. It is essential to decompose your system and apply the same rules as normal testing. Test each layer or component as independently as possible. It is good to design the system in a way that it is possible to release the different components independently. The more hard dependencies and coupling in between the different parts, the more difficult it will be to have a predictable release cadence, as a single issue in a part of the system, will stop the full release.

It is highly likely that there won’t be any off-the-shelf automation solution to test the full system, as it is very specific to the device. Therefore a good idea is to try to use the tools that are available for the subparts of the system that is standard. For example, is it possible to test the UI layer outside the device using some standard UI automation tools?

The other parts of the system can be tested at unit test level and full component level. It then becomes very important to have very well defined interfaces that are thoroughly tested to try to minimise the amount of integration testing.

The above steps will ensure that the integration-testing phase is then fulfilling its purpose, which is to ensure that all the components work well together, and not to test functionally some of the components. This can be difficult as some of the functionality might only be exposed and properly testable once the full system is put together. This is where if there isn’t a tool available, it will be required to develop a custom tool. The effort required would be substantial as it is a fully integrated tool that will be domain specific.

Embedded testing

One of the complexities of embedded testing is that the system is heavily optimised, meaning that they operate very little spare capacity and changes can easily have unexpected consequences on other parts of the system as the execution timings are slightly modified.
A second complexity is that it is difficult to debug the system by running some instrumented code or attaching a debugger will also change the way the software behaves.

The final challenge is that some defects can be very difficult to reproduce and are given in some very special conditions that are not covered by testing. In the case of a consumer device, the space to cover is virtually infinite, so having the ability to gather system information from devices that are out in the field with customers will help a lot in understanding the configuration and environments required for testing.

The principles above do not only apply to functional testing but also to non-functional testing, in this case, performance, load, volume, security, energy consumption, stress, reliability and any other non-functional characteristic.

It is important to not leave functional testing as the last phase of the development and testing cycle. Non-functional testing has got to be performed as often and as thoroughly as functional testing. Great care has to be put in tracking the performance indicators on a build per build or throughout the intermediary releases, to try to correct the cumulative degradation of performance gradually introduced with new code and features. Fixing non-functional issues at the end of the development cycle can be very expensive and add an unacceptable delay to the release schedule.

The overall aim has got to test each component as much as possible and as independently as possible. This is not just a matter of test execution, but it is also a matter of design and implementation of the functionality. The flexibility provided by the reduced interdependence will be a major step forward in being able to release regularly and predictably.

The last aspect to consider is the overall cadence and delivery cycles for the embedded system. Again, by having reduced the dependencies between components, it should be possible to adapt the release cadences to the characteristics of each component. For example, there is absolutely no reason not to go full dev ops for the back-end services with micro releases happening transparently.

Delivery consistency

Depending on how much decoupling there is inside the different layers of the device, it should be possible to release components independently and frequently. If there is little separation then it is better to try to establish parallel delivery cadences, let’s say a fast cadence that could be weekly and a slower one that could be monthly and a very slow one, quarterly and then try to fit the different functional changes according to their complexity and dependencies in the different parallel delivery slots. The more coupled is the system the less agility and flexibility of delivery will be possible. This will be a mirror of the development and testing processes, if the system and processes are not designed for fast turnaround from design to feature tested, integrated and available for release to customers, it will not be possible to engineer it once the code is developed.

Evidently, the fundamental basis underpinning this strategy is test automation. It is essential in order to have sufficient test coverage (and this is a very subjective measure), fast and reliable execution. Yet this will not be enough without some good manual testing to ensure that the good test cases are specified and automated and especially to ensure some good exploratory testing.

As we have described, embedded testing present some specific challenges. So as to be able to reduce the resulting risks, a good system design that reduces dependencies between components, a good test strategy based on testing components independently as much as possible, a strong automation strategy (for functional and non-functional testing), and a multi-speed delivery cadence adapted to the complexity of the deliverables, will go a long way towards achieving regular and predictable releases of functionality to customers.