Technical Article

Why You Should Use Unit Tests to Write Better Embedded Software

February 08, 2016 by Matt Chernosky

An introduction to software unit testing for embedded systems developers.

Unit tests can help you write better embedded software. Here’s how.

Unit tests are additional software functions that you write to test the software "units" of your application. These tests help you ensure that your embedded software is working correctly -- now and as it changes over time.

In an embedded C application, a "unit" is typically a single source file (and corresponding header file). This source "module" is usually an abstraction for some part of the system and implements a group of related functions, e.g. a ring buffer or a protocol parser.

The unit tests for that module are a group of functions which exercise the "module under test." The unit test functions call functions of the module under test in a specific order and with specific arguments -- and verify that the module returns the correct results.

These test functions are not included in your release build, but are run during development (or after any changes) to test your application.

Typically, each source module to be tested has a corresponding unit test file, where all the unit tests for that module go.

 

 

For example, if you had your own implementation of some data structure -- like a stack -- then the unit tests might call the push and pop functions to make sure that the stack behaves as expected in a variety of conditions.

Here's an example of just one of those functions:

#include "some_test_framework.h"
#include "my_stack.h"

// A test for my_stack.
void test_WhenIPushAnItem_ThenTheCountIncreases(void)
{ 
    // Do something.
    stack_push('a');
    
    // Make sure it had the right effect.
    ASSERT(stack_get_count() == 1);
}

This particular test is just the tip of the iceberg. We could quickly and easily add more tests for other conditions -- even conditions unlikely to be encountered when your software is in operation.

For example, how does the stack behave when it fills up? Sure I think I know what's going to happen based on how I wrote it, but when will that code ever actually be called?

Hint: I really hope it's not 10 years from now, when the device is a mile under the ocean, and you're nowhere to be found!

If you create a unit test for this case, you can run that code right now and be sure of what it actually does.

// Another test for my_stack.
void test_GivenTheStackIsFull_WhenIPushAnotherItem_ThenItIsRejected(void)
{
    // Fill the stack.
    for (int i = 0; i < 100; i++)
    {
        stack_push('a');
    }
 
    // Try to push another.
    bool success = stack_push('a');
    
    // Make sure it was rejected.
    ASSERT(success == false);
    ASSERT(stack_get_count() == 100);
}

This is especially relevant for embedded software, since it has to deal with real hardware. With hardware, you can't usually exercise all of its behavior and so it's difficult to know with certainty that your software is going to handle all of it okay.

For example, how can I test my temperature conversion logic across all ranges of temperature, when my temp sensor is reading a comfortable 72 degrees -- the temperature of my office?

I suppose I could stick my hardware in a freezer or thermal chamber, but that is going to 1) take some physical effort to set up and 2) not be very repeatable.

A better option, as you might have guessed by now, would be to put all of my temperature conversion logic in its own source module and write a bunch of unit tests for it. I could feed in any raw sensor value that I want (including errors) and check that each is handled correctly.

The goal of a unit test is to test your software "unit" in isolation from the rest of the system. You treat the unit as a black box, call functions in a specific order and with specific arguments, and verify that you get the correct results. The reason to test in isolation is that when something goes wrong, you know exactly where the problem is -- in the module under test.

Most source modules have dependencies though. To test a module in isolation, you can not include other modules that it might depend on. So what do you need to do? Ah, the answer is that you need to "mock" those dependencies.

A mock is a fake implementation of a module that allows you to simulate and inspect the interactions to it. You can control how a mock behaves, so that you can fully exercise the module under test. 

In the temperature sensor example, the temp sensor driver (with the conversion logic) might need to use an I2C driver to talk to the sensor. To test the temp sensor driver in isolation, you would need to mock the I2C driver. 

 

 

The I2C driver mock allows you to return whatever test data you want to the temp sensor driver when it makes calls into the I2C driver. When reading the current temperature register, instead of actually going out the hardware, you just tell it to return 0xFF (or whatever value you want) instead.

The other great thing about mocking the I2C driver is that it removes any hardware dependencies from the tests. This means you don't actually need the real hardware to test the application. You can compile the tests and run them on the host PC.

Sounds great so far, right? Good. So how do you actually do this? Okay, okay, I'm getting to that.

There are two main components to any unit test setup: the unit test framework itself, and the mocking framework. The unit test framework is what allows you to define and execute tests, and gives you some "assertion" functions to assert that a particular test has passed or failed. The mocking framework is what you use to mock your dependencies and test each module isolation.

If you're developing a .NET application in Visual Studio or a Java app in Eclipse, the unit test support is built right in to the IDE. You just set up your tests and click the "run tests" button. This is automatic test discovery, and is super convenient. When you set up your test files correctly, the test framework can automatically run all your tests in a single step.

If you're writing an embedded application in C, the best option right now is Ceedling. It's a unit test system built around Rake (like make but for the Ruby language). To use it you'll need to install Ruby, but you don't actually have to know anything about Ruby.

Ceedling uses Unity as its unit test framework and CMock as its mocking framework. The reason it's so great is that it provides automatic test discovery and execution. This makes it easy to get up and running quickly. And it also will automatically generate mock modules if you ask it correctly.

Ceedling is designed to work by running tests on a host PC -- not on target hardware. The tests are compiled using a native compiler (gcc by default). This means that the tests run quickly -- no waiting to flash your hardware -- and can be run continuously during development without slowing you down.

Since the tests are running on your host PC, all your hardware dependencies need to be mocked -- like the I2C driver in the temperature sensor above. Since the tests are running on a PC, the tests can't access the target processor's I2C registers because they don't exist.

This encourages a well-designed, layered architecture where the hardware interfaces are decoupled from the rest of the application logic.

Have you ever worked on a project where the hardware wasn't ready yet? Or there wasn't enough to go around? Or it was changing in the next board rev?  Being able to develop and test some, or maybe even most, of your application without the hardware can help in each of these cases. 

You're still going to need to test on real hardware at some point, but you can get pretty far without it.
 
Since the application is built from a bunch of individually unit tested modules, when you do test on real hardware there will be a lot less to test. You're only testing the integration of those modules. And... the best part is that there will be fewer bugs to find and fix.

Next Article in Series: How to Write Better Unit Tests For Embedded Software With TDD

2 Comments
  • P
    Pattern-chaser February 29, 2016

    Nice introduction, Matt! These comments are always difficult. Criticism is too negative (and unnecessary in this case!) and just saying “Yes, I agree” is nice but unhelpful. I’ll try my best. 😉

    “even conditions unlikely to be encountered when your software is in operation” - ‘unlikely’ means they *could* occur, so test them! 😊 Sooner or later, the right conditions *will* occur, and you don’t want that to be the first time your code is tested.

    I haven’t used Ceedling, so I can’t compare directly. I always used CuTest. I had to manually create stubs for all the out-of-class methods called, and some automation would’ve been nice there. Other than that, CuTest allowedd us to write and run our tests easily and quickly. Incidentally, our bug-count did not start to drop as we hoped when we introduced TDD. It did start to drop—- a lot! 😊—- when we installed a run-unit-tests button on the toolbar of our preferred editor. When it became convenient (a single mouse click) and quick (less than 10 sec when our company network was running slowly; normally less than 2 sec) to run the tests, we did it all the time. We were as lazy as any firmware team; it was the *convenience* that got us using TDD, and boy, was it worth it! Never underestimate human psychology, and pay attention to the details. 😉

    Another thing: we found it conveneint to #include our unit tests into the module-under-test, so that the tests automatically gained access to private (static) class variables. Where possible, we didn’t use it, but occasionally we found we couldn’t do without it.

    Finally: we only tested API methods. Internal (private) methods should be called by the API methods anyway, so their code is tested. This ensures that when you refactor, your tests don’t all need changing to suit. And that’s the point: refactoring must retain functionality as seen via API calls, so don’t make your tests dependent on the internal implementation, only on the API. [Just as your example test shows: you confirm the test has passed by using an API method to access your stack count.]

    Like. Reply
  • mattchernosky March 01, 2016

    Thanks for the feedback!

    Ah yes, in this case “unlikely” means unlikely to happen at your desk during development… but almost certain to occur in operation… at some point.

    Manually creating stubs? That sounds like a lot of work! Yeah, the automation you get with Ceedling is really helpful, especially I think if you’re just getting started. It removes one more barrier to get over. Helps with that whole “laziness” angle too 😊

    Like. Reply