More flexible mocks for Ceedling with the Fake Function Framework (FFF)

I've created a plug-in for Ceedling which lets you use the Fake Function Framework (instead of CMock) to automatically create the mock interfaces used in your unit tests. You can find the plug-in (along with complete instructions for how to use it) in the GitHub repository.

Ceedling is a nice build/test system for C applications that makes it easier to create unit tests, especially those needing mocked interfaces. It does this by automatically generating mocks for the source modules you tell it to. This lets you test the interactions between your modules (and is especially useful when testing interactions with hardware).

By default, Ceedling uses a mocking framework called CMock. It's pretty good (hey mocking in C is hard!), but it's not my preferred tool for this sort of thing. In particular, CMock is very strict about expected mock behavior. You need to explicitly define all of the all of the expected interactions ahead of time. In my experience this leads to "brittle" tests -- those that break easily during continued development -- and disrupts the natural (given-when-then) flow of the test.

A "brittle" test with CMock

For example, consider a system with a speaker and some LEDs. When we get an "critical alarm", we want to:

  1. Turn on the red LED, and
  2. Play a specific sound on the speaker.

There are three source modules in this system:

Here we'll test the alarm_handler in isolation by mocking the led and speaker modules (the led and speaker modules also likely have hardware dependencies, and so are a good candidate for mocking as well).

In this test, we'll know that the alarm_handler behaves correctly by testing its interactions with the led and speaker modules. In each case, we expect a particular function to be called.

#include "alarm_handler.h"
#include "mock_led.h"
#include "mock_speaker.h"

void
test_whenACriticalAlarmIsDetected_thenTheRedLedIsTurnedOnAndTheWarningSoundIsPlayed()
{
     // (Then) we expect these functions to be called
     led_turn_on_red_Expect();
     speaker_play_warning_sound_Expect();

     // When this happens
     alarm_handler_critical_alarm_detected();
}

This isn't too bad (for now), but first notice how the "when" and the "then" clauses are backwards. We have to define the expected behavior (the functions we expect to be called) before we can call the function under test.

Secondly, it isn't apparent here yet... but there is a bigger problem lurking. Since CMock is strict about its expectations, if alarm_handler_critical_alarm_detected() calls any other function than the two we have expected then the test will fail. Is this what we want? It could be, but not necessarily.

What if in the future we wanted something else to happen when an critical alarm occurred (like turn off a power signal somewhere somewhere)? The additional function call in alarm_handler_critical_alarm_detected() would break this test. But this test shouldn't break in this case. This is a brittle test. We haven't changed what the speaker or the LED do during a critical alarm, but the test still fails.

In this case, the work required to fix the test isn't that siginificant. But as the system grows and becomes more complicated, it will be harder to go back in and figure out where the test broke and how to fix it. We don't want our test to keep breaking as we add functionality. We don't want the tests to make things harder for us.

Using the Fake Function Framework instead

So, I've created a Ceedling plug-in that lets you use the Fake Function Framework (FFF) instead of CMock. FFF creates "fake functions" which you can inspect after you've run your function under test (preserving the natural given-when-then test flow). And, FFF doesn't need you to strictly define all calls to expect ahead of time. By being less strict, FFF lets you more easily ignore function calls (since that's the default behavior).

The FFF version of the test above would look like this:

#include "alarm_handler.h"
#include "mock_led.h"
#include "mock_speaker.h"

void
test_whenACriticalAlarmIsDetected_thenTheRedLedIsTurnedOnAndTheWarningSoundIsPlayed()
{
     // When this happens
     alarm_handler_critical_alarm_detected();

     // Then we expect these functions to be called
     TEST_ASSERT_CALLED(led_turn_on_red);
     TEST_ASSERT_CALLED(speaker_play_warning_sound);
}

Now we're only checking if these two functions were called. If alarm_handler_critical_alarm_detected() calls any other additional functions, this test won't break. Also note that the "then" component of the test at the end -- where it makes the most sense.

Refactoring tests with FFF

Since FFF is less strict, we can actually refactor this test now. This wasn't possible with CMock. Instead of testing two things in one test, we can create a separate test for each behavior:

void
test_whenACriticalAlarmIsDetected_thenTheRedLedIsTurnedOn()
{
     // When this happens
     alarm_handler_critical_alarm_detected();

     // Then we expect this function to be called
     TEST_ASSERT_CALLED(led_turn_on_red);
}

void
test_whenACriticalAlarmIsDetected_thenTheWarningSoundIsPlayed()
{
     // When this happens
     alarm_handler_critical_alarm_detected();

     // Then we expect this function to be called
     TEST_ASSERT_CALLED(speaker_play_warning_sound);
}

I find that this approach is a clearer to understand and makes the tests more maintainable.

Failed line numbers are now correct

An additional benefit of using FFF over CMock is that the line numbers for failed test assertions are now correct. When a CMock expection fails, you don't get the exact line number of the failure (you get the line number of the start of the test instead). This can make it really difficult to find an error in a test with a lot of assertions... which your tests are bound to have because it's difficult to refactor ;)

With FFF though, you get the exact line numbers where your test assertions fail. This is because you just use Unity and its TEST_ASSERT macros to test your fake function calls.

Getting started

This has just been a quick comparison, but there's actually a lot more you can do with FFF and the plug-in. Try it out by creating a new Ceedling project and "installing" the plug-in (get the source and enable the plug-in in the project.yml file). For more details and examples see the readme in the GitHub repository.