CMock vs FFF - A Comparison of C Mocking Frameworks

Did you know you have options when it comes to creating mocks for your C-language unit tests?

I've been spending a lot of time working with CMock -- since it's used by Ceedling -- but I've just been checking out FFF (the "fake function framework"). It's well done and I think it deserves a closer look.

To compare CMock and FFF, I created an example software module (the event_processor) to build some unit tests for. The event_processor interacts with a display module, which is the interface that is mocked during the tests.

Note that a mocking framework is nothing without a unit test framework (i.e. some "test" or "assert" functions to use in our tests).

In the CMock tests, I use Unity as the unit test framework. To preserve consistency in the FFF tests, I've created a simple unit test framework of my own, with a TEST_ASSERT_EQUAL() macro identical to the one provided by Unity. It's loosely based on MinUnit.

Setup

CMock

I did not find it possible to configure CMock to use it on it's own. What little documentation there is was no help. So instead for these examples I used CMock in a Ceedling project. Ceedling just makes it a whole lot easier to autogenerate the mocks needed for each test.

FFF

FFF is very simple to set up. It comes as a single header file (fff.h) which you can drop right into your source folder. Then you just #include "fff.h" and you're ready to start using it in you tests.

One note: I did need to #include <string.h> ahead of the include for fff.h to get it to build. Looks like FFF is missing an include.

You create a fake function by using either the FAKE_VOID_FUNC() or FAKE_VALUE_FUNC() macro. These macros take the function name, arguments and return value (if applicable) as arguments.

There are also two other important FFF macros to be aware of. RESET_FAKE() is used clear the internal stats for each fake function. FFF_RESET_HISTORY() resets the call history for all fake function calls. Typically, both of these would be called before each test. These calls are left out of the examples below for clarity, but are done in the "set up" function of the test framework... before each test is executed.

Now lets look at a few different examples of how to use each framework.

Test that a single function was called

CMock

void test_whenTheDeviceIsReset_thenTheStatusLedIsTurnedOff(void)
{
  // Then
  display_turnOffStatusLed_Expect();

  // When
  event_deviceReset();
}

FFF

FAKE_VOID_FUNC(display_turnOffStatusLed);

void test_whenTheDeviceIsReset_thenTheStatusLedIsTurnedOff(void)
{
  // When
  event_deviceReset();

  // Then
  TEST_ASSERT_EQUAL(1, display_turnOffStatusLed_fake.call_count);
}

A lot of times, all I want to do is verify that a particular function was called.

With CMock, I set up the "call chain" before calling the function under test. CMock itself verifies that everything listed in the call chain is called correctly. Note that this usually means reversing the order of the "then" and the "when" in my test.

With FFF, each "fake function" behaves more like a traditional mock object. I create the mock, call the function under test, and then I can inspect the mock after-the-fact. I use the unit test framework to make sure it was called the correct number of times. This lets me write the test in the logical given-when-then order that things actually occur.

Another issue here is when an expectation fails with CMock, the line number reported is the line number of the start of the test function... which is not very much help. When you use the unit framework to inspect the mock (as with FFF), any failures yields the real line number at which the failure occurred.

Test that a single function was NOT called, while other functions may be called

CMock

void test_whenTheDeviceIsReset_thenTheStatusLedIsNotTurnedOn(void) 
{
  // Then
  // NOTE: We have to ignore a function we know could be called.
  display_turnOffStatusLed_Ignore();

  // When
  event_deviceReset();
}

FFF

FAKE_VOID_FUNC(display_turnOnStatusLed);

void test_whenTheDeviceIsReset_thenTheStatusLedIsNotTurnedOn()
{
  // When
  event_deviceReset();

  // Then
  TEST_ASSERT_EQUAL(0, display_turnOnStatusLed_fake.call_count);
}

CMock requires that every expected function call appears in the call chain. Even if we don't care when or how often a particular function is called, we need to "ignore" it explicitly. If we're not careful, this can lead to long chains of expectations.

With FFF, calls are ignored by default. As long as we've created a fake function, we don't need to tell FFF what we're expecting. We can control what to test for ourselves after running the function under test.

I think this is the single biggest benefit of using FFF. By only looking at the interactions that we actually care about, it becomes possible to write simpler (and less fragile) tests.

Test that a single function was called with the correct argument

CMock

void test_whenTheVolumeKnobIsMaxed_thenVolumeDisplayIsSetTo11(void)
{
  // Then
  display_setVolume_Expect(11);

  // When
  event_volumeKnobMaxed();
}

FFF

FAKE_VOID_FUNC(display_setVolume, int);

void test_whenTheVolumeKnobIsMaxed_thenVolumeDisplayIsSetTo11(void)
{
  // When
  event_volumeKnobMaxed();

  // Then
  TEST_ASSERT_EQUAL(1, display_setVolume_fake.call_count);
  TEST_ASSERT_EQUAL(11, display_setVolume_fake.arg0_val);
}

The FFF version includes one extra line because the function call count is tested independently from the expected value.

Test a sequence of calls

CMock

void test_whenTheModeSelectButtonIsPressed_thenTheDisplayModeIsCycled(void)
{ 
  // Then 
  display_setModeToMinimum_Expect();
  display_setModeToMaximum_Expect();
  display_setModeToAverage_Expect();

  // When
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed();
}

FFF

FAKE_VOID_FUNC(display_setModeToMinimum); FAKE_VOID_FUNC(display_setModeToMaximum); FAKE_VOID_FUNC(display_setModeToAverage);

void test_whenTheModeSelectButtonIsPressed_thenTheDisplayModeIsCycled(void) 
{
  //When
  event_modeSelectButtonPressed();
  event_modeSelectButtonPressed(); 
  event_modeSelectButtonPressed();

  // Then
  TEST_ASSERT_EQUAL(fff.call_history[0],
          (void *)display_setModeToMinimum);
  TEST_ASSERT_EQUAL(fff.call_history[1],
          (void *)display_setModeToMaximum);
  TEST_ASSERT_EQUAL(fff.call_history[2],
          (void *)display_setModeToAverage);
}

The CMock syntax here is a bit nicer, but I'd argue that we don't want to make tests like this (with long call chains) easier to write. This is the slippery slope to brittle and un-maintainable tests.

Mock a return value from a function

CMock

void test_givenTheDisplayHasAnError_whenTheDeviceIsPoweredOn_thenTheDisplayIsPoweredDown(void)
{
  // Given
  display_isError_ExpectAndReturn(true);

  // Then
  display_powerDown_Expect();

  // When
  event_devicePoweredOn();
}

FFF

FAKE_VALUE_FUNC(bool, display_isError);
FAKE_VOID_FUNC(display_powerDown);

void test_givenTheDisplayHasAnError_whenTheDeviceIsPoweredOn_thenTheDisplayIsPoweredDown(void)
{
  // Given
  display_isError_fake.return_val = true;

  // When
  event_devicePoweredOn();

  // Then
  TEST_ASSERT_EQUAL(1, display_powerDown_fake.call_count);
}

Both tests are pretty simple, although the FFF test can be written in the correct "given-when-then" order.

Mock a function with a value returned by reference

CMock

Note: This CMock test doesn't work out-of-the-box. The "returnthruptr" and "ignorearg" plug-ins need to be enabled in the Ceedling project.yaml file._

void test_givenTheUserHasTypedSleep_whenItIsTimeToCheckTheKeyboard_theDisplayIsPoweredDown(void)
{
  // Given
  char entry[] = "sleep";
  display_getKeyboardEntry_Expect(0,0);
  display_getKeyboardEntry_ReturnArrayThruPtr_entry(entry, strlen(entry));
  display_getKeyboardEntry_IgnoreArg_entry();
  display_getKeyboardEntry_IgnoreArg_length();

  // Then
  display_powerDown_Expect();

  // When
  event_keyboardCheckTimerExpired();
}

FFF

FAKE_VOID_FUNC(display_getKeyboardEntry, char *, int);
FAKE_VOID_FUNC(display_powerDown);

void test_givenTheUserHasTypedSleep_whenItIsTimeToCheckTheKeyboard_theDisplayIsPoweredDown(void)
{
  // Given
  char mockedEntry[] = "sleep";
  void return_mock_value(char * entry, int length)
  { 
      if (length > strlen(mockedEntry))
      { 
          strncpy(entry, mockedEntry, length); 
      }
  } 
  display_getKeyboardEntry_fake.custom_fake = return_mock_value;

  // When
  event_keyboardCheckTimerExpired();

  // Then
  TEST_ASSERT_EQUAL(1, display_powerDown_fake.call_count);
}

CMock starts to get a little hairy in this test. Writing this test requires a lot of knowledge of the CMock API. With FFF, you do a little more work in the test, but it's just C so you already know how to do it.

When writing the CMock version, I had to refer back to the documentation a few times, but with FFF, I got it working on the first try.

Mock a function with a function pointer parameter

CMock

void test_givenNewDataIsAvailable_whenTheDisplayHasUpdated_thenTheEventIsComplete(void)
{
  // A mock function for capturing the callback handler function pointer.
  void(*registeredCallback)(void) = 0;
  void mock_display_updateData(int data, void(*callback)(void), int numCalls)
  {
      //Save the callback function.
      registeredCallback = callback;
  }

  // Given
  display_updateData_StubWithCallback(mock_display_updateData);
  event_newDataAvailable(10);

  // When
  if (registeredCallback != 0)
  {
      registeredCallback();
  }

  // Then
  TEST_ASSERT_EQUAL(true, eventProcessor_isLastEventComplete());
}

FFF

typedef void (*displayCompleteCallback) (void);
FAKE_VOID_FUNC(display_updateData, int, displayCompleteCallback);

void test_givenNewDataIsAvailable_whenTheDisplayHasUpdated_thenTheEventIsComplete(void)
{
  // A mock function for capturing the callback handler function pointer.
  void(*registeredCallback)(void) = 0;
  void mock_display_updateData(int data, void(*callback)(void))
  {
      //Save the callback function.
      registeredCallback = callback;
  }
  display_updateData_fake.custom_fake = mock_display_updateData;

  // Given
  event_newDataAvailable(10);

  // When
  if (registeredCallback != 0)
  {
      registeredCallback();
  }

  // Then
  TEST_ASSERT_EQUAL(true, eventProcessor_isLastEventComplete());
}

In this case, both CMock and FFF work similarly. You register your own function to be called when the mock is called, and set up your function to capture the provided function pointer. Then you can manually do whatever you need to with it.

Summary

CMock and FFF take very different approaches.

With CMock, you run it ahead of time on a header files to generate a whole bunch of mock functions for you. There is a lot more "pre-processing machinery" that runs ahead of time.

And with CMock, you need to explicitly declare all expected function calls ahead of time. This can lead to tight coupling of tests and code.

FFF is much lighter weight. You just include the single header file, and you're ready to go.

It works more like a traditional "mock object" you'd see in an object-oriented mocking framework. There is no need to set up all expectations ahead of time -- you just inspect the mock when you're done... and only for the things that you care about.

CMock is certainly easy to get started with via Ceedling, which provides automatic discovery, generation and running of tests and mocks.

FFF has me excited to work with it further though. In particular, being able to write tests in a more logical given-when-then order, and the ability to ignore mocked calls by default looks to be powerful.

With FFF there is a bit more manual work to set up each test, but I think I may be able to write some better quality tests.

Get the test project source on GitHub.

Ceedling makes mocking easier

Save time when using mocks by using Ceedling to automatically generate them for you. See how easy it is in in my downloadable "how to" guide.

Sign up to get it here.