Hack w/ Limbic Media

Interactive Electronica and IoT

Visit Limbic Media

hack.limbicmedia.ca was created by the developers and engineers at Limbic Media to share their knowledge and experiences with DIY and hacker community-at-large.

by Steven Bjornson

Choosing a C++ Unittest Framework

Last week I began searching for a unittest framework for C++. For the short term, I need a tool I can get running as soon as possible (so I can get writing tests as soon as possible and not succumb to test-apathy). In the long term, I want my tests to remain usable in the future and so this unittest search has become a way to vet possible testing frameworks to be used for for the rest of the project's lifespan (by every member of the team).

A quick Google got me to this: Exploring the C++ Unit Testing Framework Jungle

This article, from 2004 (updated in 2010), reviewed 6 different frameworks and provided some solid benchmarks for comparing the different frameworks; you might say, a framework for frameworks. But, while the article is quite extensive and well written, the best the author can say is there are 'reasonable candidates'.

So, I did a little more searching and realised there is a candidate missing from the list (because the article pre-dates it by nearly 10 years; ): Google Test

Reading the GitHub, I got excited. The framework is reasonably cross-platform (especially for our purposes), is used by at least a few large projects, and has a rich set of features.

So, in an effort to extend the knowledge base of the internet (and to make an argument to my team for choosing this tool), below is an examination of Google Tests through the same methodology as used in Exploring the C++ Unit Testing Framework Jungle:

1) Minimal amount of work needed to add new tests:

There is no issue with this, tests can be created quickly. Here is the basic form of test (from the Google Test Primer):

TEST(test_case_name, test_name){  
    ... test goes here ...
   }

The test code is placed in a file, such as myfunction_unittest.cc, and compiled with the code for myfunction.cc (along with a main for the test, found here) and an executable is produced which runs the test.

Furthermore, this form allows for the same method to be tested in multiple and isolated ways. Again, from the Primer:

// Tests factorial of 0.
TEST(FactorialTest, HandlesZeroInput) {  
    EXPECT_EQ(1, Factorial(0));
}

// Tests factorial of positive numbers.
TEST(FactorialTest, HandlesPositiveInput) {  
    EXPECT_EQ(1, Factorial(1));
    EXPECT_EQ(2, Factorial(2));
    EXPECT_EQ(6, Factorial(3));
    EXPECT_EQ(40320, Factorial(8));
}

Where the above code shows two different streams of testing for a single method.

Basically, creating new test is easy and they can be appended to the end of the other tests.

2) Easy to modify and port:

This functionality is less important for our project purposes. While it is probably not trivial to modify the code base, it is definitely not out of the question (and Google even welcomes patches and contributions).

Porting to different platforms, again, is not all that important because the framework is supported on all the platforms we'll be using.

Furthermore, they have tried to keep the project platform neutral and it works with a bunch of different compilers (on my system, I was able to compile with gcc/g++ and clang++ with C++11).

3) Supports setup/teardown steps (fixtures):

Yes, fixtures are supported. Furthermore, these fixtures only have to be declared once and can be shared between tests. And, importantly, sharing fixtures does not break the much needed isolation of tests because fixtures are created (fresh) at the beginning of every test and thrown away at the end.

In the case of Google Test, fixtures are setup as classes.
They can have SetUp and TearDown methods as well as constructor and destructor methods. Below is a code snippet from the Primer showing the basic makeup of a fixture.

class QueueTest : public ::testing::Test {  
 protected:
  virtual void SetUp() {
    q1_.Enqueue(1);
    q2_.Enqueue(2);
    q2_.Enqueue(3);
  }

  // virtual void TearDown() {}

  Queue<int> q0_;
  Queue<int> q1_;
  Queue<int> q2_;
};

A fixture can then be used as part of test like this:

TEST_F(QueueTest, IsEmptyInitially) {  
  EXPECT_EQ(0, q0_.size());
}

Note the difference in signature between the earlier code example and the fixtures example (TEST vs TEST_F).

4) Handles exceptions and crashes well:

This is something I haven't put much researched into. At minimum, there are assertions to check that a piece of code throws exceptions. However, Google Test itself doesn't use exceptions.

As for crashing, time will tell.

5) Good assert functionality:

For exceptions, there are two kinds: ASSERT_* and EXPECT_*:

  • ASSERT_* generates fatal failures, killing the current test and and jumping to the next

  • EXPECT_* generate non-fatal failures which do not abort the current test.

The fact that ASSERT does not kill all the tests is important. It simply marks down the issues for the test report and carries onto the next test. This is preferred over the alternative (which is that all tests just stop after hitting one failed assert).

To extend the discussion further, there are many varieties of assert for specific purposes: floating-point (almost) equal assertions, thread-safe death tests, string comparison (string references). More can be found in the Advanced Guide.

6) Supports different outputs:

Test output can be printed to screen or exported as an XML test report. This is probably enough for what we're doing but if it can export XML, it is likely possible to get the test report in any other form.

7) Supports suites:

A test suite is a collection of test cases. Google Test makes the creation of test suites trivial.

A typical setup would involve a test directory where all test files are located (usually located in the project being tested). For each class, a test file (usually classname_unittest.cpp) is created, filled with whatever tests are required.

Next, a main file is created (or copied; it's almost always identical between different projects) which looks like this:

GTEST_API_ int main(int argc, char **argv) {  
    printf("Running main() from gtest_main.cc\n");
    testing::InitGoogleTest(&argc, argv);
    return RUN_ALL_TESTS();
}

This main file finds ALL THE TESTS and runs them. This is possibly the best part of the whole test system, there is no extra overhead to attach tests to be run. This means, a test can be written with minimal effort.

It is also possible to run different test segments individually if need be.

Conclusion:

Google Test appears to be well written, extensive, and easy to use. It is also maintained. While it may not be the best tool for every job (although it could be), for our purposes, it meets all the criteria.

For anyone having trouble installing Google Test on OS X (command line), here is a little guide.

Need help with a similar project?

Innovative Digital Art

Limbic Media

Consulting Engineering

Limbic Consulting