Basic testing with Catch2

Introduction

The easiest way to test software is to write a bunch of assertions. These are simple statements which assess that given certain predetermined inputs, your code should behave in a certain way (produce a certain result, throw an exception…).

It is certainly possible to write basic assertions by hand…

// Assume this is the code we want to test
int triple(int x) {
    return 3 * x;
}

// Somewhere else in the source tree, we might write this test program:

#include <exception>
#include <iostream>
#include <sstream>

int main() {
    const int result1 = triple(0);
    if (result1 != 0) {
        std::ostringstream s;
        s << "expected triple(0) to be 0, but got " << result1;
        throw std::runtime_error(s.str());
    }

    const int result2 = triple(1);
    if (result2 != 3) {
        std::ostringstream s;
        s << "expected triple(1) to be 3, but got " << result2;
        throw std::runtime_error(s.str());
    }

    const int result3 = triple(2);
    if (result3 != 6) {
        std::ostringstream s;
        s << "expected triple(2) to be 6, but got " << result3;
        throw std::runtime_error(s.str());
    }

    std::cout << "All tests completed successfully" << std::endl;
}

…but as you can see, this gets verbose quickly, and we have not even started introducing other commonly desired test harness features like…

Being able to run all tests even if some of them fails and report on all failing tests at the end.
Being able to selectively run some tests, without running the other ones.

Therefore, you will soon want to introduce some abstractions that reduce the amount of code you need to write for each test. Thankfully, you do not need to write this code yourself, as there are many high-quality test frameworks available for C++. In this practical, we will use Catch2.

Using it, the above test program becomes…

#include <catch2/catch_test_macros.hpp>
#include <catch2/catch_session.hpp>

TEST_CASE("triple() multiplies a number by 3") {
  REQUIRE(triple(0) == 0);
  REQUIRE(triple(1) == 3);
  REQUIRE(triple(2) == 6);
}

int main(int argc, char* argv[]) {
  return Catch::Session().run(argc, argv);
}

…which as you can see is a lot more readable, focused on the task at hand, and yet more featureful (you get the aforementioned test harness features for free, among other).

Basic Catch2 usage

In Catch2, as in many other test frameworks, testing is done using test cases, which are code blocks described using a string and containing more assertions. If one of the assertion fails, you will automatically get an error message telling which test case(s) failed and why:

Randomness seeded to: 3094418497

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_triple_catch.bin is a Catch2 v3.5.3 host application.
Run with -? for options

-------------------------------------------------------------------------------
triple() multiplies a number by 3
-------------------------------------------------------------------------------
test_triple_catch.cpp:7
...............................................................................

test_triple_catch.cpp:9: FAILED:
  REQUIRE( triple(1) == 3 )
with expansion:
  2 == 3

===============================================================================
test cases: 1 | 1 failed
assertions: 2 | 1 passed | 1 failed

Test cases terminate on the first failed assertion, so if you want to test multiple mathematically unrelated properties, it is a good practice to test them using separate test cases. But for any nontrivial test, doing this naively means duplicating common setup code. The standard way to avoid this in Catch2 is to use sections:

#include <catch2/catch_test_macros.hpp>
#include <catch2/catch_session.hpp>

TEST_CASE("std::vector size/capacity matches expectations") {
  // Common setup
  std::vector<int> v(42);
  const size_t initial_capacity = v.capacity();

  // Common expectations
  REQUIRE(v.size() == 42);
  REQUIRE(initial_capacity <= 42);

  // Section that tests push_back
  SECTION("adding an element increases the size, may affect capacity") {
    v.push_back(123);
    REQUIRE(v.size() == 43);
    if (initial_capacity >= 43) {
      REQUIRE(v.capacity() == initial_capacity);
    } else {
      REQUIRE(v.capacity() >= 43);
    }
  }

  // Section that tests pop_back
  SECTION("removing an element decreases the size, capacity is unaffected") {
    v.pop_back();
    REQUIRE(v.size() == 41);
    REQUIRE(v.capacity() == initial_capacity);
  }

  // Section that tests clear
  SECTION("clearing the vector zeroes the size, capacity is unaffected") {
    v.clear();
    REQUIRE(v.size() == 0);
    REQUIRE(v.capacity() == initial_capacity);
  }
}

int main(int argc, char* argv[]) {
  return Catch::Session().run(argc, argv);
}

From this code, Catch2 will generate three test cases that begin by performing the same initial setup (and initial assertion checking), and then go on to each test how a different method of std::vector affects the vector’s size and capacity.

For more information on the functionality provided by Catch2 and the way it is used, please refer to the library’s reference documentation.

Exercise

Set up Catch2 on one of the example programs provided as part of this course, or a program of your own, and write some test cases with assertions. If you are not using the provided Makefiles, bear in mind that you will need to add -lCatch2 to your compiler flags in order to link to the Catch2 library.

Robust Programming

Basic testing with Catch2

Introduction

Basic Catch2 usage

Exercise