Unit Testing with Google Test and GNATcoverage

In this blog post, we will explore the use of Google Test to drive our unit test cases, combined with GNATcoverage to measure our code coverage.

Introduction

Testing is an important part of every software development effort. Teams integrate different levels of testing into their processes. In this blog post, we will take a look at unit testing. Automation is crucial when doing testing. Ideally, a software developer changes or adds code, and the relevant set of unit tests is automatically executed to ensure that the change or addition does not have unintended consequences. When code is added, ideally, additional tests are added as well.

Testing by itself is not sufficient, though; you also need to understand how deep into your code your tests reach. At the very least, you need to know that every function was tested, but ideally, you would want to understand how many lines of code of your function were covered and/or if all the decisions and conditions in your if statements were covered. The amount of coverage information you need often depends on the regulations you must comply with. DO-178C DAL A requires the most thorough coverage requirements, for example.

Google Test and GNATcov

Google Test is an open-source, lightweight unit testing framework for C and C++, based on xUnit, similar to AUnit. Numerous third-party tools offer integration support directly for it, such as VSCode. GNATcoverage is a qualify-able coverage analysis tool for Ada, C, and C++. We will use GPRbuild to build our project, which is an open-source build tool for multi-language projects, used, for example, when projects want to combine C/C++ code with Ada code.

GNATcoverage works by instrumenting the source code in a fully automated fashion. It adds instructions that track which segments of the code were executed and collects that information when the application shuts down. It can do this on a host development workstation, but also on an embedded target. This instrumentation is very flexible and an easy way to obtain coverage information, but there is a run-time cost, so it will change the timing behavior of the application.

Basic Workflow

Moving on with our project, we will start with a simple GPRbuild project, structured as follows:

$ google_test$ tree
.
├── my_algorithms.gpr
├── my_algorithms.h
├── my_algorithms.cpp
└── main.cpp

The my_algorithms.cpp file contains a single function tested in main.cpp:

#include <my_algorithms.h>

int
algorithm1 (int a, int b)
{
  if (a == 0)
    {
      return a * b;
    }
  else
    {
      return a + b;
    }
};

The my_algorithms.gpr project file sets up the build, the Google Test dependency, and the coverage configuration. We assume that Google Test has already been installed globally.

The Coverage package uses Excluded_Source_Files to exclude main.cpp from the coverage report — we are only interested in the coverage of the my_algorithm.cpp file. GNATcoverage still processes main.cpp during instrumentation to insert what it needs for the data gathering.

project My_Algorithms is
   for Languages use ("C++");
   for Source_Dirs use (".");
   for Main use ("main.cpp");
   for Object_Dir use "obj";
   for Exec_Dir use ".";

   package Linker is
      for Default_Switches ("C++") use ("-lgtest");
   end Linker;

   package Coverage is
      for Excluded_Source_Files use ("main.cpp");
   end Coverage;
end My_Algorithms;

Google Test works through the definition of test cases in standard source code. The Google Test Primer is a great place to learn about how to write test cases. Typically, you would create a test directory and a structure of test cases underneath that, but for simplicity, we implement the test for the algorithm1 function in main.cpp.

We instantiate our own Google Test main and configure the global teardown procedure to dump coverage buffers, using the manual dump trigger mechanism. This requires inserting a /* GNATCOV_DUMP_BUFFERS */ annotation at the point where coverage data should be flushed. See the TearDown method below.

This is a very simple and basic setup and will suffice for our small test case. Rest assured that Google Test has elaborate capabilities to scale this to larger software architectures.

#include <gtest/gtest.h>
#include <my_algorithms.h>

TEST (AlgorithmTest, Algorithm1)
{
  EXPECT_EQ (algorithm1 (2, 3), 5);
  EXPECT_EQ (algorithm1 (-1, 1), 0);
}

class MyEnvironment : public ::testing::Environment
{
public:
  void
  TearDown () override
  {
    /* GNATCOV_DUMP_BUFFERS */
  }
};

int
main (int argc, char **argv)
{
  ::testing::AddGlobalTestEnvironment (new MyEnvironment);
  testing::InitGoogleTest (&argc, argv);
  return RUN_ALL_TESTS ();
}

Build

Now, with our source code ready, we can build and run our test cases. For this, we need to build and install the GNATcoverage runtime library. This is a one-time operation using gnatcov setup:

gnatcov setup

Then, to instrument the project we can call gnatcov. We pass --dump-trigger=manual because we rely on the GNATCOV_DUMP_BUFFERS annotation in main.cpp to trigger the trace dump:

gnatcov instrument -P my_algorithms.gpr --level=stmt --dump-trigger=manual

This produces instrumented source files under the gnatcov-instr subdirectory and a ‘SID’ file for my_algorithms.cpp in obj/. A SID file contains detailed configuration about the coverage obligations. With --level=stmt we instruct gnatcov to provide us with statement coverage, which is the lightest coverage option; other options could be stmt+decision all the way up to stmt+uc_mcdc, which stands for unused condition, modified condition/decision coverage on top of the statement coverage.

We then build the instrumented project with gprbuild, using --src-subdirs to pick up the instrumented sources and --implicit-with to link against the coverage runtime:

gprbuild -P my_algorithms.gpr --src-subdirs=gnatcov-instr
--implicit-with=gnatcov_rts

Run and Analyze

Running the instrumented executable produces a .srctrace file. In our case, we are running on the build host, but as mentioned before, this can happen on embedded targets as well, in which case we would have the added task of transferring the results file from the embedded target back to the host.

$ ./main
[==========] Running 1 test from 1 test suite.
...
[  PASSED  ] 1 test.

We can then analyze the trace with gnatcov coverage. We need to pass the project file to the analysis process. This points gnatcov to some of the supplemental files that were created during the instrumentation process, specifically the SID file(s) we created earlier:

gnatcov coverage -P my_algorithms.gpr --level=stmt *.srctrace -axcov

This generates a coverage report in xcov format in the obj/ directory, which is human-readable and explains which lines were covered and which were not. The covered lines have a ‘+’ in the margin, non-covered lines have ‘-’:

$ cat obj/my_algorithms.cpp.xcov
google_test/my_algorithms.cpp:
67% of 3 lines covered
67% statement coverage (2 out of 3)

Coverage level: stmt
   1 .: #include <my_algorithms.h>
   2 .:
   3 .: int
   4 .: algorithm1 (int a, int b)
   5 .: {
   6 +:   if (a == 0)
   7 .:     {
   8 -:       return a * b;
   9 .:     }
  10 .:   else
  11 .:     {
  12 +:       return a + b;
  13 .:     }
  14 .: };

Other report formats are also available as well, such as HTML files for easy visualization, report files to track violations, and SARIF files that can be loaded into VS Code as well.

Gnatcov can also generate machine-readable reports, which can be used in CI/CD pipelines and can be used to make policy-based decisions as to whether the changes can be merged into the codebase.

In the case of our project here, we can immediately see that line 8 was not covered. We need to elaborate our testcases to cover that line. The code makes it very easy; we need to add a test case where the first parameter is 0. In real-world applications, the team may need to think a bit harder to figure out how to add additional test cases.

Summary

Testing is important in both manual and automated workflows. Testing by itself is only the first step; GNATcov provides information on how good your test cases are and allows the team to quickly elaborate the test cases to increase coverage.

There are many more details to explore as part of introducing Google Test and GNATcov into a larger application. This blog post serves as a brief overview of how to combine Google Test with GNATcov.

GNATcoverage is available from open source and can be found here: https://github.com/AdaCore/gnatcoverage. GNATcoverage for C, C++, Rust, and Ada is also included in the GNAT Dynamic Analysis Suite from AdaCore, which provides commercial support as well as support to use the tools in programs that require functional safety certification.