In terms of formalised unit testing, we were fairly late to start generating repeatable test cases for our C products. Sure we wrote code to test our new features but they were just thrown away once the code was working.
Even so, over the past 6 years or so, as we’ve added features and fixed bugs we’ve diligently been adding tests. We’ve now got to the stage where we’ve got thousands of unit tests and hundreds of integration tests. For some products, the size of the test code exceeds the product code, admittedly not to the scale of Sqlite though.
To do this effectively, we’ve adopted check as our unit test framework and have incorporated the validating mock library from cgreen as well as our own mock JVM implementation that performs validation over the JNI interface. We’ve even changed our coding style, changing it so that the links between compilation units can be replaced at run time allowing release binary testing which is a huge confidence boost over the traditional C testing method of producing an individual binary for testing each compilation unit.
Testing the Released Binary
Testing the released binary is a neat idea (subject to usual caveats of how to obtain coverage) – it means that you’re delivering to your end user compiled code that has actually been executed and tested as opposed to trusting the compiler not to change something between the test compile and the release compile (or more likely, some preprocessor macro changing the code path). Testing the release binary is something that we do with our .NET and Java products and so it would be great if we could do the same thing with our C products without needing to rewrite them all.
Coming across TypeMock which promises this functionality for Windows led me to think about how you could achieve the same thing on a Linux system and integrate it into the rest of our test tools.
What you need to do may be inherently non-portable and given that we need to build for so many platforms it may not be something that we pursue, but it’s an interesting thought exercise. So here are my thoughts on the ways it could be implemented.
The high level concepts that we need when testing on a release binary involve overriding the implementation with mocks and trapping changes to any global variables. Since we’ve already got a mock framework we can call that when we’ve performed the appropriate traps.
My initial thoughts were around manipulating the ELF relocations in the binary to point to mock functions, however I suspect that for running multiple test cases the overhead of reloading the binary and performing the manipulations may be excessive.
My current thoughts are revolving using the functionality present in gdb. With gdb you can trap access to global variables using watches and stop execution using breakpoints. If we can intercept these traps and call our mocks we’ve achieved our aim. The functionality of adding breakpoints and watches appears to be exposed in libgdb which means that we should be able to do this in a portable way.
Performing unit testing on the release binary opens up a new range of testing strategies and would allow the testing of legacy code which is usually considered unviable for testing. Once the test coverage is in place, the code can be safely refactored to a more testable design.
Should I get a chance to try out this technique, I’ll report back, hopefully with an open source project for all to use.