Why 100% Code Coverage is not enough

Introduction

When it comes to software testing and code quality it is not always clear what the Test Coverage is – respectively what areas of the application need more attention regarding testing.

Questions like:

• Do we have enough tests?
• Do the tests cover enough paths of the code?

are quite common but tricky to answer.

Measuring the Code Coverage seems to be the perfect solution. But to be honest – I am not absolutely convinced and do not trust these numbers alone.

If you write your Tests just to meet the coverage target, you might miss some critical tests around edge cases and unexpected exceptions which will lead to a wrong behaviour of the application under test.

Martin Fowler’s blog about  Test Coverage gives a good theoretical overview about this topic.

I would like to go into more detail by showing a code example pointing out that 100% Code Coverage of a class is not enough and should not stop you from using your mind and experience to continue writing some more tests. But at the same time if you do not have any ways to track the test coverage, it could be equally dangerous too.

Code example

The TestCalculator class below represents a simple add method:

 public class TestCalculator {
     public Double add(Double a, Double b) {
         return a + b;
     }
 }

Let’s have a look at a simple JUnit test for this method.

@Test
public void testAdd() {
    Double a = new Double(1);
    Double b = new Double(2);
    Double c = new Double(3);
    assertEquals(c, testCalculator.add(a, b));
}

When we run the test in Eclipse by using the EclEmme Code Coverage Plugin we will get 100% Line-Coverage for this class.

That sounds perfect – doesn’t it?!

Well, as a curious tester I might want to add another test. I am interested how the application behaves when one of the parameters is null.

@Test
public void testAddNullPointerException() {
    Double a = new Double(1);
    Double b = null;
    Double c = new Double(3);
    assertEquals(c, testCalculator.add(a, b));
}

There we go, the test fails with a NullPointerException although we have 100% Code Coverage.

Conclusion

As a conclusion I would like to encourage everyone involved in development and testing activities to think a bit more out of the box. I do not say that you should not use any of the Code Coverage tools. But I recommend not to rely on the numbers only.

It is also worth thinking about the interfaces and how components interact with each other (Integration Testing). Maybe the component calling the method on the TestCalculator validates the parameters before using them or there is a contract (Test by contract) that defines which component is responsible for the validation. Branch and Path Coverage of the code are also of interest to get more information about different scenarios and data values. However this will be discussed in a separate blog post.

So just relying on numbers without knowing what is going on behind the curtain is not an indication of good quality reporting. High code coverage is not an indication of good quality but low coverage is an indication that something is wrong, untested and worth investigating.