Mutation Testing – Killed or Survived?

This blog is a follow up from “Why 100% Test Coverage is not enough“.

Mutation Testing is a method of software testing where the source code gets modified step by step.

If there are no failing tests after a mutation to the code it is considered defective, meaning that none of the existing tests detected the change to the source code which brings a possible risk to the application.

Some of the mutational operators are:

• statement deletion
• replace each boolean subexpression with true or false
• replace each arithmetic operation with another one
• replace all boolean relations with another one e.g > with >=

In addition to traditional line and branch coverage, mutation tests will highlight the quality of existing tests and potentially missing tests.

Example

Let’s have a look at a simple example I have prepared using PIT Mutation Testing.

public String getSomething(int someParameter) {
    if ( someParameter > 0 ) {
        return "foo";
    } else {
        return "bar";
    }
}

@Test
public void testOne() {
    assertEquals("foo", mutationClass.getSomething(1));
}

@Test
public void testMinusOne() {
    assertEquals("bar", mutationClass.getSomething(-1));
}

If we now run the unit tests we will get a line and branch coverage of 100%.
However if we run the tests with PIT we can see that there is a test that survives a mutation.

From the result we can conclude the following. PIT mutated the code and changed the if-statement to “someParameter >= 0”. As our tests only test 1 and -1 we are missing to test the value 0.

So let’s add another test.

@Test
public void testZero() {
    assertEquals("bar", mutationClass.getSomething(0));
}

The result of the second run is:

Rather than adding another test I could have just replaced the -1 with 0.

Conclusion

I think mutation testing is an interesting technique to explore the behaviour of the application under test and the existing tests in connection with small code changes.  Whether you do it manually by changing some statements and variables or with an existing tool. Sometimes a developer makes a tiny change by replacing  “>” with”>=”, runs the tests and everything passes although the tests might not be designed for this change.

A line and branch coverage of 100% does not say anything about the quality of the tests. Getting feedback about the quality of the tests in addition to the coverage is absolutely vital. The example shows that choosing arbitrary values will result in passing tests but with a risk to future changes. Applying a boundary-value analysis for example would be a first step to improve the quality of the tests. And that is exactly what the mutated code illustrated.