Testing APIs for Backwards Compatibility

A couple of weeks ago, I wrote about Agile Framework Development and the various techniques that we have used here at Caplin to build our APIs in a way that is sympathetic to agile methodologies. One of the key problems with the iterative development of an API is how to ensure that it remains backwardly compatible. Although each new release of an API is focussed on providing more functionality, easier use, or increased robustness for its users, those same users want their existing code to continue working when they upgrade.

This article addresses the issue of how to identify, with confidence, whether an API is backwardly compatible or not. As I mentioned previously there are times when an API is just too incorrect to be kept, however the decision for a breaking change must be a deliberate one – not an accidental one that is typically only discovered when the users of the API find that their existing code no longer works.

Preventing Breaking Changes

It is easy to tell developers that they should keep an API backwardly compatible, however even with the best of intentions it is possible to introduce subtle changes to the behaviour, particularly if you are refactoring collaborating classes that may not directly be part of the public interface. Every developer should be aware of the importance of keeping the API backwardly compatible, however this cannot be relied upon. After all, every developer should be striving to keep their code bug free, but we still write tests to verify this.

The solution is to detect any API changes before it is released, preferably in an automated fashion, so that it can be fixed. The problem is how to know whether an API has actually changed.

Compilation

A starting point for a compiled language, such as Java, is whether some code that was written against the old API still compiles against the new one. Any interfaces that have changed, including those with changes to method signatures and so on, will be flagged up immediately by the compiler. However this is no guarantee that the behaviour of the API hasn’t changed, and this simple verification isn’t available for non-compiled languages like JavaScript.

Unit Tests

The next obvious tool to detect a change are unit tests. The problem with this is that the unit tests will also be updated when the underlying code is modified. The fact that a unit test for a public API is having to be changed should start alarm bells ringing for the developer making the change; are the changes that they are making backwardly compatible or not? Again, however, the problem with this is that sometimes the changes can be subtle and the breaking change may be missed, particularly if the changes are made to a collaborating class rather that the one that is directly part of the public API.

Acceptance Tests

Acceptance tests can be used to detect breaking changes in a similar way to the unit tests. My problem with this approach is that they tend to be less granular and therefore susceptible to missing subtle breaking changes. Ideally I want my tests for API changes to have excellent coverage and to test all possible code paths, which is usually something that isn’t done by acceptance tests.

API Tests

Taking a step back, it does look like unit tests seem a good match for solving this particular problem. The only issue is that the unit tests will be modified when the code is changed, which may mask any breaking changes that have unwittingly been introduced. However we do have a set of unit tests perfectly suited to this which haven’t been modified, those from our previous release. We know that they proved the API for that release worked, so if the same tests pass against the new code it stands to reason that we should be backwardly compatible.

Creating API Tests

At Caplin we have a set of API tests that are a specialised group of unit tests aimed at verifying that all aspects and behaviours of the public API have not changed. They are usually a subset of the unit tests written for a particular library.

Only Running the API Tests

The first problem is specifying which tests are those for the API and which are standard unit tests. This is important since the implementation of collaborating classes can be changed without impacting the public API. As a result the old unit tests for refactored collaborating classes may now fail, however these are unimportant if they have no impact on the API.
Within our JavaScript unit tests we have added the ability to flag a particular test as an API test. We can specify whether we want to run only the API tests or all the tests, including the API tests.

Alternate approaches would be to create different test suites for the API and standard tests, or perhaps to use a custom annotation like @ApiTest rather than @Test in JUnit 4.

Dangers With Mocking

One danger with testing APIs is the use of mock objects. Mock objects combined with dependency injection provide a powerful way to test the logic of a particular piece of code as a single unit, without having to rely on the real logic in the collaborating objects. Instead the mock object acts as a replacement for the real collaborating object, verifying that it is invoked in the correct way and returns the appropriate values.

The danger is that changes to the collaborating object, such as the addition of a statement that throws a runtime exception if an argument is null, will not be emulated by the mock object. These problems are not insurmountable, however you need to be aware of the potential risks. In the case above there is a reasonable argument that there should have been an API test written for the collaborating class that demonstrated that the arguments could have been null previously which would now fail.

Getting the API Tests

At Caplin we upload the API tests to our Maven repository along with the other build artifacts for our libraries. For example when we release version 1.0 of a library, the API tests for 1.0 are uploaded to Maven. When we are building version 1.1 of the library we download the API tests for 1.0 to verify that the API has not changed.

Alternatives to getting the tests from Maven are to use another dependency management system, to pull the tests from a branch or label from source control, or even copying them from a network share.

Updating API Tests

It is possible that a public API hasn’t changed, however the API test for it fails. The reason for this could be that the interaction between the API method and its collaborating objects, particularly if they have been mocked out, have changed, and the test expectations are no longer valid. As a result the code for the API tests will need to be updated.

This change needs to be managed via source control, with a new version of the API tests pushed into Maven, or wherever they are being stored.

Backward Compatibility With Older Versions

At the moment we are only checking backward compatibility against the previous version of the API. I believe that if an API is compatible with the previous one, and that previous one was compatible with the one before, and so on, then it stands to reason that the current API should be compatible with the oldest one.

Conclusions

We are still in the early days of trying out this approach, however the early signs have been encouraging.

Ultimately a large amount of the responsibility still lies with developers to ensure that their code changes to a public API maintain backwards compatibility. The API tests are a safety net that gives confidence that it really is backwardly compatible and provide an early warning of any breaking changes, allowing them to be fixed before a release is made.

Any breaking changes within an API must be a conscious decision, not an inadvertent one.