In the same way that design by contract can work to interfaces, with a new or existing system testing by contract can work to the interfaces between subsystems. These interfaces are not necessarily coding interface classes, but they are the externally exposed APIs of the different subsystems.
An example
Consider an example where there is a frontend GUI, a communication layer, and a backend system including a datastore. There are two interfaces in this system: one between the GUI and the communication layer; and another between the communication layer and the back-end system. This provides us with three subsystems that can be tested by contract.
Assuming that there is a publicised and reviewed external interface API at each of the integration points then rather than testing the whole system, or trying to test the back-end via the front-end, or the communication layer via the GUI, or the back-end via the communication layer, it should be possible to test each subsystem by contract.
The GUI should test all of it’s own functionality, but provides mocks for both the communication layer inputs to the system, and outputs to the communication layer. These mocks should be based on the public interface contract that represents the communication layer’s public API, to the GUI side of the system.
Testing the GUI by contract means that it can be tested independently of any other subsystem. It will be dependent only on it’s own code, and it’s own infrastructure. This should ensure that any bugs can be found quickly and efficiently, and that they are genuine bugs with the GUI, rather than integration issues with a complicated environment.
Testing the communication layer by contract, means that contracts must be considered and defined for each side that the communications layer communicates with; but it doesn’t have to actually connect to them in order to do its own testing. Again mocks can be used to simulate the other two sides, and the communication layer can then be developed and tested in isolation. This will mean far less resources will be required for testing, and developers will get quick feedback on their work, without the need to ensure ongoing compatibility with the latest versions of other teams code. This prevents any bugs in external code from affecting the test results, allowing accurate and timely results to be obtained.
Testing the back-end by contract can then work in a similar way to the GUI testing. The back-end can concentrate on testing its own code in isolation while relying on mocks to represent the public API contract of the Communication layer.
Third party dependencies
There may also be third-party dependencies within the system. E.g. the communication layer may be http/https. In this case the assumption is made that the third- party dependency (in this case the communication layer itself) has been tested by the provider/maintainer of that third party dependency.
In this instance, consider a back-end that will produce an XML file. The back-end contract will assume that once the XML file is provided to the communication layer, it will be delivered with the header sent to the browser. At the browser, the GUI testing by contract can be based on a static XML file, representative of the contents of the file that the back-end is contracted to produce, and that the communication layer is assumed to deliver.
Integration Tests
You still need integration tests across the whole stack, but far fewer, and only to find integration issues, not subsystem level functionality issues. Also these will not be run until each subsystem has been validated in isolation. This ensures that integration tests will find integration issues. There may still be emergent issues when the whole stack is used together: but if each subsystem has been validated in isolation, then any integration bugs are just that; due to integration, not any of the subsystems.
Each subsystem is tested according to it’s mocks which are coded according to the interface definition. The purpose of the integration test is to ensure that both sides agree on that interface definition. This ensures that the mocks are correctly mimicking the behaviour of the real system. If any issues are found the mocks should be updated accordingly. There may be timing issues and hidden semantics as well as the method signatures themselves.
Advantages
There are several advantages to testing by contract. The main ones for us are:
- Isolation: Each subsystem can be tested in isolation
- Speed: Testing an individual subsystem will be much quicker than testing the whole stack
- Granularity: Bugs found are guaranteed to be bugs with the current subsystem, not integration bugs or environment issues.
- Separation of concerns: Integration bugs are just that, since the individual subsystems should all have been tested in isolation first.
- Early validation: Defining the contracts up-front should highlight potential issues at design time
- Total Cost of Ownership: Finding bugs earlier in the design process, or within a specific subsystem, or as a known integration issue makes them easier to isolate, track and ultimately makes them much cheaper and easier to fix.
- Reliability: Integration tests tend to be the most unreliable and costly type of test to maintain.
You may find others, or find that your focus is on particular aspects of those listed above.
Our own experience
To take an example from our own experience we created a business logic testing framework behind our GUI. We treated the GUI as one subsystem, and the business logic behind that GUI, and before the transport layer as another subsystem.
Doing this enabled us to reduce the overhead for cross-browser testing hugely. We previously had a suite of VM stacks (one for each browser-type we wanted to test), and a set of tests that had to run overnight, and which would still experience numerous environment related issues. We now have a set of tests that can run on a single machine in around 10-20 minutes. Developers can run the same set of tests on their own machines in 2-5 minutes, allowing them to do so before each check in without any onerous waiting time.
A lot more than just the idea of testing by contract was needed to achieve this huge reduction, but testing by contract was the vital idea that allowed it to happen.
The testing framework we created turned into a major part of Verifier which is now a part of BladeRunner.