At Caplin we have a lot of products which do not have user interfaces. Of course, a user interface for a internal component makes no sense, they are designed to be used by other components, who themselves may not have user interfaces. This brings up an interesting problem. How does one perform exploratory testing on such components?
The Problem
While acceptance tests are a very important feature of a regression suite, they are not intended to discover new bugs. They will test the same thing again and again tirelessly but this means they will also miss the same bugs again and again. Bugs often manifest when we perform regular actions in a peculiar or unpredicted way such as clicking the right button at the wrong time or doing the right things in the wrong order. So it is important that we use other methods such as exploratory testing to find these bugs.
Of course, this is pretty easy to test with GUI applications. We can click different buttons and enter test data into different fields to expose bugs. There are also many resources of different things to test for such as lists of SQL injection code and so forth. It is still important to thoroughly test the application by performing actions which the user may perform in as many different scenarios as one can imagine. This can be tricky in practice as the user can often be unpredictable and it is important that you do not simply perform the same tests repeatedly. It is often worth getting “a fresh pair of eyes” to test the system; often a user who has no idea what the system does will find a new bug. In fact for the Microsoft Surface, Microsoft even created a framework to simulate hundreds of random presses for stress testing their Microsoft Surface (http://www.youtube.com/watch?v=g4xxNpijEKQ&t=7m11s). This kind of test will definitely expose some issues that may be present in your application.
But what about components that don’t have a user interface? There is no easy way to test these products. They must be built into some kind of other application which has a user interface of some kind, but testing at this level it can be very hard to link the cause of effect of bugs. “The button didn’t do anything; Now is that because of component A, B or C? Or was it just because the button has been disabled on the view layer?”
Of course, every component must have some kind of interface, there wouldn’t be much point to it if it never communicated with anything. And we can quite easily create a mock of an object which can communicate with the products interface, this is after all what we do when we are writing our unit tests as part of Test Driven Development.
However, as each class is unit tested in isolation to the others, errors can often go unnoticed. And while ATs are meant to catch these errors, they often have trouble finding more subtle bugs. For this reason it is very important that we perform exploratory testing.
The Solution?
It is important that we are able to create a exploratory testing environment which allows us to quickly test these interfaces in the same way that we test graphical interfaces. And why can’t we do this by creating a GUI, allowing us to perform different tests at run time rather than having to define all our tests before hand. Of course creating a “useless” GUI for every component and updating it every time the product updates is an effort. But perhaps there is a better way. It should be possible to create a GUI which uses reflection to extract all the available methods from a component at run time. This product would be able to be used on any product without needing to be updated.
In fact products similar to this already exist. For example the much maligned BlueJ IDE is intended for teaching Java and programming principles. While some argue that it is not good at this particular task, it does provide the functionality of allowing users to execute methods at their own pace and direction, ideal for exploratory testing.
A more testing oriented version of this application may give us the kind of functionality that we require for quickly and effectively performing exploratory testing on non-GUI components. To perform exploratory testing on one of these components we need to come up with a unique solution. And creating direct interface for us to get immediate feedback seems to be the perfect solution. Stay Tuned.