Recently we had a discussion about testing all possible combinations within a FX Trade Ticket.
To give a context, a trade ticket consists of several parameters like:
- Trade Type (SPOT, FORWARD, SWAP, TIMEOPTION)
- Currency Pair (BASE, TERM)
- Direction (BUY, SELL)
If we combine all possibilities we will have to execute 16 tests.
TYPE CURRENCY DIRECTION:
- SPOT BASE BUY
- SPOT BASE SELL
- SPOT TERM BUY
- SPOT TERM SELL
- FORWARD BASE BUY
- FORWARD BASE SELL
- FORWARD TERM BUY
- FORWARD TERM SELL
- SWAP BASE BUY
- SWAP BASE SELL
- SWAP TERM BUY
- SWAP TERM SELL
- TIMEOPTION BASE BUY
- TIMEOPTION BASE SELL
- TIMEOPTION TERM BUY
- TIMEOPTION TERM SELL
During this discussion I remembered a Tech Talk I went to last year about Pairwise and Combinatorial Testing and I started to do some more research on it.
What is Pairwise Testing?
Pairwise Testing is a software testing approach that can help to test a big number of input parameters which normally results in hundreds/thousands of different permutations.
A good example to show the immense number of test cases is a configuration dialog with 12 checkboxes (checked, unchecked) and a dropdown with 3 values.
If we wanted to cover all possible combinations we would have to execute 12,288 different tests which is calculated by (2 power n) * 3.
Amazingly these 12,288 tests can be reduced to 10 if we apply the 2-way pairwise testing strategy.
See http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
How does it work?
Pairwise testing is a way of testing a big set of parameters by only testing pairs of the given parameters. If we look at the example above we can see that there are 2 tests for SPOT, BASE (SPOT-BASE-BUY, SPOT-BASE-SELL).
One of the inspirations for Pairwise Testing is the fact that bugs occur more often on a pair or triple of parameters rather than on a single input parameter. Therefore it is quite popular and often used for configuration testing but it can also be applied to any other area of software quality assurance.
This sounds all quite magic. So I would not recommend to blindly apply this strategy without analysing the software under test and getting some more information about architecture and risk before using it.
More information can be found on the following pages:
What I personally find quite interesting is the combination of a pairwise analysis with exploratory testing. By getting a good idea of the input parameters it can definitely support exploring the application.
Finally I would like to show the result of the 2-way Pairwise Testing strategy applied to the example above.
2-WAY TYPE CURRENCY DIRECTION:
- SPOT BASE BUY
- SPOT TERM SELL
- FORWARD BASE SELL
- FORWARD TERM BUY
- SWAP BASE BUY
- SWAP TERM SELL
- TIMEOPTION BASE BUY
- TIMEOPTION TERM SELL
Thomas,
Great article on an important topic.
Thanks for mentioning out Hexawise test case generating tool. We’re comitted to both continuing to make it available for free to most of our users and to using the majority of the revenue we get from our paying clients (primarily Fortune 500 firms) to improve our tool. We’ve tried to make it both the most powerful tool of its kind and the easiest to use.
Extremely few commenters have made the very important point you did: “What I personally find quite interesting is the combination of a pairwise analysis with exploratory testing. By getting a good idea of the input parameters it can definitely support exploring the application.”. You’r absolutely right. You can, for example, add variety to your tests by adding non-traditional test inputs like user style: direct, indirect / user starts out the process in what kind of mood: happy, inquisitive, determined to do something dramatic, etc. When you add factors like that, you’ll find that – even though they are vague – they’ll guide your explorations of the application in interesting directions. Whether or not you add such variation-triggering ideas as test inputs, you can (and, in my opinion, should almost always) intentionally include Exploratory Testing strategies to explore beyond the confines of the test conditions generated by your tool of choice.
Justin Hunter
Founder of Hexawise