Thus far, automating the tests and deployments that run on the VMs has been relatively simple, using GO agent services as proxies to carry out scripted commands on whatever machine that you want the build to run on. The problem with automating these tasks, is the constant battle with the state of the build environment.

Saving the environment!

At Caplin, we use VMware vSphere to manage the VM farm. This gives us the option to snapshot each machine to a known state. In theory, an automated build should pass, when it is running on a clean environment. However, browsers cache information, Maven builds cache artifacts, files can get deleted, people can add or remove programs – very quickly, a VM can become a complete mess.

Via the GUI, it’s very simple to click “Revert to snapshot”. However, this is hardly convenient to do on a regular basis, especially when you have 50 VMs you want to revert at a time.

Rolling out changes

Another headache is upgrading all of your VMs. Should you wish to update Java across a group of VMs, you have to;

1. Revert every VM to snapshot

2. Either manually install Java yourself, or run a loop over SSH

3. Delete every snapshot

4. Create new snapshots

Doing this manually is a colossal job that can take several hours, and is subject to human error.

Another option is to edit the template that the VM is based on, delete all the VMs and redeploy them. Again, a mammoth task to repeat through the GUI.

Let the computer do the work

Raising our concerns with Vmware about the challenges we were facing, they suggested that we script our tasks using PowerCLI. PowerCLI is a snap-in that lies on top of Windows Powershell. It’s a well documented set of cmdlets that can do anything that vSphere manager can do, and more besides. It’s 100% script based, which makes it perfect for automation.
I had never used Powershell before, so I was sent on a 2 day course in London with a VMware specialist, who could teach me to effectively use PowerCLI.

Back to school

The course I went on was titled “VMware vSphere: Automation with vSphere PowerCLI”. When I entered the classroom, I was very surprised to find that I was the only person taking part that week. It was not advertised as a 1-on-1 tutoring arrangement, but due to my lack of Powershell experience (I’m a bash man usually), it was a very convenient arrangement.

I picked up the basics of Powershell very quickly by following a training handbook that gave you tasks to follow. I found it to be a very effective method of learning – the trainer briefly explained the concepts of each section, answering my varied questions about the tasks, and how PowerCLI could be used at Caplin.

The first day mainly covered the creation of virtual hardware (datastores and virtual network interfaces), I wasn’t quite as interested in this aspect of PowerCLI, but it was a great way to get to grips with the Powershell syntax.

The second day was when I made the most notes, it covered reverting a VM (or a group of VMs from a CSV file) to a specified snapshot, deploying VMs, checking to see if VM hardware was consistent, running remote scripts on VMs via VMware tools and producing reports about the VMs.

I recieved my certificate for completing the course material, and left the classroom with a confident grasp of Powershell and PowerCLI.

Sadly, the course that I went on no longer exists (I was the last one to take the course in London!), however they are planning on starting a new course that uses an updated version of ESX. I was highly impressed with the VMware course material and the professionalism of the instructor. I would happily recommend them to anyone that is interested in learning.

The aftermath

Currently we have a pipeline in GO dedicated to reverting a group of VMs to snapshot every Saturday morning, this has increased the stability of the build. A more ambitious task that I’ve undertaken is to automatically revert a specific VM when a build pipeline fails in Jenkins. I have published the solution online – simply google “vmware revert jenkins slave to a snapshot” and it should be one of the top results.

PowerCLI is an invaluable tool for anyone that is managing more than 1 VM at a time, I find it hard to remember how we managed without it.

There’s a flurry of activity when projects kick off, resources have to be marshalled and often teams need to be informed of a good deal of information around the thinking behind the work at hand.  As that ship leaves port upon its venture for treasure (aaargh!), point to point conversations are difficult to manage across a sizable group.

For companies on the move, it raises the ever present question of how to scale agile practices.  For smaller teams in the same room or space, these conversations often go on in the open.  But as you delegate tasks and a set of leads begins to occupy their own time managing tasks and activities others don’t have regular exposure to, you can start to run into problems.

The start of a project is significant.  It’s where plans meet action, where the rubber meets the road.  You may notice one symptom of project failure when problems arise, decisions are made and after a few cycles the original aims and expectations get modified or lost.  Team members regardless of role are left wondering, “Why are we doing this?”

Fail to establish a clear set of facts about the state of affairs at the beginning of a project and you can kiss many things good bye.

What things?

Accountability, the ability to gauge plan vs reality, a distinction between assumptions and truth, feedback cycles that would support continual improvement across various roles and the ability for an organisation to improve its effectiveness in managing its own investments in a development effort.

As agile organizations scale, groups of people can often fall prey to the same pitfalls that the basic agile techniques work to avoid.  Techniques that were effective in small groups lose their impact as people bring old but trusted behaviours to new situations.  For example, I’ve not been a fan of scrum of scrums and have found them to be less effective as the mental maps—the views of the state of affairs—across participants are not aligned.  I’ve witnessed numerous cases where a group of people can earnestly agree around a table and then go off to work independently at cross purposes.

What to do about it

Establish a core set of facts. Agree on the state of affairs at any one time.  Start by asking the basic questions and make sure you can construct a view of the basic answers.  For example, each PM has their own check list of essential questions they ask to gauge the viability of a project.

In the early phases of project conception, these might entail questions like:

At the lead up to kickoff these might include:

Questions Common Across Stakeholders

And, regardless of role, there are questions common across stakeholders—for anyone who has an interest, stake or dependency on the outputs:

Our Approach

At Caplin, we leverage the concept of incremental planning. Our approach is adaptive and solves the issue at hand without a heavy amount of process.  We’ve just gone through the kick-off of a large program composed of many different projects. To manage this, we’ve created a kickoff dashboard where an initial plan can be viewed against a revised plan as we learn and discover.

How did we build it?  We started by asking the basic questions to check for ourselves whether we had the answers.  It was an essential first step where, the more we gathered this basic info, the more productive our discussions as a group became.

After you have established this agreed upon set of facts, what do you do next?  Argue! 

Have a vibrant exchange of ideas that evaluates alternatives and works to select the best one.  Remember to do this as a group and to speak to an agreed upon established set of facts. When these facts and agreements are not present often it can seem like conversations go one with no resolution and you feel like you are trapped within the eternal recurrence of the same.  The next time you feel this happening sit back and take a moment to ask yourself how much of the discussion is conflict over the state of affairs and what portion is a discussion of what to do about it.

Conclusion

Many agile practices are about having the right conversation at the right time and at the appropriate level of detail.  Many techniques are simple.  When your organisation scales and people are relocated out of earshot to what used to be a group conversation, the next simple step you can take is to have that quick conversation to agree on the state of affairs. The key is to remember to ensure that there is this preliminary agreement across your group.

The easiest way to make sense of a lot numbers is to turn them into a picture. This doesn’t come as much of a surprise: While it takes a lot of conscious effort to see trends, movements and tendencies within a large set of figures we find it rather easy to deduce the same from a visual relationship.

Consequently, financial professionals find themselves surrounded by all kinds of charts and diagrams every day. Line Charts to visualize performance, candlestick charts to show the relationship between intraday movement and open / close rates, heat maps to visualize volume and so on…

Fortunately, Web Browsers provide a wealth of functionality to render such charts. SVG and Internet Explorer‘s early equivalent VML have been providing vector based graphics for ages, HTML 5’s canvas element offers a very rich pixel drawing API and even CSS can be used to draw less complex visualisations like bar charts and maps.

That’s all good and well as long as we are dealing with a one to one relationship between two values. One stock, one price, at one time. But how can one handle more complex relationships, say, an option’s price.

An European option‘s price depends on at least two factors: The price of its underlying asset and the time that’s left for the option to mature. In addition there are other static factors to be considered like the historic volatility or the risk free rate. Elaborate mathematical models like the Black-Scholes formula have been developed to price these options.

Visualising these prices however was up to now restricted to highly specialised software products. But with the advent of HTML 5 and more powerful web browsers we now have the tools at hand to calculate and visualize such models directly on the client side. WebGL allows the browser to directly access the computer’s hardware and to utilize its unrestricted processing power.

But with these new possibilities also come new challenges for both developers and designers. Utilising the 3D space can easily become a very confusing experience for the end-user. And while frontend-engineers need to familiarize themselves with an as yet largely unknown field it is especially up to the UX departments to find beneficial and sensible solutions for this wealth of possibilities.

When it comes to software testing and code quality it is not always clear what the Test Coverage is – respectively what areas of the application need more attention regarding testing.

Questions like:
- Do we have enough tests?
- Do the tests cover enough paths of the code?
are quite common but tricky to answer.

Measuring the Code Coverage seems to be the perfect solution. But to be honest – I am not absolutely convinced and do not trust these numbers alone.

If you write your Tests just to meet the coverage target, you might miss some critical tests around edge cases and unexpected exceptions which will lead to a wrong behaviour of the application under test.

Martin Fowler’s blog about  Test Coverage gives a good theoretical overview about this topic.

I would like to go into more detail by showing a code example pointing out that 100% Code Coverage of a class is not enough and should not stop you from using your mind and experience to continue writing some more tests. But at the same time if you do not have any ways to track the test coverage, it could be equally dangerous too.

Code example

The TestCalculator class below represents a simple add method:


public class TestCalculator {
public Double add(Double a, Double b) {
return a + b;
}
}


Let’s have a look at a simple JUnit test for this method.


@Test
public void testAdd() {
Double a = new Double(1);
Double b = new Double(2);
Double c = new Double(3);

assertEquals(c, testCalculator.add(a, b));
}

When we run the test in Eclipse by using the EclEmme Code Coverage Plugin we will get 100% Line-Coverage for this class.

That sounds perfect – doesn’t it?!

Well, as a curious tester I might want to add another test. I am interested how the application behaves when one of the parameters is null.


@Test
public void testAddNullPointerException() {
Double a = new Double(1);
Double b = null;

Double c = new Double(3);
assertEquals(c, testCalculator.add(a, b));
}

There we go, the test fails with a NullPointerException although we have 100% Code Coverage.

Conclusion

As a conclusion I would like to encourage everyone involved in development and testing activities to think a bit more out of the box. I do not say that you should not use any of the Code Coverage tools. But I recommend not to rely on the numbers only.

It is also worth thinking about the interfaces and how components interact with each other (Integration Testing). Maybe the component calling the method on the TestCalculator validates the parameters before using them or there is a contract (Test by contract) that defines which component is responsible for the validation. Branch and Path Coverage of the code are also of interest to get more information about different scenarios and data values. However this will be discussed in a separate blog post.

So just relying on numbers without knowing what is going on behind the curtain is not an indication of good quality reporting. High code coverage is not an indication of good quality but low coverage is an indication that something is wrong, untested and worth investigating.

Will you be using third party tools or writing your own framework, or just writing to log files and then analysing them?

What are you going to use to do your testing and in particular will the performance monitoring be separate to the application or embedded within it. Even if you use third party tools it may still be necessary to embed links or functions within your code. Alternatively it may be possible to produce all the data you need to analyse your application using external tools. Some tools/alternatives to consider are:

•  JMeter
•  LoadRunner
•  Dynatrace
•  Write your own code/framework – how?
•  Writing a simple log method, e.g. in java that can be called from anywhere (so a public static method) can work very well.

Environments

You also need to consider the production and testing environments.  Do they need to match exactly, or is closely enough, or does it not matter?  Do both the server and client(s) need to match?  And OS?  Browser version(s)? Network?  Other environmental variables?

What server will be used to host the application. Will this be a VM or a physical machine? Does this match that which will be used in deployment. Is testing on a developers local machine a reasonable test, or is a separate custom built performance testing machine required. Even if a separate performance testing machine is available and used does this replicate the same constraints as the deployment hosting environment.

Observer effects

An important issue to consider is: How much does adding performance testing monitoring degrade the performance of the application under test? Is this another example of the more you observe the less you (can) know? Also even checking server logs can be inconsistent or provide ambiguous results. Sometimes the server has to be shutdown in order to flush the logs to file so that you can read them. On Windows machines when looking at the log file could lock the file and mean it can’t be written to. And also if you already have the file open, and then open it again it may show you the already open version rather than the updated version.

Acceptance Criteria

One area that has particularly not been covered is the definition of Acceptance Criteria for performance tests.  Like any other tests we need to define what ‘done’ is, and in broad terms for performance testing this will be when the System Under Test is appropriately responsive.  In practice this is likely to be converted into specific timings for specific functions.  Exactly how long and which functions will necessarily be a part of the particular System Under Test.  Naturally defining these criteria is left as an exercise for the reader.

More to do

I am sure there are many more factors that could influence performance in particular circumstances, and this list is by no means meant to be exhaustive.  Hopefully it will help to suggest areas that need to be considered when thinking about performance testing, and help to stimulate ideas for different ways of approaching performance testing.  There is also the question of how to incorporate this into regression testing for version 2.0 and onwards, once a System Under Test has been performance tested?

Phoney

Barclaycard has a rather strange solution to this that they have just announced, a ‘Pay Tag’, basically a sticker that you can “decorate” your phone with and turn it in to a kind of oyster card to pay for items £15 or less, no PIN, No security checks. They have teamed up with Orange and use the phone for authentication, using all the data for your contract to prove your identity, making it far more fraud-proof apparently.  Why you would want to stick this on your phone still seems a bit unclear, why not stick it to your wallet or your hand… actually you can now get chips as wearable temporary tattoos …no really see for yourself.

Some Stats

These statistics come from a recent survey ‘Mobile Future Forward’ slideshare.  One of the things discussed at the Internetworld 2012 event is how comfortable we have all become purchasing things from our phones. With the introduction of 1 click purchases as Apple does with AppStore and iTunes, and Amazon’s app for iPhone and Android. This has increased our trust in the ability to use ‘devices’ to carry out various financial transactions.

What about Risks?

There is an obvious risk with the responsibility that retailers will carry. What will they do with all this information that is gathered? Most likely they will be using it to try to predict your spending patterns/habits to suggest you new products. This is an increasing trend we see across all eStores “recommended for you” and “Others who bought this also bought that”. Whilst initially appearing to be helpful we don’t want to end up having less choices available to us by creating a snowball effect because we always stick to the same old things due to the fact that we have been suggested to.

What type of performance testing are you doing?

In general performance testing is a measure of responsiveness. However, it can also be a measure of responsiveness over a certain period of time. In a sense then there are two types of responsiveness that performance testing is interested in:

  1. How long:  That is how long it takes for a particular page to load, or a request to be sent, or a function to return. Or an overall Use Case to be completed as a workflow through a website. Or a client call to a backend to return.

  2. Memory profiling:  In a sense this could be ‘how long’ after ‘how long’? I.e. We know doing an action once or twice is responsive from the type of testing we do with ‘how long’, but we also need to know if this stays the same over time. Profiling memory over time can help to indicate if there is a problem now, and/or one building for the future.

N.B. You may also be thinking that adding more users should also be considered when measuring performance. Yes it should. But strictly speaking this would be Load Testing rather than Performance testing. It is likely that a lot of the same techniques and tests could be used. The line between the two is not always going to be clear.

We may know that doing something once is responsive, but what if we do it 100 times, or 1000. However, this also requires some user profiling, or use case analysis. If the function will only be used once or twice a day there is little value in proving that it is still responsive after 1,000 repeats. However, if the function will run many times a minute or even second then it needs to be seen to respond within this time, and be seen to remain responsive over time.

What level of Performance are you interested in?

-     Application level

-     Component level

-     Unit level

In general performance testing often is not done until you get to the application level. At which point complex causes and fixes can be suggested if the application is found not to be responsive. Instead like development in general it is better to test units first, then the components made up of those units, and then the application that is built from those components.

Which presents another issue.  How  to know from the overall application level performance requirements what each unit’s performance requirement should be?

Nonetheless having unit level focused performance tests will make it easier to ‘zoom in’ on an unresponsive area of the application during later application level testing if necessary.  Assuming that this wasn’t done then you need to make the application level testing focus on particular components at least at one time. Thus providing another way of ‘zooming in’ on a specific area of the application. This could be done by writing separate tests, or by ensuring that it is clear in log files where one function ends and another begins. This leads us on to how you will do the testing.

In part 2 we will consider tools, environments and Observer effects for performance testing.

To give a context, a trade ticket consists of several parameters like:
- Trade Type (SPOT, FORWARD, SWAP, TIMEOPTION)
- Currency Pair (BASE, TERM)
- Direction (BUY, SELL)

If we combine all possibilities we will have to execute 16 tests

TYPE CURRENCY DIRECTION
1 SPOT BASE BUY
2 SPOT BASE SELL
3 SPOT TERM BUY
4 SPOT TERM SELL
5 FORWARD BASE BUY
6 FORWARD BASE SELL
7 FORWARD TERM BUY
8 FORWARD TERM SELL
9 SWAP BASE BUY
10 SWAP BASE SELL
11 SWAP TERM BUY
12 SWAP TERM SELL
13 TIMEOPTION BASE BUY
14 TIMEOPTION BASE SELL
15 TIMEOPTION TERM BUY
16 TIMEOPTION TERM SELL

During this discussion I remembered a Tech Talk I went to last year about Pairwise and Combinatorial Testing and I started to do some more research on it.

What is Pairwise Testing?

Pairwise Testing is a software testing approach that can help to test a big number of input parameters which normally results in hundreds/thousands of different permutations.

A good example to show the immense number of test cases is a configuration dialog with 12 checkboxes (checked, unchecked) and a dropdown with 3 values.
If we wanted to cover all possible combinations we would have to execute 12,288 different tests which is calculated by (2 power n) * 3.
Amazingly these 12,288 tests can be reduced to 10 if we apply the 2-way pairwise testing strategy.
See http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

How does it work?

Pairwise testing is a way of testing a big set of parameters by only testing pairs of the given parameters. If we look at the example above we can see that there are 2 tests for SPOT, BASE (SPOT-BASE-BUY, SPOT-BASE-SELL).
One of the inspirations for Pairwise Testing is the fact that bugs occur more often on a pair or triple of parameters rather than on a single input parameter. Therefore it is quite popular and often used for configuration testing but it can also be applied to any other area of software quality assurance.

This sounds all quite magic. So I would not recommend to blindly apply this strategy without analysing the software under test and getting some more information about architecture and risk before using it.

More information can be found on the following pages:

http://www.pairwise.org
Hexawise
An easy guide to orthogonal arrays and All-pairs combinations

What I personally find quite interesting is the combination of a pairwise analysis with exploratory testing. By getting a good idea of the input parameters it can definitely support exploring the application.

Finally I would like to show the result of the 2-way Pairwise Testing strategy applied to the example above.

2-WAY
TYPE CURRENCY DIRECTION
1 SPOT BASE BUY
2 SPOT TERM SELL
3 FORWARD BASE SELL
4 FORWARD TERM BUY
5 SWAP BASE BUY
6 SWAP TERM SELL
7 TIMEOPTION BASE BUY
8 TIMEOPTION TERM SELL