Amongst the new benchmarks will be some tests for publishing from the client, which is key for trading applications, and also benchmarking our container objects. A container is an object that represents a dynamic list of references to other data. Subscribing to the container automatically subscribes the client to its contents. Containers also allow you to subscribe to a subset, allowing efficient display of huge lists. The platform allows containers to be created dynamically based on filter and/or sort criteria too.

Anyway, back to the core benchmarks. At Caplin we have never had a customer really need more than the 30,000 client limit Liberator imposed so I never pushed it further than that. When Richard Jones blogged about A Million-user Comet Application with Mochiweb I was intrigued. Richards use case was very different to ours, but I wondered whether Liberator would handle that kind of profile. Later on Migratory published some benchmarks which used our scenarios as a base, but pushed the user numbers higher than we had done.

The hardest part of testing high numbers of users is actually the client side, the machines you need, or tricks you have to make, to get that many connections to your server. I removed the hard coded 30,000 limit from Liberator and set about pushing the user numbers higher.

Deterministic latency can be more important that the absolute value, which is why this new data is so important – Solace Systems talked about Consistent Latency recently. Something I was never happy about with our old benchmarks was they just showed average latency, which is good, but are all messages the same? How much longer do some take? What is the worst case? If you are trading and you get a message that is the worst case latency then that is important to you. So with my latest rig I am measuring min/max latency as well as standard deviation. Both these are represented by error bars on the graph below. Another change is we are now measuring in microseconds and focussing on minimum, but consistent, latency as much as possible.

So here is a taster of the results I am getting. I have pushed it up to 100,000 users. The Liberator is running on a 2x Quad core Opteron box, which was barely breaking sweat, the client machines are what stopped me going past 100,000.

As you can see, the latency is very low. At 100,000 users average latency is still only 6ms. The red max bars show the worst case hits 30ms for one sample, but standard deviation is very small. Each sample ran for 30 seconds, so at 100,000 clients, 3 million messages were measured with the worst being 26ms. If you could zoom in on the start of the graph you would see than 10,000 users can receive 1 message/sec with latency of less than 1ms!

Remember, these are benchmarks, in the real world over the Internet, things will be different – but you have to start with a good base!

I might publish some further graphs as I test the scenarios with higher message rates.

When an end to end acceptance test fails, the failure could be due to any one of a large number of source lines or expectations changes across a large body of code.  Debugging may be quick, but often isn’t.

Small tests run faster too! While a typical small test may take 10 milliseconds to run, a large acceptance test may take twenty minutes.

Just a small back of the envelope calculation tells us that if a developer checks in twice a day and saves an hour each time because potential issues had been found before check-in with fast small tests, and the faster feedback, or has reduced their time spent debugging – that is two hours a day per developer that could be put to better use.

2 hours a day * thirty developers * two working weeks = 600 additional hours per two week iteration available.

This is equivalent to adding seven and a half developers to the team!

What value do you get from tests of different sizes?

Following a succesful session that Andrew MacLeod and I ran at last year’s conference (building an automated browser testing framework using Selenium), a couple of proposals have been put in by Caplin’s staff for next year’s conference in May:

Sarah Lawfull and Duncan Brown have proposed “Persona-driven development – can it work for you?”.

Adam Iley and Phil Leggetter have proposed “Comet, Twitter and the Real-Time Web”

Fingers crossed they’ll be accepted – if they are we’ll post some more info on the contents nearer the time.

There has been a big push from Adobe towards using Flex (and to a lesser extent LiveCycle Data Services – their messaging product) for trading portals and you can certainly build a flashy offering, as can be seen in the recent launch of Morgan Stanley’s Matrix.

But questions still remain over the ease of development and deployment of such a large-scale enterprise application, and though the Flex Builder seems to make this easier, digging beneath the surface can make for a frustrating experience, as Alex Vandeberg’s excellent analysis recently described.

Could Silverlight be an answer to this?

Caplin Trader is Caplin’s toolkit and platform for building trading portals, in production at a number of banks. The client-side framework is built in Ajax (the browser’s native presentation technology), though we allow and encourage the embedding of content in any RIA techology – be that HTML, Ajax, Flash/Flex, Silverlight (and even Java Applets if you are still that way inclined!). I believe that this approach encourages building a best-of-breed portal: You use Caplin’s APIs and the browser’s native language for the container and most pricing and trading components, but where appropriate (for example rich media, 3D, vector graphics) you have the freedom to seamlessly embed a more suitable technology such as Flex or Silverlight. In addition you can leverage existing web content with minimal effort. The alternative approach of using one of these proprietary plugins for the container makes it very hard to embed anything else within the container.

Caplin Trader is built on top of the Caplin Platform – a generic financial client-connectivity, web messaging and trading layer. Caplin have always provided client APIs for a variety of languages (Java, .Net, Ajax, C++ currently) to allow people to either integrate with existing applications or build their own from scratch (usually where Caplin Trader isn’t suitable or desired).

What is interesting is the growing demand for Silverlight as a client-side target for Caplin’s technology. I know of at least 3 large investment banks that are kicking off projects targetted at Silverlight. Whilst not yet cross-asset single dealer portals on the same scale as Matrix (or most of the Caplin Trader projects), it’s interesting that they’ve bitten the bullet despite lingering doubts over adoption rates and install base (<30% as of today at riastats.com), though Microsoft would like you to think otherwise!

If it can gain momentum, Silverlight will quickly eclipse (no pun intended) Flex as a developers’ platform of choice, at least in terms of true software engineering tools and developer productivity. Whilst we’ve done a lot of work to build developer tools and structure our APIs to ‘tame’ Ajax and JavaScript, neither Ajax or Flex/ActionScript will catch up with C# any time soon when it comes to more advanced activities such as IDEs with truly integrated build/test/debug/profile tools and refactoring, not to mention the number of existing C# developers that could transition to Silverlight easily.

You can already use the Caplin Platform and Caplin Trader with Silverlight by creating a bridge to our JavaScript StreamLink for Browsers library. We’ve recently built a prototype native Silverlight library, with an identical API to our existing C#/.Net offering, and we expect to be able to release a full version in July. On top of our Caplin Trader subsystems this could be used as a powerful front-end for those who choose to build single dealer portals with it and it will certainly be interesting to see who jumps first…

What do I mean by not being able to “create an instance of yourself using new Class()“?

MyClassDefinition = function()
{
};

MyClassDefintion.prototype.myFunction = function()
{
    // do some stuff
};
MyClassDefinition = new MyClassDefinition();

Here we lose the ability to do new MyClassDefinition();

What do I mean by “objects that are dynamically created”?

// create an object using the object literal notation
var oObj = {};

// dynamically append a function
oObj.myFunction = function()
{
    // do some stuff
};

Code written in both these ways make it more difficult to create a mock using Mock4JS because it expects a function reference to be passed to the mock method call.

A way around this is to dynamically create a definition from the object. This is done using the following piece of code:

function createMockDefinition(oObject)
{
    var oMockedDefinition = new Function();
    for( var x in oObject )
    {
        // Only add functions to the dynamically created prototype
        if( typeof oObject[ x ] == "function" )
        {
            oMockedDefinition.prototype[x] = function(){};
        }
    }
    return oMockedDefinition;
};

With the above code we have now created a class definition that we can pass to the Mock4JS mock() function.

function myTest()
{
    var oObjectIWantToTest = {};
    oObjectIWantToTest.functionIWantToTest = function(){};

    var oMyMockDefinition = createMockDefinition( oObjectIWantToTest );
    var oMyMock = mock( oMyMockDefinition );
    var oMyMockProxy = oMyMock.proxy();

    // setup expectations
    oMyMock.expects(once()).functionIWantToTest("myArgument");

    // call function and execute mock method.
    oMyMockProxy.functionIWantToTest("myArgument");

    // Finally, verify the expected mock methods have been called.
    // If not, an exception will be thrown.
    Mock4JS.verifyAllMocks();
};

The main theme of the talk was to demonstrate that, by building a thin business-domain framework on top of Selenium and structuring test cases appropriately, it’s possible to construct a very simple DSL that makes test scripts intelligible to non-programmers.

The workshop involved a couple of pairing exercises using the Firefox Selenium IDE add-on which enabled participants to get their hands dirty writing basic tests in a very short time. I hadn’t used the Selenium IDE much prior to the conference, but it’s a brilliantly simple record & playback tool which Caplin should consider exploiting.

We supplied delegates with USB keys containing Firefox Portable that was pre-configured with the necessary add-ons and bookmarks. This offered a complete client side Selenium runtime environment for all the exercises, and connected to a web-site that we hosted on a wireless LAN. This setup was quite effective in providing a full development environment in minimal set-up time.

This was our first time presenting at a conference, and despite all the hours sweating over composing slides and exercises, it was really good fun and the session was well received. We’re very grateful to Ivan Moore for encouraging us to put this together: SPA is a really friendly and collaborative forum that have frequent events, and I would recommend it to anyone as a good place to cut their teeth presenting a conference session.