Leading Caplin’s UX Design team from research through to visual design, Matthew is focused on ensuring Caplin products are appropriate, intuitive, fit for purpose and a pleasure to use and look at.

Matthew takes two minutes to tell Jennifer Reid a little bit more about leading UX design forward in real-time trading.

Q: You’ve had a really interesting career as a creative thinker. Tell us what attracted you to UX.

A: For a quiet life?  No, it’s the opposite.  Throughout my career I’ve taken complex information and put it in front of people who need to make use of it in the best way for them, and for the client.  Whether it’s television news, staff development or FX trading, the challenges are the same:  Innovating and improving user experience is a broad discipline, not a webby nicety, it’s the core of all good design.  At Caplin the need for clarity, speed of comprehension and action, and the projection of brand values are all amplified by the nature of the domain.

Q: Everyone’s got their own definition of UX. Tell us yours.

A: Well, I’m not a fan of UX being unhitched from design and engineering, because I think what we do is virtual product design.  When you design a car it has to work right, look right and feel right, what we do is no different.

Q: In 5 sentences, tell us what you do in a typical day at Caplin.

A: I think I can do it in 5 words; look, listen, think, decide, do.  Although I could replace ‘do’ with ‘delegate’, and there is also a fair bit of talking involved.  So it could be; absorb, consider, conclude, convince & act.  Either way it’s fun.  This is five sentences I notice.

Q: Last tech gadget you bought?

A: Depends what you call a gadget I suppose.  I don’t think I buy ‘gadgets’, but the last technical things that I got because I liked them more than I needed them were apps for my iPhone; ‘Solar’, a weather app, because it’s so beautiful and ‘Linxy’ a dictionary which gives you definitions of words, live, as you point the camera at them, because it made me say “Wow” out loud like a child.

Q: What are you favourite UX books or resources?

A: As I think we are actually in the product design business, Raymond Loewy’s book ‘Industrial Design’ does it for me.  Loewy invented what America looks and feels like – twice.  Once with the opulent, excessive, overtly futuristic streamlined 40’s, and then in the clean-cut, slick, spare, modernist late 50’s and early 60’s.  Always functional and – yes I’m going to say it – a bit sexy, and always projecting joy and optimism.  Loewy had a framed letter on his desk from the astronauts who used Space Lab thanking him for making their environment more functional and a pleasure to live in – if that’s not a UX success I don’t know what is.

As for my favourite resource, it’s my team here at Caplin Systems and people like Paul Blank and Scott McLeod who are so good at communicating user and business needs, and demand more and better, from us every day.

Q: Creative thinkers often get inspiration from everyday objects or experiences. What inspires your thinking?

A: Music inspires me:  I once attended a lecture on the nature of genius by Dave Stewart.  He broke the Aretha Franklin song ‘Until you come back to me’ down to the component tracks, and was unable to find a single flaw. Every isolated ‘doo-wah’, drum break and guitar flourish was perfect – but meaningless.  On bringing it all back together he found it was still flawless, but now it had soul.  Now it was genius.  So it was context and concerted effort that made it a work of genius.  That’s inspiring.

Also, there has been a renaissance in ‘statement’ architecture in London, and I find something inspiring to look at every day as I walk from Waterloo into the city, you don’t have to like it all but you have to admire the ambition and guts it demonstrates.

Q: Where is UX going? What are you excited about?
A: Some physical tools and machines have had thousands of years to become ‘natural’ in use, whereas no widely used virtual interaction has had it’s fortieth birthday yet.  It’s no wonder we sometimes resort back to imitating familiar physical activity.  But a generation is coming that, through gaming and social media, is comfortable interacting virtually with the real world and ‘really’ with the virtual world:  All the lines are blurred.  Forever.  For me that’s weird, but for my children it’s everyday.  Once people who interact with technology ‘naturally’; unthinkingly are calling the shots we may be cut loose from reiterative and legacy-based thinking forever.

Uh-oh.

Say it again!

That is certainly the response I got when I asked an architect about code coverage. And it also seemed to be the conclusion from a long email thread/discussion we had here last year. And it was the response I expected.

The referral to Martin Fowler’s blog post on Test Coverage was the most useful. To some extent this blog confirmed what I thought Code Coverage was good for, which is that it indicates where the untested code lives. What code coverage is not good for is setting a target that can then be happily ‘gamed’ to produce the number everyone first thought of (N.B. a handy skill if you are an accountant).

Why is code coverage a good for nothing?

The reason it is seen as a good for nothing that adds no value to a team is because historically it has represented a number that will be compared to one arbitrarily chosen by management. The expectation being that all anyone is interested in is making sure that the actual code coverage is higher than the limit number. Management will impose sanctions if this is not the case, and developers will be encouraged to write reams of meaningless tests in order to ‘game’ the code coverage number produced by their code to be the right side of the limit. See Why 100% code coverage is not enough for an example of this in action.

TDD Discipline results in good Code Coverage

I had also recently been on a TDD course, and the most important thing about TDD that I learned from that course is that it is a discipline. Discipline is the most important thing because without it TDD will not work. Writing tests for your code is not TDD, having 100% code coverage is not TDD. Writing your tests before you write your code, and only writing the code you need to to pass your tests is TDD. If you do that (and no more) every time.

It is very difficult to maintain this discipline as a developer, because as soon as you start writing some code then you start adding extra functions or variables that you ‘know’ you will need later. As soon as you start doing that then you have broken the discipline of TDD. A side effect of this is that your code coverage will be less than 100%, but that is not a problem. The problem is that you have untested code that you don’t know isn’t tested, and that you don’t need.

The Test-driven Development Maturity Model can help you to understand what level of TDD you are working at, and also highlights at the end what type of developers don’t need to do TDD.

Code coverage lets you know where the untested code is, and thus where the untested risk in your system is located. If the risk in those locations is low value, then this may be acceptable, however, if the risk in those locations is high value, then this is unacceptable. TDD is an extreme form of this risk detection strategy, in that code is only written if it will be used, and tested. You only write code that has a failing test, and you only write tests for required business features.

Discipline vs. focus on critical areas

TDD is a discipline. Our trainer said he could get test coverage on any code within a couple of hours. He also said he’d expect it to be around 80% if the TDD was being done right. Not at least 78.457%, but around 80%. I.e. not chasing a particular number, but knowing it’s likely to be around 80% if the TDD is being done right. That is because TDD is premised on writing tests first as a discipline, so there should be a high coverage as a by product of the process.

In comparison BDD will not have the same by product, as it is premised on describing behaviours, not implementing tests first. Dan North at QCon (Accelerating Agile: hyper-performing without the hype) said that his teams test coverage was more like 6-7%, but in the critical areas it was 150%. I.e. sufficient, but not unnecessary. There needs to be sufficient testing in all areas, but to add the most value, no more than that, and in particular areas that may be very little indeed. At the same time, where there is critical functionality then this needs much higher coverage.

One example in our case might be that the core APIs are always critical, so will always need good coverage. Whereas when building an application this is built on top of the core APIs, so the coverage needs to be sufficient to cover the new features to an appropriate level, but not unnecessarily repeating coverage that already exists in the core libraries.

It is not really discipline vs. critical focus. They are two different processes that can each achieve good results. The point is to know which one you are doing and do it properly. Or whichever process your shop uses. In either case they know their code coverage and what it means to their process. The same should be true in any other process too.

So what is it good for?

Code coverage can inform you of the quality of your code. Interpreting and understanding that code coverage, and why it is as it is across your codebase will evolve over time, and will also involve an understanding of your domain and it’s requirements.

Code coverage at it’s most basic allows you to know whether you have tests or not! It can highlight where the testing effort has been targeted. Examining these results allows a team to take a measure of the healthiness of their code and their development practices.

Code Coverage enables safe Refactoring

Refactoring (Martin Fowler again), is a common and much used practice in BDD and TDD. The reason agile teams using these methodologies can refactor safely is because of the safety net that their extensive regression test packs give them. They know that if they break some previously working functionality then the automated regression tests running on CI will catch those failures. They know this because they wrote lots of unit tests and acceptance tests which give good code coverage. They know they give good coverage because they code reviewed and QAs checked those tests when they were first written.

Unfortunately even agile teams suffer from technical debt, and deadline pressure, which means maybe not all of those tests did get written. So, in fact, they may not have a safety net, and they don’t know where they need to focus their efforts to mend the holes in that net if they don’t run code coverage because they all agreed there was no need as it was just a management tool to produce a meaningless metric.

Code Coverage what is it good for?

Giving you the confidence to refactor safely.

Telling you when your code has started to rot, and where to look first to fix it.

Without it you have no idea!

The term “Quality Assurance” however is a much better fit for software development. We make sure that quality is defined and built from beginning of software development process, not just to check at the end of production.

In major Agile techniques, the workflow (shown as columns in Kanban and Scrum boards) is divided up the into following stages:

• Defining the requirements (Backlog)
• In development
• In test
• Done

There might be more columns on the board but the ones above are common to all variations. Each of the two week sprint turns into a mini waterfall when the team members do not collaborate and there is a clear handover from BA to developers and from developer to testers. As you can see, testing contributes to the project as ‘Quality Control’ instead of ‘Quality Assurance’.

So how can we quality assure from the beginning? This article aims to address this challenge. In part I of this article we will concentrate on only defining requirements part. Part II will deal with development and testing stages.

Defining the requirements

At the beginning of the sprint developers and testers are given a list of stories. If requirements for these stories are determined without any collaboration between developers, QA Engineers and BA’s or PO’s then several problems will rise from this:
1. The requirements can get lost in translation when multiple stakeholders are involved.
2. When the requirements are defined in abstract terms, they can be ambiguous and overlooked until developer(s) start working.
3. The requirements do not tend to include negative or edge case or unlikely scenarios.
4. A particular requirement may conflict with existing functionality, or the requirements may be inconsistent with each other.
5. The requirements may force a certain type of implementation or technology without defining what business problem it is trying to solve.
6. Business Analysts and Product Owners may be the domain experts and what seems obvious to them may not be obvious to developers or testers (or worse the words might mean something else to them).

Team members with different expertise may not have the same understanding from the same subject. Like one comedian pointed out, when he hears “disabling cookies” he thinks of biting the legs off a ginger man biscuit.

So how do we prevent ambiguity during the requirements definition phase?

The solution is to acknowledge there is a communication gap and resolve it. The best way of achieving this according to Gojko Adzic author of ‘Bridging the Communication Gap’ book is to run a specifications workshop before each sprint. The workshop should be attended by BA’s, developer(s), QA Engineers and if possible stakeholder(s).

Preventing the workshop from turning into another boring meeting

1. Every participant should focus on what the requirements are and not how to develop or test them.
2. The workshop should take no longer than 2-3 hours.
3. Only stories which will be implemented in the next sprint should be discussed nothing else.
4. To cover as many important cases as possible, each requirement should be explained by specific examples.
5. The specific examples should be documented and signed off by the BA.
6. If the BA needs to obtain more details about the specific examples from stakeholders, then this should be noted down and one of the participants should be assigned to chase for the answers.
The list item 4 deserves more explanation. One of the best ways of understanding a new concept is to give examples. So I am going to give you an example about how to write examples:

Example of “Specification by Example”

Requirement: “User has to enter a valid amount into the textbox to execute a spot trade on a tile. See the UX design below”:

The questions a developer or QA Engineer might ask to the BA:

Now that we have all the questions we can think of answered, we can organise it into tables for neat presentation:

We now have a clear requirement which is ready to develop and write acceptance tests. In this approach ‘Business Requirement’ becomes equal to ‘Acceptance Testing Criteria’.

The benefits of specifications workshop are:

• Everyone in the team think harder about needs of business and users,
• Identify gaps in the initial set of requirements,
• Potential bugs are eliminated,
• Acceptance criteria for testing are already determined (QA starts now).

A specification workshop is not a:

• Sprint planning meeting,
• Software design meeting,
• Sizing meeting,
• Test design meeting.

In the next part of this article, I will take you through how to validate this requirement which happens in development and testing phases.

JQuery 1.9 and 2.0 – Richard Worth

Richard Worth was up next, and spoke mainly about the coming divergence between the 1.x line which will continue to support IE6-8 and the 2.x line which will not.  Apparently, this allowed them to remove 12% of their codebase and has better compatibility with environments that aren’t desktop browsers.

He talked about the intentionally limited scope of core jQuery, covering only Dom manipulation, events, animations and ajax, but also highlighted how plugins can let you add functionality seamlessly.

A topic that came up a few times was that you need more than JQuery to build a well architected large application, but Richard talked about how ubiquitous jQuery is (9/10 javascript using websites) and mentioned that many MV* frameworks build on top of JQuery.

JQuery 2.0 lets you opt out of different pieces, ultimately getting as small as 10k when minified and gzipped for just the dom manipulation and selection.

He also promised that JQuery 1.x would be supported “until IE6, 7 and 8 is no longer relevant” and that 1.10 upwards will have the same API as 2.0 upwards (1.10 -> 2.0, 1.11 -> 2.1, etc.)

I know JQuery.  Now What? – Remy Sharp

Remy Sharp was aptly represented by the joker card, and he gave a talk about the importance of carefully considering whether you really need jQuery at all.  He’s put the talk up as a blog post here.

He started provocatively by saying that for many people “JQuery is the gateway drug to Javascript programming.  This talk is about how to get clean and kick your dirty habit.”

One of the key things is that much of the functionality of jQuery has begun to be folded in to browsers and for basic functionality they’ve converged on standards too.  This means that if you’re working in more recent browsers, you may not need JQuery at all.

He suggested some common possible replacements:

• querySelectorAll instead of $
• putting javascript at the end so you don’t have to worry about onready.
• Use the standard dom methods for setting classes and getting attributes.
• Dataset allows you to store values on dom nodes.
• XHR2 has a bunch of nice features that give you more control than jquery for Ajax.
• HTML5 form support now has a lot of features, e.g. pattern, required and type that reduce your need to write jquery code.
• Lots of stuff can be done in CSS now that people previously used JQuery for.
• Injecting a stylesheet and using classnames is likely to be more performant than using JQuery .css.
• RequestAnimationframe is important for smooth animations and vanilla jquery only uses setInterval.
• Use translate to move stuff and the transitionend event to fire javascript.

“If it’s native to the browser let the browser get on with its job.”

As an example of the kind of thing that Remy hates, he provided FitText, a jquery plugin that really didn’t need to be.  He rewrote it in vanilla javascript which you can see here.

It’s fun that the jQuery community is confident enough to have these kinds of talks at their conference.

JQuery is a Swiss Army Knife (and that’s OK!) – Adam Sontag

While Adam Sontag was ostensibly the counter point to Remys slightly anti-jQuery talk, apart from professing a deep affection for jquery, he said many of the same things, even to calling JQuery a ‘gateway drug’.

He adopted a swiss army knife metaphor to describe jQuery: In a pinch, you could build an entire house with it.  As long as you don’t want to live in it for very long. But you might use it to open a can of paint, etc, etc.  It’s a swiss army knife. You need bigger building blocks than just what jquery gives you, such as MV* frameworks.

The elitism of many developers and the way that jQuery is sometimes maligned clearly irritates Adam, he compared people complaining about JQuery users to people saying “We’re not going to teach you English because one day you might say something dumb”.

He also talked about how jQuery is engaging with the community and what that is like – learning the mechanics of open source can be intimidating, like programming itself, but jQuery have open sourced everything, including documentation and site design. There’s an exciting new contribution hub, contribute.jquery.org.  Supporting the community is an investment that keeps it growing.  Everyone who contributes regularly is invited to join.

JQuery happens every day online and regularly in person on IRC. Relationships that go beyond code improve code.

On top of all that, jQuery became a foundation and is now a member of ECMA and W3C.

Taking control with Machina.js – Doug Neiner

Doug talked about the open source state machine for JavaScript Machina.js.  It’s an interesting project, with some similarities to a state machine library we have created at Caplin.  Our library has implementations in both Java and Javascript and the client and server can have their statemachines kept synchronised.

Use events to glue full-stack frameworks together – Garann Means

Cramming everything into MVC sucks. Model = state management, View = rendering, controller = everything else?

Solving some of these problems tends to lead to a mixture of promises, custom events, pub/sub, event emitters and callbacks.  You can use events to augment MVC, which let you hook all your pieces together, going from the client side right back to the server side.

Garann also talked about some of the full stack frameworks out there that she likes, hoodie.js and meteor.js.

What, Chrome DevTools can do that? – Ilya Grigorik

Ilya recommended discover-devtools, a new course on codeschool.  You can also get his presentation here.

He talked about some sneaky ways to speed pages up. For example google start sending the search results page before they’ve even worked out what the search is, so that browsers can start rendering and requesting resources immediately.

A good place to investigate the speed of pages is the ‘waterfall’ display in Chrome Dev Tools.  You can save the data behind the display as a HAR file, and there are tools that can parse the format and let you inspect it offline or integrate the process into your build.  This can be good if a client complains that your site is slow – they can send you the HAR file and you can investigate.

Some other good tips:

• chrome.loadTimes()   Snippets: chrome.loadTimes().firstPaintTime – chrome.loadTimes().startLoadTime;
• local modifications
• chrome://tracing
• console.time(“name”) / console.timeEnd(“name”)

Dev Tools has a number of options to help you debug where your rendering time is spent, and rendering time is important.  User interaction with timeline dropped when facebook went from 60fps to 30fps, which means that you need to be spending less than 16ms per frame.

Faster Dom Manipulation with Category theory and wield – John Bender

John enjoys investigating and talking about the link between theory and practice.  In this case, he showed how observing the link between JQuery and category theory led him to start using ‘loop fusion’ optimisations to make his jQuery faster.

It’s difficult to summarise the theory as well as he did (so check out his presentation here), but:

A category consists of objects and morphisms which map from one object to another.  You can think of the DOM objects and functions as one category and the world of JQuery objects and the functions that work on those as another.

A functor maps from one category to another.  You need to be able to map both the functions and the objects.  In JQuery, the $ function is part of a functor that maps from Dom objects to JQuery objects and the jQuery function ‘map’ converts functions that work on dom objects to functions that work on JQuery objects.

There is a composition functor law; the mapped composition of two morphisms in one category has to be the same as the composition of two mapped morphisms.  This essentially means that mapping a function that makes two dom changes is the same as mapping two functions that make one dom change, which means that you can just go ahead and combine the behaviours and just loop over your jquery collections once (‘loop’ fusion).

John has created a library based on these ideas. wield.

Complex client-side apps and legacy browsers (A story of Frustration) – Joe Petterson

Joe spent much of his talk describing ways of testing against multiple different browsers.  He suggested using Selenium with a group of VMs.  You can manage your VMs with a script like the one here although it will take an enormous amount of RAM.

While his talk was interesting, we’ve moved away from testing via user interaction in the browser as much as we can.  We found Selenium tests to be brittle and slow, so we turned a nights worth of Selenium tests into 10 minutes worth of view model tests and are really happy with that decision.  There are still things that you need Selenium for, but we try to keep that to a minimum.

Build an Experience… Not another Framework – Jason Scott

Jason’s talk focussed on his work using jQuery Mobile on Blackberry 10.  He talked about using Ripple to emulate the mobile web, and boasted that the Blackberry browser has a higher HTML5 rank than desktop Chrome.

He talked a lot about things to consider when serving data to mobile, but also made the point that sometimes it’s better to admit defeat in terms of mimicing native behaviour in favour of something more appropriate: “better to have a decent experience than copy a great experience and end up with a bad one”.

Daniel Owen’s (former futures trader and current member of Caplin’s dev team) latest two part series on the history and evolution of executing futures will appeal to techies looking for background in trading platforms and execution and how systems have developed over time.

Sell Orders on a Chart Trader

An excellent read for anyone involved in trading systems development.

The style of the conference continued with a Lewis Caroll theme running through everything.  The 3m tall dragon that greeted you as you entered the building turned out to be a jabberwocky and was also featured on some rather cool jquery uk t-shirts that all attendees received.

After chatting to the guys from Infragistics and getting a tea and coffee in the sponsors hall, we were all ushered into a large hall to begin the conference proper.

The conference guide took the form of a deck of laminated cards with the speakers details on the obverse.  The Ace of Spaces took the stage: Brendan Eich.

Javascript : The Future

Brendan covered two main areas in his talk; the exciting things coming up in ecmascript 6 and the amazing performace gains that have been made through the use of emscripten and asm.js.

Although “Harmony is like a family that has gone through some hard times”, it’s been exciting to see people finally getting on board.  Brendan characterised Microsoft’s attitude to the process as “No, Never! No, No, Yes.”  And while it will be some time before the whole standard is finalised and widely implemented, he encouraged us to start using it soon, since parts have already been implemented in all the major browsers.

Then we got a whistle stop tour of all the features that are coming up.

arrow functions  / classes / Object.mixin (super will work with mixin) / modules / symbols / private syntax (@ not dot) / default parameters / rest parameters / spread operator / iterators, generators, for of (based on an iterator symbol) / Comprehensions / Maps with arbitrary keys / Sets / weakmaps / proxy (allows no such method)

It sounds like there’s been a lot of inspiration from Python, but that’s a good thing. We’re certainly excited about a these and a lot of the audience seem pretty stunned by the rapid fire revelation of feature after feature.

Brendan also addressed some common complaints, like ‘callback hell’. At the same time as talking about some of the things that could be done (maybe promises in es7?), he did say that he thought a lot of the complaining was overdone: “there’s too much whining about nesting”. He’s thinks the right answer is a combination of generators and a library called spawn.js which allows you to write ‘straight line code’ that looks synchronous but is actually asychronous.

One of the challenges of ecmascript 6 is to improve the experience both for developers writing javascript and also make it a better language target for other languages to be compiled to. Brendan talked about why javascript is better than bytecode (it’s similarly dense, it’s higher level, it’s not future hostile,
it’s safer) but said “For now, we’re serving two masters, humans and computers.  May the best animal win.”

We got a brief plug for firefoxOS phones, e.g. from geeksphone but apparently a number of manufacturers are on board, including Sony.

Then we jumped into the next part of the talk – the things that have been done to make the web ready for triple-A games. According to Brendan, triple A games are the wrench dodging of the software world. If you can run a triple-A game, you’ll be able to run nearly anything.  And then he showed us firefox doing
just that, loading up the Unreal Engine Citadel demo live, and finally playing a few rounds against some bots in first person shooter Sanctuary.  The bot ‘Torque’ drew first blood, but Brendan managed to get some applause for shooting ‘Matrix’.  The graphics were beautiful and the performance was in excess of what most people
would have ever expected from a browser scripting language.  They’d managed to achieve this by transpiling from C++ to a subset of heavily optimisable javascript.  That was enough to elicit a ‘Bloody Hell’ from someone behind me.

In the Q&A section, an interesting question was raised about how to ‘protect’ your javascript code. Brendan said that the Mozilla Foundation is strongly opposed to most forms of DRM.  He spoke passionately against having ‘policemen’ in your computer controlling what you could do with it.  He said that actually, very
few of the organisations they were collaborating with were too worried about their source code being extractable but he also spoke about considering pragmatic solutions to help provide rental models which customers seem to want. He mentioned a watermarking approach, which apparently many of the online video providers are moving towards, and seems to be a better approach.

Write ups for the other talks coming up soon – I just have  a hack day to go to first!

That particular leak gained a great deal of infamy and you can find a lot of information online about the DOM/JS circular reference memory leak (e.g. here or here). This leak was so serious because most of the natural patterns for event handlers would end up creating circular references between javascript and dom elements that IE didn’t know how to garbage collect. This was finally fixed in IE8.

Fortunately, we (at Caplin) don’t have to support versions of IE lower than 8 any more, so does that mean we can stop worrying about memory leaks? Sadly not. The applications we work on are single page applications with a requirement that they can run for days without needing to be reloaded. In that time the user might make multiple trades, hunderds of thousands of updates might come in over the network and need to be rendered to screen, elements might be removed and added from the screen, and grids with many thousands of live updating rows might be scrolled, sorted, reordered or filtered.

In situations like that, there are plenty of opportunities to leak memory in your own code, let alone (for the moment) from browser bugs.

Memory Leaks That Are Your Fault

By far the most likely way you are going to leak memory is by adding listeners and failing to remove them. You need to develop discipline to remember that every time you add a listener you should be writing the mirror image code to remove the listener.

Hierarchy of Possibly Listening Objects

Something that can make it more difficult to remember to remove listeners is when you have a tree of objects, and you need to remove an object and its children from the tree. You might remember to clear all the listeners from the object you’re removing, but did you remember to clear the listeners from the children?

This is the kind of situation where it’s useful to have a dispose method that calls the dispose methods of its children so that each child can worry about cleaning its own references up and then instructing its children to do the same.

Running Initialisation Code Twice

Something else that can catch you out is if you have a bug that causes your initialisation code to run twice.  It’s very common that initialisation code registers listeners and then keeps hold of a reference that is later used to deregister the listener.  If you run a naïve version of this code twice, it will register the listener twice, then throw away one of the references meaning that when you do eventually clean up your object, you’ll only remove the most recently added listeners.

For this reason, if it’s possible for initialisation code to run twice, I like it to check to see if it has already been initialised and throw an exception if so. That means that your code fails fast and you can’t just have a double-initialisation bug lurking there without you realising. If your code needs to be able to initialise twice (e.g. perhaps a setModel method) then it should check to see if it needs to deregister a listener before it starts adding any listeners.

Third Party Libraries

There are sometimes leaks in third party libraries, particularly since many thirdparty libraries haven’t been written with week long run times in mind, but more often, those libraries provide a mechanism for you to clean up that people forget or don’t know to use. If you’re using jquery controls, you should be calling destroy. If you’re using knockout you should remember to call ko.cleanNode (despite the fact that there doesn’t seem to be any documentation for it).

Obvious Stuff

The simplest kind of memory leak is adding things to a data structure and never removing them. One easy way to do this is when logging. If you’re adding a log line to a DOM element or an array and never removing it, that’s a memory leak. One solution is to ensure your logs never grow beyond a particular maximum size by removing the oldest line every time you add a new one once you get over a particular size.

Memory Leaks That Are Not Your Fault (But you need to work around anyway)

Despite the fact that things have improved a great deal, there are still browser bugs that are likely to bite you when it comes to memory management.

XHR

It’s common when making a request to a server to create a local XMLHttpRequest object and add a listener to it, and then allow that XMLHttpRequest to go out of scope.  Now there is no code reachable from your own code that has a reference to that XHR, but of course it would be nonsense for the javascript engine to clean it up as if it were garbage, you’re actually expecting the request to happen and ultimately for the callbacks to get called.

Browsers like Chrome keep the xhr from being garbage collected only while the request is outstanding, although even that is longer than you might expect.  Once the request is completed, if there are no further references to the xhr, then it will be collected.  In IE8, any reference to the xhr from within the onreadystatechange callback function would stop the whole xhr (including any data it might have loaded) from being collected.  There is more information here.  My testing seems to suggest that it’s been fixed in IE9.

IE Map

Even basic object access can cause a leak in Internet Explorer.  In most browsers, if you add a key/value pair to an object, then delete that mapping, the memory of that mapping is reclaimed.  In IE, even IE9, while the mapping itself is reclaimed, the key is not.  You can see this by writing code that stores something under an enormous key and then deletes that key.

var globalObject = {};

function start() {
    var randomKey = new Array(5000000).join(String(Math.random()));
    globalObject[randomKey] = true;
    delete globalObject[randomKey];
}

If you run start a few times in Chrome, it will use a lot of memory and then reclaim it all, while if you run it in IE9, it will grab the memory and never let go. You can easily push it up to its maximum at which point it will start throwing Out Of Memory errors.

This is usually not a serious problem, as in most maps you use a small number of short keys, however if you are creating something like a cache or anything that will have a large number of unique keys added and removed over a long period of time it can become significant.

Interestingly, this leak manifests itself in another way. If you iterate over the keys of an object, they are returned in the order that they are added. In most browsers, if you add a key, then remove it, then add it later, iteration over the keys will show that key in the second place it was added.  In IE, the key is returned in the order of when it was first added, indicating that IE has ‘remembered’ that it had been previously added.  Here’s some example code:

var x = {};
x.first = 1;
x.second = 2;
x.third = 3;

// Logs "first", "second", "third" in all browsers.
for(var i in x){console.log(i)};

delete x.second;

// Logs "first", "third" in all browsers.
for(var i in x){console.log(i)};

x.second = 2;

// Logs "first", "third", "second" in Firefox and Chrome
// but "first", "second", "third" in IE.
for(var i in x){console.log(i)};

Things that Make Leaks Worse

Functions have access to variables from the scope in which they are defined, a feature called ‘lexical scoping’.  When a function accesses a variable from its defining scope, it is said to ‘close over’ the variable, and such functions are often called closures.  This means that if you have a reference to a closure, then it may hold references to variables from where it was defined that cannot be garbage collected.

IE will keep all the elements within the lexical environment alive while a closure is referenced, but many browsers will work out exactly which ones are accessed and only keep those from being garbage collected.

You’ll sometimes find lots of nulling of references in clean up code.  That usually indicates that the developer knows that this object is leaking, but isn’t sure why.  Nulling references in clean up code doesn’t stop the leak, but reduces how much is leaked.  Far better to find out why and fix that.

Finding Leaks

Finding leaks can be pretty dispiriting since if the same objects are leaked in multiple places, you can fix a leak and see no improvement in memory usage.  It’s only when you remove the final leak pointing to that object that you start to see an improvement.

Years ago, tracking down leaks for us would be a tedious process of observing a leak, commenting out swathes of functionality and then trying the code again to see how much it leaks, rinsing and repeating.  We used to also insert special code around large datastructures to make it easy to find out if they were growing more than expected.

There are tools now that allow you to inspect heaps and compare them with each other.  In IE, we often use dynatrace, although these days we tend to start by debugging with Chrome heap snapshots. A typical session might involve taking a snapshot, doing something that might leak 7 times, then taking another snapshot and examining the difference, starting by looking for objects with counts of multiples of 7.  There’s a good article talking about how to go about finding leaks here.

While fixing things in Chrome first doesn’t mean you’ve definitely fixed everything in IE, it’s an easy way to get started, as things that leak in Chrome almost certainly leak everywhere, and often leak worse in IE.

After some much needed sleep we’re able to reflect on the inspired (and at times manic) hacking that took place this weekend in New York. Over 30 teams presented their demos competing for the grand prize of $10,000 from Novus.

Hack Projects
The various hacking projects can be found here on Hacker League

There were some great demos put forward by all participants and I think everyone was impressed with the creativity of the attendees and also their caffeine consumption capabilities.

Best use of Caplin Technology Prize
Congratulations to FinGraph for winning our $500 Apple Voucher prize with their “amazing Graph RESTful API for discovery and visualization of financial relationships”!

Networking
These type of events present a great opportunity to not only work on some really cool stuff, but also to network with fellow developers and companies and build up good relationships. I think we can safely say we had the coolest t-shirts at the event!

The event schedule was well structured and the organisation by Nick and Francesca was excellent. All attendees were more than catered for in terms of snacks and good food throughout the night, generously donated by the sponsors. The look on some faces when it was announced Ice Cream Sandwiches were going to be available just past midnight was quite a joyous moment.

Caplin and OpenFin hack project

As well as assisting teams using our APIs, tools, data and services, we also formed our own hack team. We worked with OpenFin to create a desktop-integrated HTML5 alerts notification application. The app integrated BladeRunner, Caplin Trader Blades, Caplin Platform’s Alerting Service and OpenFin’s Desktop. This meant we could deliver a slick, multi-window application, with the UI elements built entirely in HTML5, but with the feel of a native app.

The idea was to provide traders with a mailbox-like view of alerts that may happen during their trading day. We wanted it to be minimally obtrusive, but to allow traders to focus on specific alerts for action now or later, or ignore those that were less interesting. The alerts might be to do with instrument price thresholds being reached, trades or orders being filled, or anything else. For the purposes of the hackathon, we focused on FX price thresholds.

How did it work?

We used a Caplin Trader blade to set up alerts that were managed by the alerts service, hosted at platform.caplin.com. OpenFin used the Caplin Alerts API to receive notifications when alert triggers occurred, and then displayed a small, borderless notification window containing the alert summary.

The user then makes a decision – ignore the alert, in which case it will fade out in time, or use a gesture to action it. Swipe right to snooze or dismiss the alert (inspired by mailbox), swipe left to add to a watchlist, or double-click on the instrument to open a one-click trade tile.

We also used some cool animations and transparency capabilities in OpenFin to make the UX really slick. After the swipe you would see the alert animate across to the watchlist and dock itself with any other instruments already present.

When you want to trade, either from the watchlist or double-clicking, a new borderless window popped up containing a Caplin Trader FX trade tile blade. This had streaming one-click tradable prices coming from platform.caplin.com, and the ability to execute a trade immediately.

Overall we were really pleased with the outcome – we’d built a well integrated app, using features from OpenFin and Caplin,  that would significantly reduce the workflow steps needed by a trader to react to an alert and take action. And it was animated, with rounded corners and transparency, so it must be cool!