Pitfalls of Exception expectations in JsUnit

At Caplin we use the JsUnit framework to test our JavaScript code. Over time we have made minor modifications to it, integrating Mock4JS and JSCoverage, as well as adding a few of our own extensions that are sympathetic to our JavaScript coding style, however, by and large, our tests are written using plain old JsUnit assertions.

One of the main reasons we were keen to adopt JsUnit five years ago was its similarity to JUnit 3. Back then our developers had been using JUnit for a few years, and familiarity of the JsUnit API meant that we could immediately start writing tests. Unfortunately the old adage “familiarity breeds contempt” reared its ugly head, and we discovered that our main issue with JsUnit were the occasional, very subtle, differences between it and JUnit. The logic for a test that works in JUnit might not work in JsUnit.

An example of this is highlighted by the test that would be written for the example JobQueue class defined below. Its purpose is to queue JavaScript functions for invocation at some point in the future. The addJob() method employs the fast-fail philosophy to throw an exception if the fJobToExecuteLater argument is null or undefined, rather than fail later on when an attempt is made to execute it.

function JobQueue()
 {
    this.m_pQueue = [];
 }
 JobQueue.prototype.addJob = function(fJobToExecuteLater)
 {
    if (fJobToExecuteLater == null) // this is true if it is undefined
    {
       throw new Error("JobQueue.addJob: Job cannot be null or undefined");
    }
    this.m_pQueue.push(fJobToExecuteLater);
 };
 // other methods…

The JsUnit test written to verify the behaviour would look something like:

var g_oJobQueue = null;
 function setUp()
 {
    g_oJobQueue= new JobQueue();
 }
 function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
 {
    try
    {
       g_oJobQueue.addJob(null);
       fail("Unexpected success invoking addJob() with null job");
    }
    catch (e)
    {
       // expected exception
    }
 }

Casting a casual (JUnit 3) glance over this test, everything looks to be in order. The test checks that an exception is thrown if the addJob() method is passed a null argument and remembers to invoke fail() if the exception isn’t thrown. This latter point is important since if it is omitted then the test will pass regardless of whether an exception is thrown or not.

The equivalent test in Java would work without any problems, however in JavaScript there is a hidden bug within this test logic. Within JUnit the fail() method throws a subclass of Error, which won’t be caught by a catch block expecting an Exception. Unfortunately JavaScript has no such distinctions between errors and exceptions, and the call to fail() throws an exception which will be caught within the catch block.

Solutions to the Exception expection issue

There are two approaches that we have taken to resolving this issue.

1. We extended the JsUnit API to add a new assertion method, assertFails(), that encapsulates the logic necessary to ensure that an exception is thrown, and fail if it hasn’t. It takes three arguments:

i. A comment to output if the test fails

ii. A reference to the function to invoke which is expected to throw an exception, if this doesn’t throw an exception then the assertion fails

iii. The exception that is expected to be thrown (optional, if omitted the assertion just ensures that an exception is thrown)
Using the addJob() test can be rewritten to use assertFails():

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
 {
    assertFails("Unexpected success invoking addJob() with null job",
                    function() {
                       g_oJobQueue.addJob(null);
                    });
 }

2. We verify that the exception that is thrown is the one that we expect. This particular approach has been met with trepidation from some of our developers who are concerned that verification of exceptions will lead to fragile test code, prone to break if someone changes the text within an exception (presumably to make it more helpful). There are certainly pros and cons to this, however in my experience it is rare that changes are made to the descriptions passed into the exceptions thrown by our code, and the overhead of these verifications is negligible.

The original test code can be rewritten to assert that the exception is the expected one:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
 {
    try
    {
       g_oJobQueue.addJob(null);
       fail("Unexpected success invoking addJob() with null job");
    }
    catch (e)
    {
       var oExpectedException =
                new Error("JobQueue.addJob: Job cannot be null or undefined");
       assertEquals(oExpectedException.toString(), e.toString());
    }
 }

Alternatively assertFails() can be used as such:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
 {
    assertFails("Unexpected success invoking addJob() with null job",
                    function() {
                       g_oJobQueue.addJob(null);
                    },
                    new Error("JobQueue.addJob: Job cannot be null or undefined"));
 }

Conclusion

This article is not intended to be a criticism of JsUnit. The issue highlighted here is simply due to a difference between the JavaScript and Java languages with respect to their exception handling which led to some tests being marked as successful when they shouldn’t. Over the last five years I have found that JsUnit has been reliable unit testing framework and, combined with Mock4JS and a TDD philosophy, it can form the solid foundations from which enterprise level JavaScript APIs and applications can be built.

Since we made our decision to use JsUnit we have never really looked back. So far it has managed to do everything that we have needed from it. However if you are looking into unit testing your JavaScript, then this list of JavaScript unit testing frameworks should be a good starting point to uncover one that is suitable for you. Just remember, it doesn’t matter which framework you choose, you need to be aware of the subtle nuisances of JavaScript, compared to any other languages you might be familiar with, to avoid the sort of trap that we fell into.