When an end to end acceptance test fails, the failure could be due to any one of a large number of source lines or expectations changes across a large body of code.  Debugging may be quick, but often isn’t.

Small tests run faster too! While a typical small test may take 10 milliseconds to run, a large acceptance test may take twenty minutes.

Just a small back of the envelope calculation tells us that if a developer checks in twice a day and saves an hour each time because potential issues had been found before check-in with fast small tests, and the faster feedback, or has reduced their time spent debugging – that is two hours a day per developer that could be put to better use.

2 hours a day * thirty developers * two working weeks = 600 additional hours per two week iteration available.

This is equivalent to adding seven and a half developers to the team!

What value do you get from tests of different sizes?

Ignoring this testing just generates projects which will fail – not may fail – will fail, categorically will fail!

There are two sets of tests that are critical here…

Good unit test coverage and a TDD approach to development ensure that the rest of the organisation is supplied with product that works as developers expect it to.

Component tests and functional acceptance tests ensure that the product does what the product owner expects it to.

• Practically, the only way to do solid testing on new features is to have a stable product.
• Practically, the only way to assure a stable product is to either have solid regression testing on each release or to not make any changes.
• Practically, the only way to make sure that adequate regression testing is done is to automate the regression tests (unit, component, functional).

Without these tests in place, iterating quickly places an impossible to meet burden of verification demand on the testers and product owners, which just gives you a fragile process not an agile one.

Is your process Agile or Fragile?

Although four years can seem like an eternity within software development, I found this video extremely pertinent to what I find myself doing today. I have been following the principles of Test Driven Development (TDD) for many years now, however this is the first time I have really delved into Behaviour Driven Development (BDD). I would thoroughly recommend watching the video if you have 45 minutes to spare, however, if not, I have summarised the notes I took whilst watching it below.

My Notes

• BDD is an evolution of TDD. Dave Astels described it as what TDD looks like if it is done well.
• When talking about unit tests, what is a unit? Is it a method? Is it a class? The definition of unit is ambiguous.
• BDD is attempting to move away from several of the words that Dave Astels believes have negative connotations after their use within TDD:
  • Instead of calling something a test it’s called a specification or spec for short.
  • Instead of using assertions we should use expectations instead. Anyone using mock objects will already be familiar with this concept.
• One of the key things to avoid is state based testing. Depending on the internal state of an object to verify a particular method works correctly breaks encapsulation, even if the test is intentionally meant to be friendly with the class it’s testing. This leads to fragile tests and creates a barrier to refactoring.
• The focus with RSpec was to make it readable. The writer and reader should not have to remember things like whether the first argument to the assertion is the actual or expected value.
• Although Dave Astels jokes that he has until version 1.0 to rename the should method with must it looks like he either lost the argument or changed his mind about it; RSpec is now version 1.3 and should is still being used.
• In TDD there is a tendency to create a one-to-one relationship between a class and its test. BDD says that we should be creating specs that verify a particular behaviour, rather than writing tests for each class and its methods. A coverage tool can be used to ensure that all possible code paths have been exercised.
  • The example that was given for this were the specs for EmptyCollection which might be “should return empty iterator”, “should have zero size” and “should return is empty true”.

Some BDD Frameworks

• RSpec – The BDD framework for Ruby that Dave Astels is talking about in the video.
• JBehave – A BDD framework for Java.
• JSpec – A BDD framework for JavaScript.
• Cucumber – A BDD framework with implementations in various languages, including Ruby, Java, .NET, Flex and web applications.

Please look here for a more comprehensive list.

One of the main reasons we were keen to adopt JsUnit five years ago was its similarity to JUnit 3. Back then our developers had been using JUnit for a few years, and familiarity of the JsUnit API meant that we could immediately start writing tests. Unfortunately the old adage “familiarity breeds contempt” reared its ugly head, and we discovered that our main issue with JsUnit were the occasional, very subtle, differences between it and JUnit. The logic for a test that works in JUnit might not work in JsUnit.

An example of this is highlighted by the test that would be written for the example JobQueue class defined below. Its purpose is to queue JavaScript functions for invocation at some point in the future. The addJob() method employs the fast-fail philosophy to throw an exception if the fJobToExecuteLater argument is null or undefined, rather than fail later on when an attempt is made to execute it.

function JobQueue()
{
   this.m_pQueue = [];
}

JobQueue.prototype.addJob = function(fJobToExecuteLater)
{
   if (fJobToExecuteLater == null) // this is true if it is undefined
   {
      throw new Error("JobQueue.addJob: Job cannot be null or undefined");
   }
   this.m_pQueue.push(fJobToExecuteLater);
};

// other methods...

The JsUnit test written to verify the behaviour would look something like:

var g_oJobQueue = null;

function setUp()
{
   g_oJobQueue= new JobQueue();
}

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   try
   {
      g_oJobQueue.addJob(null);
      fail("Unexpected success invoking addJob() with null job");
   }
   catch (e)
   {
      // expected exception
   }
}

Casting a casual (JUnit 3) glance over this test, everything looks to be in order. The test checks that an exception is thrown if the addJob() method is passed a null argument and remembers to invoke fail() if the exception isn’t thrown. This latter point is important since if it is omitted then the test will pass regardless of whether an exception is thrown or not.

The equivalent test in Java would work without any problems, however in JavaScript there is a hidden bug within this test logic. Within JUnit the fail() method throws a subclass of Error, which won’t be caught by a catch block expecting an Exception. Unfortunately JavaScript has no such distinctions between errors and exceptions, and the call to fail() throws an exception which will be caught within the catch block.

Solutions to the Exception expection issue

There are two approaches that we have taken to resolving this issue.

1. We extended the JsUnit API to add a new assertion method, assertFails(), that encapsulates the logic necessary to ensure that an exception is thrown, and fail if it hasn’t. It takes three arguments:

i. A comment to output if the test fails
ii. A reference to the function to invoke which is expected to throw an exception, if this doesn’t throw an exception then the assertion fails
iii. The exception that is expected to be thrown (optional, if omitted the assertion just ensures that an exception is thrown)

Using the addJob() test can be rewritten to use assertFails():

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   assertFails("Unexpected success invoking addJob() with null job",
                   function() {
                      g_oJobQueue.addJob(null);
                   });
}

2. We verify that the exception that is thrown is the one that we expect. This particular approach has been met with trepidation from some of our developers who are concerned that verification of exceptions will lead to fragile test code, prone to break if someone changes the text within an exception (presumably to make it more helpful). There are certainly pros and cons to this, however in my experience it is rare that changes are made to the descriptions passed into the exceptions thrown by our code, and the overhead of these verifications is negligible.

The original test code can be rewritten to assert that the exception is the expected one:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   try
   {
      g_oJobQueue.addJob(null);
      fail("Unexpected success invoking addJob() with null job");
   }
   catch (e)
   {
      var oExpectedException =
               new Error("JobQueue.addJob: Job cannot be null or undefined");
      assertEquals(oExpectedException.toString(), e.toString());
   }
}

Alternatively assertFails() can be used as such:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   assertFails("Unexpected success invoking addJob() with null job",
                   function() {
                      g_oJobQueue.addJob(null);
                   },
                   new Error("JobQueue.addJob: Job cannot be null or undefined"));
}

Conclusion

This article is not intended to be a criticism of JsUnit. The issue highlighted here is simply due to a difference between the JavaScript and Java languages with respect to their exception handling which led to some tests being marked as successful when they shouldn’t. Over the last five years I have found that JsUnit has been reliable unit testing framework and, combined with Mock4JS and a TDD philosophy, it can form the solid foundations from which enterprise level JavaScript APIs and applications can be built.

Since we made our decision to use JsUnit we have never really looked back. So far it has managed to do everything that we have needed from it. However if you are looking into unit testing your JavaScript, then this list of JavaScript unit testing frameworks should be a good starting point to uncover one that is suitable for you. Just remember, it doesn’t matter which framework you choose, you need to be aware of the subtle nuisances of JavaScript, compared to any other languages you might be familiar with, to avoid the sort of trap that we fell into.

What do I mean by not being able to “create an instance of yourself using new Class()“?

MyClassDefinition = function()
{
};

MyClassDefintion.prototype.myFunction = function()
{
    // do some stuff
};
MyClassDefinition = new MyClassDefinition();

Here we lose the ability to do new MyClassDefinition();

What do I mean by “objects that are dynamically created”?

// create an object using the object literal notation
var oObj = {};

// dynamically append a function
oObj.myFunction = function()
{
    // do some stuff
};

Code written in both these ways make it more difficult to create a mock using Mock4JS because it expects a function reference to be passed to the mock method call.

A way around this is to dynamically create a definition from the object. This is done using the following piece of code:

function createMockDefinition(oObject)
{
    var oMockedDefinition = new Function();
    for( var x in oObject )
    {
        // Only add functions to the dynamically created prototype
        if( typeof oObject[ x ] == "function" )
        {
            oMockedDefinition.prototype[x] = function(){};
        }
    }
    return oMockedDefinition;
};

With the above code we have now created a class definition that we can pass to the Mock4JS mock() function.

function myTest()
{
    var oObjectIWantToTest = {};
    oObjectIWantToTest.functionIWantToTest = function(){};

    var oMyMockDefinition = createMockDefinition( oObjectIWantToTest );
    var oMyMock = mock( oMyMockDefinition );
    var oMyMockProxy = oMyMock.proxy();

    // setup expectations
    oMyMock.expects(once()).functionIWantToTest("myArgument");

    // call function and execute mock method.
    oMyMockProxy.functionIWantToTest("myArgument");

    // Finally, verify the expected mock methods have been called.
    // If not, an exception will be thrown.
    Mock4JS.verifyAllMocks();
};