When an end to end acceptance test fails, the failure could be due to any one of a large number of source lines or expectations changes across a large body of code.  Debugging may be quick, but often isn’t.

Small tests run faster too! While a typical small test may take 10 milliseconds to run, a large acceptance test may take twenty minutes.

Just a small back of the envelope calculation tells us that if a developer checks in twice a day and saves an hour each time because potential issues had been found before check-in with fast small tests, and the faster feedback, or has reduced their time spent debugging – that is two hours a day per developer that could be put to better use.

2 hours a day * thirty developers * two working weeks = 600 additional hours per two week iteration available.

This is equivalent to adding seven and a half developers to the team!

What value do you get from tests of different sizes?

Ignoring this testing just generates projects which will fail – not may fail – will fail, categorically will fail!

There are two sets of tests that are critical here…

Good unit test coverage and a TDD approach to development ensure that the rest of the organisation is supplied with product that works as developers expect it to.

Component tests and functional acceptance tests ensure that the product does what the product owner expects it to.

• Practically, the only way to do solid testing on new features is to have a stable product.
• Practically, the only way to assure a stable product is to either have solid regression testing on each release or to not make any changes.
• Practically, the only way to make sure that adequate regression testing is done is to automate the regression tests (unit, component, functional).

Without these tests in place, iterating quickly places an impossible to meet burden of verification demand on the testers and product owners, which just gives you a fragile process not an agile one.

Is your process Agile or Fragile?

This article addresses the issue of how to identify, with confidence, whether an API is backwardly compatible or not. As I mentioned previously there are times when an API is just too incorrect to be kept, however the decision for a breaking change must be a deliberate one – not an accidental one that is typically only discovered when the users of the API find that their existing code no longer works.

Preventing Breaking Changes

It is easy to tell developers that they should keep an API backwardly compatible, however even with the best of intentions it is possible to introduce subtle changes to the behaviour, particularly if you are refactoring collaborating classes that may not directly be part of the public interface. Every developer should be aware of the importance of keeping the API backwardly compatible, however this cannot be relied upon. After all, every developer should be striving to keep their code bug free, but we still write tests to verify this.

The solution is to detect any API changes before it is released, preferably in an automated fashion, so that it can be fixed. The problem is how to know whether an API has actually changed.

Compilation

A starting point for a compiled language, such as Java, is whether some code that was written against the old API still compiles against the new one. Any interfaces that have changed, including those with changes to method signatures and so on, will be flagged up immediately by the compiler. However this is no guarantee that the behaviour of the API hasn’t changed, and this simple verification isn’t available for non-compiled languages like JavaScript.

Unit Tests

The next obvious tool to detect a change are unit tests. The problem with this is that the unit tests will also be updated when the underlying code is modified. The fact that a unit test for a public API is having to be changed should start alarm bells ringing for the developer making the change; are the changes that they are making backwardly compatible or not? Again, however, the problem with this is that sometimes the changes can be subtle and the breaking change may be missed, particularly if the changes are made to a collaborating class rather that the one that is directly part of the public API.

Acceptance Tests

Acceptance tests can be used to detect breaking changes in a similar way to the unit tests. My problem with this approach is that they tend to be less granular and therefore susceptible to missing subtle breaking changes. Ideally I want my tests for API changes to have excellent coverage and to test all possible code paths, which is usually something that isn’t done by acceptance tests.

API Tests

Taking a step back, it does look like unit tests seem a good match for solving this particular problem. The only issue is that the unit tests will be modified when the code is changed, which may mask any breaking changes that have unwittingly been introduced. However we do have a set of unit tests perfectly suited to this which haven’t been modified, those from our previous release. We know that they proved the API for that release worked, so if the same tests pass against the new code it stands to reason that we should be backwardly compatible.

Creating API Tests

At Caplin we have a set of API tests that are a specialised group of unit tests aimed at verifying that all aspects and behaviours of the public API have not changed. They are usually a subset of the unit tests written for a particular library.

Only Running the API Tests

The first problem is specifying which tests are those for the API and which are standard unit tests. This is important since the implementation of collaborating classes can be changed without impacting the public API. As a result the old unit tests for refactored collaborating classes may now fail, however these are unimportant if they have no impact on the API.

Within our JavaScript unit tests we have added the ability to flag a particular test as an API test. We can specify whether we want to run only the API tests or all the tests, including the API tests.

Alternate approaches would be to create different test suites for the API and standard tests, or perhaps to use a custom annotation like @ApiTest rather than @Test in JUnit 4.

Dangers With Mocking

One danger with testing APIs is the use of mock objects. Mock objects combined with dependency injection provide a powerful way to test the logic of a particular piece of code as a single unit, without having to rely on the real logic in the collaborating objects. Instead the mock object acts as a replacement for the real collaborating object, verifying that it is invoked in the correct way and returns the appropriate values.

The danger is that changes to the collaborating object, such as the addition of a statement that throws a runtime exception if an argument is null, will not be emulated by the mock object. These problems are not insurmountable, however you need to be aware of the potential risks. In the case above there is a reasonable argument that there should have been an API test written for the collaborating class that demonstrated that the arguments could have been null previously which would now fail.

Getting the API Tests

At Caplin we upload the API tests to our Maven repository along with the other build artifacts for our libraries. For example when we release version 1.0 of a library, the API tests for 1.0 are uploaded to Maven. When we are building version 1.1 of the library we download the API tests for 1.0 to verify that the API has not changed.

Alternatives to getting the tests from Maven are to use another dependency management system, to pull the tests from a branch or label from source control, or even copying them from a network share.

Updating API Tests

It is possible that a public API hasn’t changed, however the API test for it fails. The reason for this could be that the interaction between the API method and its collaborating objects, particularly if they have been mocked out, have changed, and the test expectations are no longer valid. As a result the code for the API tests will need to be updated.

This change needs to be managed via source control, with a new version of the API tests pushed into Maven, or wherever they are being stored.

Backward Compatibility With Older Versions

At the moment we are only checking backward compatibility against the previous version of the API. I believe that if an API is compatible with the previous one, and that previous one was compatible with the one before, and so on, then it stands to reason that the current API should be compatible with the oldest one.

Conclusions

We are still in the early days of trying out this approach, however the early signs have been encouraging.

Ultimately a large amount of the responsibility still lies with developers to ensure that their code changes to a public API maintain backwards compatibility. The API tests are a safety net that gives confidence that it really is backwardly compatible and provide an early warning of any breaking changes, allowing them to be fixed before a release is made.

Any breaking changes within an API must be a conscious decision, not an inadvertent one.

One of the main reasons we were keen to adopt JsUnit five years ago was its similarity to JUnit 3. Back then our developers had been using JUnit for a few years, and familiarity of the JsUnit API meant that we could immediately start writing tests. Unfortunately the old adage “familiarity breeds contempt” reared its ugly head, and we discovered that our main issue with JsUnit were the occasional, very subtle, differences between it and JUnit. The logic for a test that works in JUnit might not work in JsUnit.

An example of this is highlighted by the test that would be written for the example JobQueue class defined below. Its purpose is to queue JavaScript functions for invocation at some point in the future. The addJob() method employs the fast-fail philosophy to throw an exception if the fJobToExecuteLater argument is null or undefined, rather than fail later on when an attempt is made to execute it.

function JobQueue()
{
   this.m_pQueue = [];
}

JobQueue.prototype.addJob = function(fJobToExecuteLater)
{
   if (fJobToExecuteLater == null) // this is true if it is undefined
   {
      throw new Error("JobQueue.addJob: Job cannot be null or undefined");
   }
   this.m_pQueue.push(fJobToExecuteLater);
};

// other methods...

The JsUnit test written to verify the behaviour would look something like:

var g_oJobQueue = null;

function setUp()
{
   g_oJobQueue= new JobQueue();
}

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   try
   {
      g_oJobQueue.addJob(null);
      fail("Unexpected success invoking addJob() with null job");
   }
   catch (e)
   {
      // expected exception
   }
}

Casting a casual (JUnit 3) glance over this test, everything looks to be in order. The test checks that an exception is thrown if the addJob() method is passed a null argument and remembers to invoke fail() if the exception isn’t thrown. This latter point is important since if it is omitted then the test will pass regardless of whether an exception is thrown or not.

The equivalent test in Java would work without any problems, however in JavaScript there is a hidden bug within this test logic. Within JUnit the fail() method throws a subclass of Error, which won’t be caught by a catch block expecting an Exception. Unfortunately JavaScript has no such distinctions between errors and exceptions, and the call to fail() throws an exception which will be caught within the catch block.

Solutions to the Exception expection issue

There are two approaches that we have taken to resolving this issue.

1. We extended the JsUnit API to add a new assertion method, assertFails(), that encapsulates the logic necessary to ensure that an exception is thrown, and fail if it hasn’t. It takes three arguments:

   i. A comment to output if the test fails
   ii. A reference to the function to invoke which is expected to throw an exception, if this doesn’t throw an exception then the assertion fails
   iii. The exception that is expected to be thrown (optional, if omitted the assertion just ensures that an exception is thrown)

Using the addJob() test can be rewritten to use assertFails():

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   assertFails("Unexpected success invoking addJob() with null job",
                   function() {
                      g_oJobQueue.addJob(null);
                   });
}

2. We verify that the exception that is thrown is the one that we expect. This particular approach has been met with trepidation from some of our developers who are concerned that verification of exceptions will lead to fragile test code, prone to break if someone changes the text within an exception (presumably to make it more helpful). There are certainly pros and cons to this, however in my experience it is rare that changes are made to the descriptions passed into the exceptions thrown by our code, and the overhead of these verifications is negligible.

The original test code can be rewritten to assert that the exception is the expected one:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   try
   {
      g_oJobQueue.addJob(null);
      fail("Unexpected success invoking addJob() with null job");
   }
   catch (e)
   {
      var oExpectedException =
               new Error("JobQueue.addJob: Job cannot be null or undefined");
      assertEquals(oExpectedException.toString(), e.toString());
   }
}

Alternatively assertFails() can be used as such:

function testAddJobThrowsExceptionIfJobToExecuteLaterIsNull()
{
   assertFails("Unexpected success invoking addJob() with null job",
                   function() {
                      g_oJobQueue.addJob(null);
                   },
                   new Error("JobQueue.addJob: Job cannot be null or undefined"));
}

Conclusion

This article is not intended to be a criticism of JsUnit. The issue highlighted here is simply due to a difference between the JavaScript and Java languages with respect to their exception handling which led to some tests being marked as successful when they shouldn’t. Over the last five years I have found that JsUnit has been reliable unit testing framework and, combined with Mock4JS and a TDD philosophy, it can form the solid foundations from which enterprise level JavaScript APIs and applications can be built.

Since we made our decision to use JsUnit we have never really looked back. So far it has managed to do everything that we have needed from it. However if you are looking into unit testing your JavaScript, then this list of JavaScript unit testing frameworks should be a good starting point to uncover one that is suitable for you. Just remember, it doesn’t matter which framework you choose, you need to be aware of the subtle nuisances of JavaScript, compared to any other languages you might be familiar with, to avoid the sort of trap that we fell into.

What do I mean by not being able to “create an instance of yourself using new Class()“?

MyClassDefinition = function()
{
};

MyClassDefintion.prototype.myFunction = function()
{
    // do some stuff
};
MyClassDefinition = new MyClassDefinition();

Here we lose the ability to do new MyClassDefinition();

What do I mean by “objects that are dynamically created”?

// create an object using the object literal notation
var oObj = {};

// dynamically append a function
oObj.myFunction = function()
{
    // do some stuff
};

Code written in both these ways make it more difficult to create a mock using Mock4JS because it expects a function reference to be passed to the mock method call.

A way around this is to dynamically create a definition from the object. This is done using the following piece of code:

function createMockDefinition(oObject)
{
    var oMockedDefinition = new Function();
    for( var x in oObject )
    {
        // Only add functions to the dynamically created prototype
        if( typeof oObject[ x ] == "function" )
        {
            oMockedDefinition.prototype[x] = function(){};
        }
    }
    return oMockedDefinition;
};

With the above code we have now created a class definition that we can pass to the Mock4JS mock() function.

function myTest()
{
    var oObjectIWantToTest = {};
    oObjectIWantToTest.functionIWantToTest = function(){};

    var oMyMockDefinition = createMockDefinition( oObjectIWantToTest );
    var oMyMock = mock( oMyMockDefinition );
    var oMyMockProxy = oMyMock.proxy();

    // setup expectations
    oMyMock.expects(once()).functionIWantToTest("myArgument");

    // call function and execute mock method.
    oMyMockProxy.functionIWantToTest("myArgument");

    // Finally, verify the expected mock methods have been called.
    // If not, an exception will be thrown.
    Mock4JS.verifyAllMocks();
};