We kind of knew what was really delaying the release of documentation but it’s not until you make it transparent and have real measurements that it becomes something that cannot be ignored any more. As my old boss Mark Suster used to say “we manage what we measure”. This is where I think Kanban System’s greatest strength lies.

With Kanban we realised that while some of our documents are quickly produced, we have 7 documents in progress being “worked on” by 2 technical authors. I say “worked on” because all 7 are in some way being blocked by a developer who needs to provide input of some type. The average number of working days a document has been waiting for a developer is 14 days.

The documents in progress are waste – they offer no value to the business until we have released them to our clients, and this is costing us thousands!

There are three main reasons why this is happening:

1. Developers are moved onto another project before documentation is finished.

2. There are few documentation tasks that get placed directly on a team’s boards – and when they do they are mostly given low priority and are at risk of falling off the board completely.

3. It’s a job developer’s are aware of via email but given the lack of formal processes around it, it always can wait until tomorrow, consequently resulting in them never getting to it.

So what are we going to do about it?

I strongly believe that unless documentation tasks are on a team’s board and given the required focus then they become forgotten. So as a result of following Kanban we have initiated some small process improvements in our attempt to improve flow, reduce waste and deliver more value to Caplin.

• A Technical Author is now invited to the cycle kick off meeting to highlight blocked documents, discuss who can remove the block and write task cards which are given to the appropriate Project Leads to ensure these make it on the team’s board for a proper time allocation for that developer.
• Every week we will walk through the Documentation Kanban Board with all the Project Leads working together to remove the blocks and help flow so we can deliver documentation faster and get through the backlog quicker.

We are also exploring how cycle-time could help with the estimation process of when a document is likely to be finished by and thereby giving us more predictability.

I’ll keep you posted on how it goes.

We persevered with this approach for several months, repeatedly referring to the books on Scrum we had purchased and the numerous websites on the subject which were cropping up at the time, to find out ways of improving the process.

With hindsight this looks a little conservative, however back then we were still getting to grips with agile and didn’t want to make any significant adjustments that hadn’t been tried and proven elsewhere.

Eventually we built up our confidence in deciding on the sort of tweaks that we could make to the process ourselves. Coupled with this many of the new developers that joined our team at this stage also brought with them their own experiences of agile development, which we were able to use to induce change.

Some of these changes were very successful and are still integral parts of our process today, whilst others only lasted for a few sprints before being dropped for alternate solutions.

So, without further ado, welcome to our Hall of Fame. And to prove that progress doesn’t come without its fair share of mistakes, our Hall of Shame follows.

Hall of Fame

Here are some of changes that we made to the process which proved successful:

• We found the 4 week iteration cycle was too long, and restricted our ability to respond to customer needs promptly. We reduced our iterations to 2 weeks.
• We stopped calculating the number of ideal hours the team could theoretically take on. This was quite time consuming and project velocity seems to do an equally good job.
• We started to update our story board mid sprint, adding in new tasks (not stories!) as they come up. Initially we were loath to change the tasks on the board since this felt like some kind of scope creep. However those cards reflected what we thought we needed to do during the initial estimation meeting. Contact with the real code often highlighted things that we had forgotten or hadn’t even realised would need to be done, and the tasks on the board would no longer reflect what was really going on. A big personal thanks to Richard Chamberlain who introduced this practice to me.
• We now ensure that the tasks are all relatively small. The state of a task card is binary, it’s either done or not done (including all testing and documentation facets that might be associated with it). Monitoring how much work has been done and how much is left to do is hard if there are only a few task cards that will each take a few days to complete. Breaking those larger tasks down into their constituent parts will help provide visibility on the progress.
• We adopted the use of Extreme Programming (XP) style user stories and story boards instead of using a sprint backlog spreadsheet
• We split the development team out into separate scrums. The reasons for this were:
  • As the development team grew there were simply too many people to manage within a single scrum
  • A single scrum ended up containing stories that were deliverable for different customers or products. It was nearly impossible to decide which story to work on next during the sprint because they didn’t have comparable priorities. Was it more important to ensure all stories were completed for one customer, but none for another, or to ensure that some stories were completed for both?
  • Estimation was a serial process that took a longer and longer time, typically a whole afternoon, as the team grew and was able to take on more work items. Often only half of the developers would have any idea how long a particular task might take since it was in a product and/or language they didn’t know. By the end of the meeting everyone was tired, and were desperate to escape from it, which was probably a significant factor in some inaccurate estimates.
• We learnt how important it was to ensure the relative priorities are correct before determining which stories were taken into the sprint. After our estimation session had been completed we reported to the stakeholders what work had been signed up to. On a few occasions the stakeholders had assumed that a certain number of stories would be signed up to and were surprised to find out that sometimes they weren’t. When this happened the real priorities became apparent, and the team has to go back and start another estimation phase to address this. I think this is summarised by asking the stakeholders the question, “if we can only do one story, which must it be?”, then “if we can only do two stories, which should the second be?” and so on.
• We needed to focus the output of our retrospectives, rather than trying to take on too much change. The retrospectives themselves were very good, however we had a tendency to take many actions out of them, most, if not all, of which didn’t really get implemented. We now focus on agreeing the one or two most important actions to undertake from retrospective and focus on actually carrying them out.
• When we are unsure of how much effort might be needed to complete a new piece of work, we scheduled a spike (Ivan Moore’s terminology), or tracer bullet (Mike Cohn’s vernacular), within a sprint to drive out the risk and help come up with a more reasonable estimate on how long the real work will take.
• We changed the way we wrote our stories to focus on delivering thin vertical slices of functionality. This way we are more likely to deliver demonstrable features at the end of a sprint, even if things prove more difficult than we first expected.

Hall of Shame

Here are the changes that we made to the process that turned out to be less than successful:

• We attempted to improve estimates by tracking the actual time spent completing each task. At the end of the sprint we could look at the actuals compared to the estimates, and analyse the causes for any with large discrepancies. This Turned out to be complete overkill. Those stories that significantly overran were always identified without any analysis within the retrospectives anyway, and there was a small but noticeable overhead of capturing the actuals.
• We also went through a phase of trying to come up with better release plan level estimates by pre-planning a couple of iterations ahead. Unfortunately this broke the all important just in time aspect of agile development. Several times the stories that we had preplanned a few sprints ahead were de-prioritised as new customer requirements came in. These stories sat in the backlog slowly decaying. The time we spent on the pre-planning was wasted. Even if one of these stories made it into a sprint, and some didn’t even make it that far, we found that things had changed sufficiently within the code base such that the story needed to be estimated again. Nowadays we rely on higher level estimates combined with velocity to help with our release level planning.

One Size Doesn’t Fit All

The evolution of our development process has led me to the conclusion that one size doesn’t fit all. The ridiculous thing is how long it took me to realise this, since the first line of the agile manifesto pretty much points to this:

Individuals and interactions over processes and tools

Ultimately any successful agile development process that you read about within a book or a blog, such as this one, is a process that has been tailored to suit the team using it. Sometimes the author has experience of working with many agile teams and the insights that they have can be invaluable since they can provide generalised advice that is proven to have worked in several different environments. However they don’t have experience working in your team, with your unique blend of skill sets.

Even if your team is working on a similar type of problem and/or with identical technologies to another team, there is no guarantee that adopting its process will lead to success. Instead it can act as a good template to base your process on, one that can be adapted over time to suit your team’s unique skill sets.

Retrospective Evolution

This doesn’t mean that you shouldn’t look around for ideas on how to improve your development process. On the contrary, there is no point reinventing the wheel. An idea from another agile team might work perfectly within your team, or might work after some adjustments. Alternatively it may prove to be the seed that allows you to reach your own solution.

That said, the drive for the improvements to the process should come from the retrospectives and we should continually be striving to improve the it. For example:

1. Could any of the problems that were experienced during the last sprint have been avoided if there was a suitable process in place?
2. Alternatively are there any parts of the process that are no longer necessary or which can be streamlined?

The Saga Continues

Here at Caplin our development process continues to evolve. As has been mentioned before we have been trialling Kanban for a few of our internal development projects. It promises many potential improvements to the development process, the question is when will these changes occur, and what will they be.

Afterthoughts

Having written this article I have subsequently become aware that a whole track was dedicated to Agile Evolution at QCon London 2010, held between 8th and 12th of March. Much of what was spoken about in there echoes the experiences that we have had at Caplin and I would thoroughly recommend looking through the slides:

• Nothing New Under the Sun: continually rediscovering the good ways to build software by Keith Braithwaite
• http://qconlondon.com/london-2010/file?path=/qcon-london-2010/slides/RachelDavies_BreakingYourAgileAddiction.pdf by Rachel Davies
• Kanban – Crossing the line, pushing the limit or rediscovering the agile vision? by Jesper Boeg
• Product Development in the Land of the Free by Simon Baker & Gus Power

Of course at times the estimates have been inaccurate, with features taking far more time to implement than was initially predicted. The question is how to learn from these experiences to help improve your estimates. This is especially pertinent for software engineers within financial services because this same conundrum is at the heart of pricing financial instruments.

This article looks at how the Rational Expectations Theory can help you improve your estimates in software development. It starts by looking at estimating something much easier than a software enhancement, your journey time into work, and demonstrates when that estimate should be considered good or bad. It then looks how this can be applied to software development.

Rational Expectations Theory

A few years ago, I was introduced to the Efficient Market Hypothesis whilst reading about how stocks are priced. This hypothesis is built upon the Rational Expectations Theory. Paraphrasing this theory, it states that an individual bases their estimate on:

1. Their rational outlook
2. All the available information
3. Their previous experiences

Although both the hypothesis and the theory are out of favour with many economists, the principles of rational expectations struck a chord with me.

Rational Expectations applied to Journey Times

To demonstrate rational expectations in practice, consider the question of how long it will take you the next time you travel into work? Assuming that you don’t work at home, aren’t starting a brand new job and haven’t just moved homes, then this is something that you should be pretty good at. After all you have an abundance of experience making that journey.

My estimate for my door to door journey time would be an hour. However there is a pretty big deviation between my fastest and slowest ever times. Every now and again my interconnections align perfectly and I can make the journey in 45 minutes. On the other hand, it has taken me 3 hours when my train broke down and, to compound matters, the tube line I needed to complete my journey was closed.

Rational Expectation Theory says that estimate of 1 hour is good because it is the equilibrium of my times. An estimate of 45 minutes would be bad because it’s overly optimistic and I’d rarely expect to meet it. An estimate of 3 hours is also bad because it is too pessimistic since it has only occurred once, and the circumstances were exceptional.

The theory also states that the estimate is based on all available information. My estimate of 1 hour would no longer be reasonable if there was a weather forecast for tomorrow predicting snow. In Britain snow is renowned for crippling the travel network. It doesn’t matter whether I have heard the weather forecast or not, it is my responsibility to find out all relevant information to aid my estimate. Ignorance is no excuse.

Rational Expectations applied to Development

I like the Rational Expectation Theory because I think its three rules capture the essence of what estimating in software development is about. Prior to reading about the theory I didn’t think too much about how I came up with an estimate, it was pretty much a gut feel. Now I realise that subconsciously these rules were actually what were going through my mind. The theory helps me to break the estimate into its constituent parts.

Applying the theory to development leads to these conclusions:

• Two features on the backlog, in the same area of the code base, with a comparable depth and level of complexity should have similar estimates.
• Knowledge about how easily a new feature might be implemented within the current code base is important. If you know the code will need some refactoring in order to implement the new functionality cleanly, then this should be allowed for. Of course the law of diminishing returns applies here; familiarity with, rather than intimate knowledge of, how the feature might be implemented will be good enough in most cases.
• Allow for your past experiences with your code base. Take the case where the two similar features have been sized at 1 day (or story point, or other unit of measurement). Whilst working on the first you uncover some unexpected issues, and it ends up taking 4 days to complete. Does it still make sense for the second item to still be sized at 1 day? The answer to this depends on which of these two scenarios the overrun fell into:
  1. If the reason it took much longer to complete is systematic and likely to affect the estimates for all the other features, then project velocity will be able to compensate for it. The relative sizes of the estimates are still the same.
  2. If the reason was localised to the code surrounding that feature then it is only rational to keep the estimate at 1 day if I believe that the problems were an anomaly and won’t impact the other feature.

Well, that’s nice in Theory…

I find that the Rational Expectation Theory is a useful reminder of what to do when estimating, however in software development we are never building the same thing twice (we are DRY here aren’t we?). As a result the idea of an equilibrium point is non-existent; we are dealing with a unique piece of development with a sample set of one. However a lot of the time we are doing similar things that we’ve done before, so we do have a reasonable point of reference for comparison.

When is an Estimate bad?

In my example of a journey to work I estimated that it would take an hour. I should be pretty good at knowing how long this takes since I have made the journey over 2,000 times. If the journey takes me an hour and a half tomorrow does that make it a bad estimate? That depends. If there was planned engineering work causing a reduced service that I could have found out about in advance, then it was a bad estimate. I should have allowed for the impact of the engineering work. If the delay was caused by my train breaking down then the estimate was rational and therefore wasn’t bad. After all, it is just a estimate, not a guarantee.

The same logic applies to estimates given for development. We never have the luxury of having written exactly the same functionality many times previously, but there are often significant overlaps. Provided that your estimates are rational, then you can be confident that things should average out. Some features will take longer to write than estimated whilst others will take less time. Also don’t be afraid to re-estimate specific features if fresh evidence comes to light that shows that the estimate for a particular feature is irrational.

Afterthought: A World Free of Estimates

At Caplin we have spent the last few months trialling Kanban with several of our internal development teams, which offers a idyllic world where estimates aren’t needed.

I am still trying to grapple with the implications of this, and am looking forward to understanding it better. My gut feeling is even with Kanban estimates will still be necessary at a high level due to the inexorable fact that the priority of some features will always be based on the cost to build them, which is a multiple of number of people working on it by the time taken by them to build it.

A few sprints ago I moved to a team in charge of maintaining our existing products and reducing technical debt. We had a backlog of low priority bug fixes and enhancements to work through and no specific client deadlines, so as team lead I thought this would be a good opportunity to try out the kanban methodology. After a bit of consideration the team came up with this board layout:

We thought that the workflow for any given story would be along these lines:

• A developer or pair of developers picks the top story card from the backlog and moves it to column 1. We use the power of three concept for pre-planning, so at this point the developer will go and grab the product owner and a QA engineer to have a quick chat about what the task involves. The acceptance criteria will be determined at this point. The card is then moved to column 2.
• Any free developer can pick up the card from column 2 and move it to column 3 to start work on it. At this point the developer should break the story down into tasks and place the task cards in column 4 as he works on them.
• When the developer is finished with the story he will get a QA engineer (ideally the one involved in the pre-planning stage) for a quick “show me”, to demonstrate the fix. The card then moves to column 5.
• A free QA engineer will pick up the card from column 5 and move it to column 6, then test the change thoroughly. This involves verifying the fix in every browser that we support, attempting to find edge cases, checking that sufficient automated tests have been written and so on.
• When the QA engineer is satisfied, the card moves to column 7 and is done.

That’s what we thought would happen, and we were fairly confident that the board matched our actual workflow. However when we put it into practice we quickly discovered that the adage about no battle plan surviving first contact with the enemy was coming true – cards were bunny hopping across the board with scant regard for several of the columns:

It was soon fairly obvious why this was happening.

• After the pre-planning stage (column 1) the developer would always begin work on the task immediately and move it directly to column 3 – it would be very odd for a developer to gather up the relevant people for a planning session and then leave the card in column 2, going off to work on something else. Column 2 is therefore completely redundant.
• Most of our stories were fairly trivial bug fixes which resisted our attempts to break them down into sub-tasks. This meant that in practice the entire story card would just be moved straight into column 4, skipping column 3 completely. This probably indicates that our stories were too small – it would have made more sense for a story to take the form “Fix 5 bugs in product X”, which can naturally be split into sub-tasks.
• After the “show me” session the QA engineer would almost always take on the testing straight away, moving the card straight from column 4 to column 6. Cards very rarely paused along the way in column 5.

Needless to say, this all came out during our first sprint retrospective and our kanban boards have evolved since that first attempt. But it taught me something – you might think that you can draw out your workflow with your eyes closed, but how closely does your theory match reality?

At Caplin we are living proof that framework development can be achieved following agile principles. We have been successfully building frameworks and APIs for the past 5 years using various agile methodologies from Scrum, to a mix of Scrum and XP, to Kanban.

This article represents the approaches that we have undertaken to successfully build our frameworks.

The Problem with Refactoring Framework

The specific issue in question is the idea that code should be kept lean, with the bare minimum implemented to achieve the required functionality within an iteration. If further functionality requires this code be be modified, usually within another iteration, then no problem, just refactor it until it does what needs to do. The unit tests that you wrote previously can help give you confidence that the functionality of any unmodified code will still work as expected, whilst new or updated unit tests will cover the additional functionality.

The decision to refactor, even a large refactoring effort, isn’t too hard to make when you are in complete control of the code base, for example when you are producing the whole application. However this luxury isn’t so trivial when you’re developing a framework. By all means you can refactor your code, however if the code you want to change is part of the public API, then you need to consider what impact this might have on the users of your framework.

Techniques for Building Agile Frameworks

Fortunately there are a number of techniques that can be applied to tackle this problem, which are described below. I would also recommend reading the short, but excellent, Chapter “Evolving Frameworks” from Kent Beck’s book Implementation Patterns. It provides valuable insights on the subject, based on Beck’s own experience of making JUnit 4 backwardly compatible, despite some significant changes to the API.

Backwards Compatibility

One technique is to simply keep everything backwardly compatible, no matter how difficult it is to do this. All of the classes and methods provided previously must still be present and provide the same functionality, even if the underlying implementation has been modified. New functionality is provided in parallel to this.

Any parts of the framework that you would prefer your customers to stop using, even though it is still fully functional, should be flagged as deprecated, and the documentation should link to the classes or methods that should be used instead. This provides your customers a chance to migrate away from the parts of the API that you are unsatisfied with as it suits them. This seems to be the approach that has been taken with Java where parts of the API have been deprecated for a decade but can still be used.

Once parts of the framework have been deprecated for a while, you also have the option of making an intentional breaking change to remove them. This was the approach taken for Python 3, which is deliberately backwardly incompatible with earlier versions. In this case a automated migration tool was created to convert from Python 2 to Python 3 code to help soften the blow. A similar approach is a good idea to help your customers identify the areas of their code they will need to update once they take the new version of your framework with the breaking change.

Versioned APIs

A specialisation of keeping everything backwardly compatible is to introduce a versioned API. A versioned API can be used when you have realised that one of the interfaces that your customer’s code implements is missing a particular method. Instead of adding the method to the existing interface, which will cause a compilation error for your customer, you can create a new interface that extends the original one. The new method can be added to the new interface, and your code that calls that method can check whether the object instance implements the old interface, in which case the method is not invoked, or the new interface, where it is. As before, it is worthwhile deprecating the old interface.

An example of a versioned API can be seen with the JMX interface NotificationEmitter, which extends NotificationBroadcaster, adding a single method: removeNotificationListener(NotificationListener listener, NotificationFilter filter, Object handback).

A problem with this approach is if you need to add a versioned API on top of an existing one. These multiple layers are likely to make the calling code turn pretty nasty, and can bloat out your API. At this stage it is worth considering if it is possible to make a breaking change.

Another problem is what to call the versioned API. The JMX example used different names NotificationEmitter and NotificationBroadcaster, although without knowing the API I wouldn’t be able to say which one superseded the other. It might also be difficult to come up with a different name that is meaningful. An alternate naming convention is to number to interface, for example IHTMLDocument2. My main issue with this approach is that it can complicate matters if you ever decide to remove deprecated parts of the API. If IHTMLDocument was to be removed, it looks odd to have an interface called IHTMLDocument2 hanging around – surely it should be renamed IHTMLDocument, however this enters a compatibility minefield.

Breaking Change

Most of the time, I consider breaking changes to be a last resort. Breaking changes are always going to be frustrating for your customers, and if you keep forcing them to rewrite their code you’ll soon find that they have ceased to be your customer. However, as I have alluded to earlier, there can be good reasons to make breaking changes.

Maintaining deprecated parts of an API can become an expensive overhead, bloating out what might otherwise be a concise and neat API. Periodically removing these deprecated APIs can be worthwhile, provided that your customers have had adequate time and warning to move onto the new APIs.

Alternatively it might be very difficult to make an API backwardly compatible, or an API may be just too wrong to be kept. In both cases it is vital to explore all the possibilities suggested earlier to see if there is any way to avoid making the breaking change. If it is necessary to make the breaking change, consider how you can lessen the impact for your customers.

Always Consider Your Customer

Regardless of the reasons behind the breaking changes, it is vital that you focus on making your customer’s experience as pain free as possible. After all they want to focus their efforts on adding their business value on top of your framework, not bug fixing issues introduced due to changes to your framework.

• Clearly document which parts of the API are deprecated and provide clear information about what should be used instead. Also give your customers ample warning time about when a deprecated API will be removed.
• Provide a tool to identify code that will be impacted by the breaking change, and provide hints to what changes need to be made to it, or even automatically correct the code. In some cases a good IDE can help identify and correct breaking changes, however you shouldn’t assume that it will without testing it first. If a tool or IDE is infeasible, then a clear set of instructions for what to look for might be adequate.
• Flag which areas of the API are still young and may be subject to change. At Caplin we document newer parts of the API as beta. Early adopters are able to take advantage of the latest features, however they are warned that the API may still evolve. Although this can still be frustrating for our customers this is partially mitigated by the fact that we are listening to their feedback about the beta API and are feeding that back into the next version of the API, making it something even easier to use.
• Try to hold off making breaking changes until a major version release. It is incredibly frustrating if breaking changes are made in minor incremental releases.
• Provide consultancy to your customers to help with the upgrade to the version with the breaking changes.

Conclusion

There is no reason why you can’t build a framework whilst employing an agile methodology. Most of the principles of agile are equally valid for both framework and application developers. The main issue is that your ability to refactor the framework code is restricted when it comes to the public API. As a result, the importance of getting an API “right” the first time may seem vital when developing a framework compared to an application, however failure to create this “right” API is not a disaster. As shown earlier, there are several techniques that can be applied to build up a framework in an iterative and agile way.

At times it is very difficult to maintain backwards compatibility, and sometimes breaking changes are impossible to avoid, but the role of a framework provider should be focused on empowering the users of the framework, its customers, to build their software reliably and efficiently upon it, rather than spending all their time bug fixing issues introduced by new versions of the framework. As a result the processes and practices for agile framework development need to aligned with these goals – however that is a topic for another day.

In two weeks time I will be following this up with a look at one of the processes that we have put into place to help develop our frameworks, “Testing APIs for Backwards Compatibility”.

1. It’s very difficult to use velocity for Scrum sign-up because team personnel change frequently
2. Unexpected work can impact the project teams resulting in a demotivated team because they sometimes do not complete what they have signed-up for

There must be a better way?

I have been thinking about this for some time and our vision is continuous releases (apologies to the Marketing Department) whereby development teams pull work from priority boards, with the work moving through the necessary stages, culminating in the final stage being “release into production”.  A blissful continuous work stream where capacity and demand are in harmony and value is pulled through the system and delivered to the business as early as possible. Idealistic?  Maybe, but there seems to be some momentum gathering in the world of “Kanban Systems”, which appears to deliver our vision, so here at Caplin we are trialling this new way of working with one project team.

If you want to find out more about Kanban Systems and Lean techiques these four websites are a good start:

• http://availagility.co.uk
• http://www.limitedwipsociety.org
• http://www.crisp.se
• http://www.agilemanagement.net

We will also have some attendees at the upcoming 1-day Lean and Kanban Exchange on the 1st December, run by the always excellent Skills Matter.