We persevered with this approach for several months, repeatedly referring to the books on Scrum we had purchased and the numerous websites on the subject which were cropping up at the time, to find out ways of improving the process.

With hindsight this looks a little conservative, however back then we were still getting to grips with agile and didn’t want to make any significant adjustments that hadn’t been tried and proven elsewhere.

Eventually we built up our confidence in deciding on the sort of tweaks that we could make to the process ourselves. Coupled with this many of the new developers that joined our team at this stage also brought with them their own experiences of agile development, which we were able to use to induce change.

Some of these changes were very successful and are still integral parts of our process today, whilst others only lasted for a few sprints before being dropped for alternate solutions.

So, without further ado, welcome to our Hall of Fame. And to prove that progress doesn’t come without its fair share of mistakes, our Hall of Shame follows.

Hall of Fame

Here are some of changes that we made to the process which proved successful:

• We found the 4 week iteration cycle was too long, and restricted our ability to respond to customer needs promptly. We reduced our iterations to 2 weeks.
• We stopped calculating the number of ideal hours the team could theoretically take on. This was quite time consuming and project velocity seems to do an equally good job.
• We started to update our story board mid sprint, adding in new tasks (not stories!) as they come up. Initially we were loath to change the tasks on the board since this felt like some kind of scope creep. However those cards reflected what we thought we needed to do during the initial estimation meeting. Contact with the real code often highlighted things that we had forgotten or hadn’t even realised would need to be done, and the tasks on the board would no longer reflect what was really going on. A big personal thanks to Richard Chamberlain who introduced this practice to me.
• We now ensure that the tasks are all relatively small. The state of a task card is binary, it’s either done or not done (including all testing and documentation facets that might be associated with it). Monitoring how much work has been done and how much is left to do is hard if there are only a few task cards that will each take a few days to complete. Breaking those larger tasks down into their constituent parts will help provide visibility on the progress.
• We adopted the use of Extreme Programming (XP) style user stories and story boards instead of using a sprint backlog spreadsheet
• We split the development team out into separate scrums. The reasons for this were:
  • As the development team grew there were simply too many people to manage within a single scrum
  • A single scrum ended up containing stories that were deliverable for different customers or products. It was nearly impossible to decide which story to work on next during the sprint because they didn’t have comparable priorities. Was it more important to ensure all stories were completed for one customer, but none for another, or to ensure that some stories were completed for both?
  • Estimation was a serial process that took a longer and longer time, typically a whole afternoon, as the team grew and was able to take on more work items. Often only half of the developers would have any idea how long a particular task might take since it was in a product and/or language they didn’t know. By the end of the meeting everyone was tired, and were desperate to escape from it, which was probably a significant factor in some inaccurate estimates.
• We learnt how important it was to ensure the relative priorities are correct before determining which stories were taken into the sprint. After our estimation session had been completed we reported to the stakeholders what work had been signed up to. On a few occasions the stakeholders had assumed that a certain number of stories would be signed up to and were surprised to find out that sometimes they weren’t. When this happened the real priorities became apparent, and the team has to go back and start another estimation phase to address this. I think this is summarised by asking the stakeholders the question, “if we can only do one story, which must it be?”, then “if we can only do two stories, which should the second be?” and so on.
• We needed to focus the output of our retrospectives, rather than trying to take on too much change. The retrospectives themselves were very good, however we had a tendency to take many actions out of them, most, if not all, of which didn’t really get implemented. We now focus on agreeing the one or two most important actions to undertake from retrospective and focus on actually carrying them out.
• When we are unsure of how much effort might be needed to complete a new piece of work, we scheduled a spike (coined by Kent Beck and Ward Cunningham, popularised within Caplin by Ivan Moore), or tracer bullet (from The Pragmatic Programmer by Andrew Hunt and David Thomas, introduced to Caplin by Mike Cohn), within a sprint to drive out the risk and help come up with a more reasonable estimate on how long the real work will take (thanks to Alistair Cockburn for providing the insight into the originators of these terms).
• We changed the way we wrote our stories to focus on delivering thin vertical slices of functionality. This way we are more likely to deliver demonstrable features at the end of a sprint, even if things prove more difficult than we first expected.

Hall of Shame

Here are the changes that we made to the process that turned out to be less than successful:

• We attempted to improve estimates by tracking the actual time spent completing each task. At the end of the sprint we could look at the actuals compared to the estimates, and analyse the causes for any with large discrepancies. This Turned out to be complete overkill. Those stories that significantly overran were always identified without any analysis within the retrospectives anyway, and there was a small but noticeable overhead of capturing the actuals.
• We also went through a phase of trying to come up with better release plan level estimates by pre-planning a couple of iterations ahead. Unfortunately this broke the all important just in time aspect of agile development. Several times the stories that we had preplanned a few sprints ahead were de-prioritised as new customer requirements came in. These stories sat in the backlog slowly decaying. The time we spent on the pre-planning was wasted. Even if one of these stories made it into a sprint, and some didn’t even make it that far, we found that things had changed sufficiently within the code base such that the story needed to be estimated again. Nowadays we rely on higher level estimates combined with velocity to help with our release level planning.

One Size Doesn’t Fit All

The evolution of our development process has led me to the conclusion that one size doesn’t fit all. The ridiculous thing is how long it took me to realise this, since the first line of the agile manifesto pretty much points to this:

Individuals and interactions over processes and tools

Ultimately any successful agile development process that you read about within a book or a blog, such as this one, is a process that has been tailored to suit the team using it. Sometimes the author has experience of working with many agile teams and the insights that they have can be invaluable since they can provide generalised advice that is proven to have worked in several different environments. However they don’t have experience working in your team, with your unique blend of skill sets.

Even if your team is working on a similar type of problem and/or with identical technologies to another team, there is no guarantee that adopting its process will lead to success. Instead it can act as a good template to base your process on, one that can be adapted over time to suit your team’s unique skill sets.

Retrospective Evolution

This doesn’t mean that you shouldn’t look around for ideas on how to improve your development process. On the contrary, there is no point reinventing the wheel. An idea from another agile team might work perfectly within your team, or might work after some adjustments. Alternatively it may prove to be the seed that allows you to reach your own solution.

That said, the drive for the improvements to the process should come from the retrospectives and we should continually be striving to improve the it. For example:

1. Could any of the problems that were experienced during the last sprint have been avoided if there was a suitable process in place?
2. Alternatively are there any parts of the process that are no longer necessary or which can be streamlined?

The Saga Continues

Here at Caplin our development process continues to evolve. As has been mentioned before we have been trialling Kanban for a few of our internal development projects. It promises many potential improvements to the development process, the question is when will these changes occur, and what will they be.

Afterthoughts

Having written this article I have subsequently become aware that a whole track was dedicated to Agile Evolution at QCon London 2010, held between 8th and 12th of March. Much of what was spoken about in there echoes the experiences that we have had at Caplin and I would thoroughly recommend looking through the slides:

• Nothing New Under the Sun: continually rediscovering the good ways to build software by Keith Braithwaite
• http://qconlondon.com/london-2010/file?path=/qcon-london-2010/slides/RachelDavies_BreakingYourAgileAddiction.pdf by Rachel Davies
• Kanban – Crossing the line, pushing the limit or rediscovering the agile vision? by Jesper Boeg
• Product Development in the Land of the Free by Simon Baker & Gus Power

Over the last five years we have learnt that an agile process is agile itself, and the process that we follow now is different in many ways from that which we initially adopted. This doesn’t mean that the process we first had was wrong, although there were certainly some issues. Over time things that were crucial parts of the process have become defunct and have been removed, whilst new ideas and refinements are added into the process to address fresh issues that have arisen.

Straight from the Book: The First Iteration

The first iteration started in February 2005. Being new to the process we followed one pretty much straight out of the book. We were desperate to adopt Scrum successfully and wanted to follow all the best practices:

1. The sprints were exactly 4 weeks long
2. Our development team of 7 or 8 developers were organised into a single Scrum
3. The stakeholders provided a prioritised list of the work items they wanted us to undertake
4. A sprint backlog spreadsheet was created and maintained
5. Another spreadsheet was created that calculated the theoretical number of ideal hours we had available for the sprint, allowing for developer holidays and other commitments
6. The highest priority work item was broken down into tasks which were then estimated in ideal hours
7. If the total number of ideal hours of estimated work was less than the theoretical number of ideal hours we could undertake, then we estimated the next work item, and so on, until we had used up the theoretical hours (we always left ourselves a little slack as previous experience told us that things tended to overrun)
8. The team then assessed the provisional list of work items for the sprint and decided whether they could commit to delivering them; if not then the lowest priority work item would be removed and the team would decide if they could commit again
9. Every morning we held a stand up so that each member of the team could report on their progress and the sprint backlog and associated burn down chart could be updated
10. At the end of the sprint a demo of the new software was held to show the progress that had been made to the stakeholders
11. A retrospective was also held to identify things that were working well and things that weren’t, so that we could address them as part of the next sprint

A Need for Evolution

Five years on and, for me, the analogy of marriage and an agile software development is an apt one. In order to be healthy they need to be dynamic and able to adapt to change in circumstances. Within a marriage there are mundane decisions and changes to routines such as who is responsible for washing up or ironing this week, through to major changes, like after the birth of a child (particularly after a son who refuses to sleep). In software development there are small changes to the process such as a decision to restrict the number of actions to take from a retrospective, through to larger changes like the adpotion of XP practices within your Scrum process.

In the second post of this series “Agile Software Development – One Size Doesn’t Fit All” – I’ll give details on exactly how we evolved the process at Caplin and provide examples of our problems and our solutions from both our Halls of Fame and Shame.

At Caplin we are living proof that framework development can be achieved following agile principles. We have been successfully building frameworks and APIs for the past 5 years using various agile methodologies from Scrum, to a mix of Scrum and XP, to Kanban.

This article represents the approaches that we have undertaken to successfully build our frameworks.

The Problem with Refactoring Framework

The specific issue in question is the idea that code should be kept lean, with the bare minimum implemented to achieve the required functionality within an iteration. If further functionality requires this code be be modified, usually within another iteration, then no problem, just refactor it until it does what needs to do. The unit tests that you wrote previously can help give you confidence that the functionality of any unmodified code will still work as expected, whilst new or updated unit tests will cover the additional functionality.

The decision to refactor, even a large refactoring effort, isn’t too hard to make when you are in complete control of the code base, for example when you are producing the whole application. However this luxury isn’t so trivial when you’re developing a framework. By all means you can refactor your code, however if the code you want to change is part of the public API, then you need to consider what impact this might have on the users of your framework.

Techniques for Building Agile Frameworks

Fortunately there are a number of techniques that can be applied to tackle this problem, which are described below. I would also recommend reading the short, but excellent, Chapter “Evolving Frameworks” from Kent Beck’s book Implementation Patterns. It provides valuable insights on the subject, based on Beck’s own experience of making JUnit 4 backwardly compatible, despite some significant changes to the API.

Backwards Compatibility

One technique is to simply keep everything backwardly compatible, no matter how difficult it is to do this. All of the classes and methods provided previously must still be present and provide the same functionality, even if the underlying implementation has been modified. New functionality is provided in parallel to this.

Any parts of the framework that you would prefer your customers to stop using, even though it is still fully functional, should be flagged as deprecated, and the documentation should link to the classes or methods that should be used instead. This provides your customers a chance to migrate away from the parts of the API that you are unsatisfied with as it suits them. This seems to be the approach that has been taken with Java where parts of the API have been deprecated for a decade but can still be used.

Once parts of the framework have been deprecated for a while, you also have the option of making an intentional breaking change to remove them. This was the approach taken for Python 3, which is deliberately backwardly incompatible with earlier versions. In this case a automated migration tool was created to convert from Python 2 to Python 3 code to help soften the blow. A similar approach is a good idea to help your customers identify the areas of their code they will need to update once they take the new version of your framework with the breaking change.

Versioned APIs

A specialisation of keeping everything backwardly compatible is to introduce a versioned API. A versioned API can be used when you have realised that one of the interfaces that your customer’s code implements is missing a particular method. Instead of adding the method to the existing interface, which will cause a compilation error for your customer, you can create a new interface that extends the original one. The new method can be added to the new interface, and your code that calls that method can check whether the object instance implements the old interface, in which case the method is not invoked, or the new interface, where it is. As before, it is worthwhile deprecating the old interface.

An example of a versioned API can be seen with the JMX interface NotificationEmitter, which extends NotificationBroadcaster, adding a single method: removeNotificationListener(NotificationListener listener, NotificationFilter filter, Object handback).

A problem with this approach is if you need to add a versioned API on top of an existing one. These multiple layers are likely to make the calling code turn pretty nasty, and can bloat out your API. At this stage it is worth considering if it is possible to make a breaking change.

Another problem is what to call the versioned API. The JMX example used different names NotificationEmitter and NotificationBroadcaster, although without knowing the API I wouldn’t be able to say which one superseded the other. It might also be difficult to come up with a different name that is meaningful. An alternate naming convention is to number to interface, for example IHTMLDocument2. My main issue with this approach is that it can complicate matters if you ever decide to remove deprecated parts of the API. If IHTMLDocument was to be removed, it looks odd to have an interface called IHTMLDocument2 hanging around – surely it should be renamed IHTMLDocument, however this enters a compatibility minefield.

Breaking Change

Most of the time, I consider breaking changes to be a last resort. Breaking changes are always going to be frustrating for your customers, and if you keep forcing them to rewrite their code you’ll soon find that they have ceased to be your customer. However, as I have alluded to earlier, there can be good reasons to make breaking changes.

Maintaining deprecated parts of an API can become an expensive overhead, bloating out what might otherwise be a concise and neat API. Periodically removing these deprecated APIs can be worthwhile, provided that your customers have had adequate time and warning to move onto the new APIs.

Alternatively it might be very difficult to make an API backwardly compatible, or an API may be just too wrong to be kept. In both cases it is vital to explore all the possibilities suggested earlier to see if there is any way to avoid making the breaking change. If it is necessary to make the breaking change, consider how you can lessen the impact for your customers.

Always Consider Your Customer

Regardless of the reasons behind the breaking changes, it is vital that you focus on making your customer’s experience as pain free as possible. After all they want to focus their efforts on adding their business value on top of your framework, not bug fixing issues introduced due to changes to your framework.

• Clearly document which parts of the API are deprecated and provide clear information about what should be used instead. Also give your customers ample warning time about when a deprecated API will be removed.
• Provide a tool to identify code that will be impacted by the breaking change, and provide hints to what changes need to be made to it, or even automatically correct the code. In some cases a good IDE can help identify and correct breaking changes, however you shouldn’t assume that it will without testing it first. If a tool or IDE is infeasible, then a clear set of instructions for what to look for might be adequate.
• Flag which areas of the API are still young and may be subject to change. At Caplin we document newer parts of the API as beta. Early adopters are able to take advantage of the latest features, however they are warned that the API may still evolve. Although this can still be frustrating for our customers this is partially mitigated by the fact that we are listening to their feedback about the beta API and are feeding that back into the next version of the API, making it something even easier to use.
• Try to hold off making breaking changes until a major version release. It is incredibly frustrating if breaking changes are made in minor incremental releases.
• Provide consultancy to your customers to help with the upgrade to the version with the breaking changes.

Conclusion

There is no reason why you can’t build a framework whilst employing an agile methodology. Most of the principles of agile are equally valid for both framework and application developers. The main issue is that your ability to refactor the framework code is restricted when it comes to the public API. As a result, the importance of getting an API “right” the first time may seem vital when developing a framework compared to an application, however failure to create this “right” API is not a disaster. As shown earlier, there are several techniques that can be applied to build up a framework in an iterative and agile way.

At times it is very difficult to maintain backwards compatibility, and sometimes breaking changes are impossible to avoid, but the role of a framework provider should be focused on empowering the users of the framework, its customers, to build their software reliably and efficiently upon it, rather than spending all their time bug fixing issues introduced by new versions of the framework. As a result the processes and practices for agile framework development need to aligned with these goals – however that is a topic for another day.

In two weeks time I will be following this up with a look at one of the processes that we have put into place to help develop our frameworks, “Testing APIs for Backwards Compatibility”.

1. It’s very difficult to use velocity for Scrum sign-up because team personnel change frequently
2. Unexpected work can impact the project teams resulting in a demotivated team because they sometimes do not complete what they have signed-up for

There must be a better way?

I have been thinking about this for some time and our vision is continuous releases (apologies to the Marketing Department) whereby development teams pull work from priority boards, with the work moving through the necessary stages, culminating in the final stage being “release into production”.  A blissful continuous work stream where capacity and demand are in harmony and value is pulled through the system and delivered to the business as early as possible. Idealistic?  Maybe, but there seems to be some momentum gathering in the world of “Kanban Systems”, which appears to deliver our vision, so here at Caplin we are trialling this new way of working with one project team.

If you want to find out more about Kanban Systems and Lean techiques these four websites are a good start:

• http://availagility.co.uk
• http://www.limitedwipsociety.org
• http://www.crisp.se
• http://www.agilemanagement.net

We will also have some attendees at the upcoming 1-day Lean and Kanban Exchange on the 1st December, run by the always excellent Skills Matter.