Bad requirements are often cited as the biggest reason why software projects fail. Badly understood or missed requirements drive business executives to despair. The business and development all blame each other for why things went wrong, and ultimately the end users don’t get the system they want. Over the last 20 years, the industry has tried many different ways of solving the requirements problem. Improvements range from formal methods and model-driven approaches to team organization and customer engagement.
For many Agile projects, formality is discarded and replaced with strong customer interaction and the delivery of regular working software that can be reviewed with the end user. The system specifications are replaced with stories and epics that encourage collaboration with the customer, focusing on observable acceptance criteria, instead of describing in detail what the system does. So has Agile solved the requirements problem? I would argue in part Agile has provided a great way for teams to engage with customers, but for many complex, large systems requirements, management is still needed. In fact, for large projects or products, by encouraging no specification and formality,
Agile methods have often undermined project success. I believe that for complex software projects, organizations still need to invest in requirements techniques and associated tools, but they need to understand the difference between requirements management and definition. Definition/management is an area that gets very confusing as we talk about requirements in project management and software delivery terms. A requirement is both a plan item and a specification, and unfortunately tool vendors have only amplified this problem. The Agile community has all but given up on the definition side of requirements, focusing instead on the project management or work planning perspective. Unfortunately, the truth is that requirements are both, and we need to effectively treat them in both ways.
At Tasktop, we work with this [inherent duality] of requirements daily when we are asked to create a real-time connection between the most traditional of requirements management tools to the most cutting edge of Agile and developer-centric tools. For the purpose of illustration, I will describe how Tasktop builds our products and how requirements definition has risen as a formal discipline in support of our requirements management approach.
Tasktop has a simple mission: to connect the world of software delivery. That mission means that we build tools that help organizations connect their software delivery lifecycle. One of our products, called Tasktop Sync, is an integration hub for software delivery tools. Over the last four years, Tasktop Sync has evolved from a point-to-point integration to an infrastructure tool that integrates numerous software development tools and supports multiple artifacts and platforms. Tasktop engineering has always used Agile methods with stories being allocated to sprints, releases being X number of sprints and story points and velocity being used to plan. But as the complexity of Tasktop Sync grew and as we engaged with more partners, we found that engineers were increasingly asking “what does Sync do?”, not at the macro level, but at the detail level. For example, how does Sync transform a particular field type or handle a particular error condition?
This problem was made even more intense as we built more and more connectors. Those connectors implemented similar functionality for Sync but with different product end points. We needed to describe a specification for integration. This specification describes how we expect a connector / integration to work. The implementation for each connector would be different, but the specification would be the same. Stories were a fantastic way of describing the item to be planned but gave the development and test teams very little in the way of details to ensure that each connector would implement this feature in a consistent way. Thus, we uncovered the double-sided nature of requirements. Requirements are both a specification (and asset) and a task (or work item).
But as we explored this problem more, we found that the tools we were using to capture stories did not provide us with a great way of describing the specification; even worse, once we described the specification in the tool, we found we had no way to find it again – after all, a story is used to drive development, and once completed it’s banished into history, being only used for historic velocity information. We had to describe our specifications in a tool that allows version management and supports being able to find the information, and because we are an engineering company that is very focused on productivity and automation, a specification that can automatically build acceptance tests. A story is linked to this specification in the same way that code, tests and ultimately test results are linked to this artifact, but it is not the same artifact. In fact, we found that the stories described the creation or change to that specification and held meta-data associated with the work, not the details of the capability.
As a result, we found that we could capture historic velocity not only about the teams, but also the feature we were supporting. So what does all mean? In short, we need to think about the practice of requirements management as different from the practice of requirements. The two disciplines are linked but are different. Tool vendors who historically have merged these two activities need to stop and evaluate their products and try to separate the two approaches. That would allow the industry to have tools that allow the management of work and creation and maintenance of assets to be separate. What do you think about this interesting double-sided behavior? How have you dealt with it when deploying Agile at scale?
