Organizations that can tap into the power of user feedback have seen enormous improvements in ROI over the past decades. It started with Agile via iterative deliveries of user value on a set cadence, usually every two to three weeks. As top-performers (those that understood and embraced Agile fundamentals) began to separate from their competitors, they sought more gains. Can we make our iterations shorter? How short? Enter DevOps. By bringing the development and operations teams together, elite performers again separated themselves from the competition by shortening the feedback loop by another order of magnitude. This time, instead of collecting feedback every two weeks, they began collecting feedback multiple times a day. The results of these gains in the past decade are obvious: software is more reliable, more functional, and more adaptable than ever before. This is a direct result of feedback loops. Yet it’s a whole different story in the federal space.
Try logging into your typical government-built system and you can immediately tell it was built differently. The UI is awful and the system crashes under heavy use. There are redundant fields, confusing lexicon – the type of stuff that works great in a developer-led demo, but terribly for the rest of us. This isn’t because the developers that built those systems are poor developers, they’re just using a methodology that was abandoned as a matter of survival decades ago. Those that have survived don’t code from documentation. Even your best developer at Google isn’t going to turn 200 pages of requirements and specs into something useful. Most won’t even try. These creators require feedback.
Despite this, federal agencies have an overwhelming tendency to disregard user feedback in their software development. They think the answer is to just “do DevOps” or “be Agile” while failing to recognize that holding daily scrum meetings—while continuing to work in exactly the same way—is not likely to produce any measurable gains. Assigning SAFe titles, renaming or rewriting requirements as user stories, measuring story points instead of lines of code or hours… it’s all meaningless without adopting the underlying principles that led to these new frameworks and techniques. To do this is not easy. The most meaningful change never is. And there are many reasons why your organization is resisting.
Here’s the top three we’ve heard:
Excuse #1: There Are No Users (Yet)
Sounds strange but this is more common in defense development than some would think. Sometimes we are building a capability to respond to an anticipated future threat. Other times, we’re incorporating new technology (e.g. advanced sensors) targeted for a user cohort that won’t exist for another year or two. Even more common, we’re building data processing capabilities for a space or airborne system that won’t be deployed until a future date.
Note that in each of these common scenarios our solution is to “code blind”. We develop to the spec or requirement, leaving it up to the contractor or program office to decide when the need has been met. The move towards Agile has led some organizations to implement Sprint demos. These typically materialize as developer-led demonstrations attended by the Scrum team with occasional attendance by program office stakeholders. While these demonstrations are a valuable form of feedback, they fall far short of the feedback that would be derived from having real users hammer on the code that has been delivered to production. It’s exceedingly easy to spot systems that have been developed this way. They typically require expert assistance, lengthy training materials and have steep learning curves. Users gradually become proficient with the intricacies of the system over a period of months or years. When they eventually move on, the operational organization will experience significant strain.
Commanders understand this and often seek to protect their most adept operators by keeping them in their role as long as possible. This works well for the mission but is often at the expense of a service member’s opportunity for learning new skills and advancing their career. Statistically, each of these operators can be expected to have upwards of 40 applications hosted on their personal mobile devices which they can intuitively use on-demand without training or expert assistance. Some are starting to wake up to the possibility that it may be possible to code DoD applications to be just as intuitive. Yet many more still disagree. Which leads to explanation #2…
Excuse #2: We’re Not Selling Books or Streaming Videos!
Within every organization that we’ve worked, our team will typically attempt to draw parallels between federal development methods and the methods of companies like Amazon or Netflix. We believe that there is much to be learned from these types of elite performers and that some if not all of their practices could be adopted at a huge benefit to the federal government. The most common response we hear to this assertion is “that won’t work because what we’re doing is more complex.” I think we need to be suspicious of this assertion.
There are clearly esoteric technical aspects to many aspects of defense technology. But are there not highly sophisticated technicalities required to stream video to every corner of the globe, servicing 200M users on demand? Maybe that sounds easier than detecting a missile launch from space, and it may well be. But is it that much more complex to necessitate sticking with legacy software development methodologies? Is Netflix recklessly risking their business with continuous updates that regularly top 100,000 daily? The shareholders certainly don’t think so, as they’ve led the way demonstrating that change and stability can (and most always, will) move together in software development.
Excuse #3: We Can’t Bother the Users With Unproven Software Because They Already Are Overtasked
Again, this may be true, but how well do we understand why? Would quality software help alleviate their burdens? Would listening to their pain points and needs allow developers to provide targeted solutions? It quickly becomes a circular argument: operators are overtasked because they are using poor software and therefore can’t take the time to participate in the process of developing better software. So we build them what we think they need and force them to use it. Then we start again.
If our operators are truly unavailable for this role, we may need to get creative. We might apply workarounds to achieve the desired result. We can hire former operators as contractors, set aside time for specific feedback sessions with off duty operators, or some combination of both. All of this is an easy sell provided leadership understands and believes in the benefits of feedback.
Time To Change
Without visionary leaders that understand the underlying concepts of the software age, who are also willing to exert the massive force required to turn this ship, we will continue to drift along in the same direction we’ve been headed for the past 30 years. What worked in the 1990s is unlikely to be an effective approach if we desire to maintain the security and safety we’ve come to take for granted in decades past. The change, particularly within a defense industry that has come to expect profit without risk, is going to be resisted massively on many fronts, as it has to date. Despite all the tough talk from government officials including many who have taken to social media to tout their advances, there is to this day, not a single major defense contractor that has meaningfully changed how they approach software development on a federal contract. Every single one of them has put off fundamental change – and for good reason. Maintaining the status quo is simply too profitable. Until this changes, I expect we will keep drifting.
Why Feedback is as Important as Development
In this e-book, I explain how those working in or around the U.S government can begin leveraging rapid feedback loops to become a high-performing continuous learning organization:
- The critical role of user feedback in product development
- The dangers of relying only on specifications and documentation
- The nuances of product development
- The shrinking timeline to adapt
- Real-life stories from the field
- Common reasons for overlooking feedback debunked
- Steps to replicate conditions that capture feedback
About the Author
Scott Worden is an engineer and former acquisition officer for the USAF. He co-founded Centil in 2017 with the aim of improving the government’s ROI in defense industry development.
