The launch of my upcoming book, Project to Product (pre-order here), will be on October 20th, and I’ll be previewing it at one of my favourite events of the year, DevOps Enterprise Summit, Las Vegas (October 22-24). In the lead up to the launch, I will post a series of blogs related to the genesis and ideas of book – including the Flow Framework™, a new managerial paradigm for surviving and thriving in the Age of Software.
For me, Project to Product has been a 20 year journey that started with a decade of learnings as an open source developer, and applying those learnings to my last ten years working with IT leaders. These posts represent some of the most interesting learnings and collaborations along the way. The articles were originally published in my “On DevOps” column in IEEE Software magazine, and targeted at an audience interested in the evolution of software architecture.
To get these blogs delivered directly to your inbox, please sign up for the Project to Product Newsletter.
For over a decade, my colleagues and I have focused our research efforts into applying architectural principles to artifacts that fall outside the code base. Modern software production involves countless non-code artifacts scattered across processes and tools. The bigger the applications and code are, and the more applications there are, the more likely this fragmentation will slow delivery to a crawl. But what if we could come up with a unified way to architect not just the code but also the entire flow of software delivery?
What’s Different about DevOps?
The DevOps movement has catalyzed a need to connect all of the disparate tools and stakeholders involved in delivering large-scale enterprise software. As software and startups disrupt market after market, the world’s largest organizations are being forced to transform. For example, BMW plans to have more than 50 percent of its staff be software engineers in the next five years. Bank of America’s Global Technology & Operations arm has 100,000 employees and contractors. These numbers are mind-boggling, yet little has been published on proven practices for managing software delivery at this scale.
Consider that The Phoenix Project has continually been one of the top ranking books (in Amazon’s Computers & Technology area). The novel relates the journey of applying lean-manufacturing principles to turn around a large-scale IT delivery failure that’s representative of the current state of many Fortune 500 organizations. The breakthrough popularity of the book is an indication that we need to start paying much closer attention to the concepts underpinning the DevOps movement. One thing that’s overwhelmingly clear is that the connection between the business processes of software delivery and the software architecture is far less mature than the end-to-end flows that are so carefully optimized in modern manufacturing.
Programming languages, toolkits, and frameworks are no longer the bottleneck in developing and deploying software. Developers can choose almost any flavour of type safety in their language of choice, and almost every imaginable design pattern has been implemented in ubiquitously available open source frameworks. Yet this elegance breaks down once the software scales, the number of customer requests increases, and the complexity of deployment and release management grows. The popularity of the DevOps movement is telling us that we need to look at formalizing the artifacts and flows outside the software architecture.
A Little History
At its inception 10 years ago, the DevOps movement was catalyzed by the notion of turning deployment and provisioning processes and infrastructure itself into code, thereby making deployment an extension of the software architecture. With this, an industry movement began, and the principles of automation and architecture started reaching beyond what we thought of as “the code.”
On the academic side, research on formalizing abstractions beyond the code was already well underway. Aspect-oriented programming enabled the capturing of crosscutting concerns, while model-driven software engineering and software product lines further formalized the production process of building software.
What inspired me most was research at the University of British Columbia Software Practices Lab that extended the notion of software structure to collaborative artifacts within tools such as issue trackers. I focused on extending abstract-syntax-tree models, which I became familiar with from working on compilers, to capture additional delivery concerns. These concerns ranged from requirements, to agile user stories, to tests, code reviews, and builds. While working on my PhD with my supervisor Gail Murphy, I became obsessed with trying to create one structure model for all artifacts and interactions involved with software delivery.
Toward a Value Stream Architecture
Fast forward to today, where the market conditions have materialized to create a new software discipline centered not on the software architecture but on the architecture of the end-to-end value stream. The goal of Project to Product is to elevate the architecture discussion to include all artifacts involved in the software delivery value stream. Inspired by lean manufacturing, I use “value stream” to mean the end-to-end feedback loop of flowing software to customers in a way that maximizes the business value delivered.
The rise, popularity, and amount of industry investment in DevOps are the perfect catalysts to establish a new discipline that will add the rigour we already have with software architecture to the software delivery process. I call this discipline value stream architecture and propose that it’s the key to delivering on the DevOps vision at scale. The DevOps Handbook describes the three ways of DevOps: flow, feedback, and a culture of continual experimentation and learning. To achieve these goals, we need a comprehensive view of architecture that will let us rigorously apply lean-manufacturing principles to software delivery.
In applying systems thinking, every IT leader must be able to answer the question, “Where’s the bottleneck in my software value stream?” In software, we’ve become expert at this, applying tools such as profilers to identify where we have O(n3) complexity, and quickly resolving those performance bottlenecks. Conversely, almost none of the IT leaders I’ve asked could answer the bottleneck question. Neither could they state which of the following investments would increase velocity the most: automating deployment, hiring more developers, hiring more designers, or investing in training and tooling. Consider that some of these leaders are making $10M to $100M IT budget decisions without a principled way to identify their key bottlenecks, which would make any lean-manufacturing expert versed in the theory of constraints cringe.
Project to Product aims to start the journey to providing practitioners a framework that can answer those questions, and to use their experiences and insights to piece together the discipline of value stream architecture. We’ll look at how to use this new kind of architecture to formalize feedback loops and to provide the visibility needed for the data-driven experimentation and learning that will let us succeed with DevOps at scale.
On this journey, we’ll review the experiences of thought leaders and practitioners who are elevating the state of the practice beyond accepted architectural principles simply because they’re forced to deal with software delivery at organizational sizes at which established practices fail. This will include stories from financial-services companies pushing the limits of scale in terms of IT staff and from car manufacturers that are putting processes in place to manage the 200 million lines of code running in each of their new vehicles. We’ll review the findings of leading researchers focused on the value stream, as well as the lessons learned from cutting-edge technologists advancing the state of the art through their startups.
I hope the journey we’re embarking on provides you with inspiration to drive the research and innovation that the software industry needs to move beyond ad hoc heroics toward a unified view of the software delivery value stream. As IT organizations continue to scale at an incredible pace, architecture is more important than ever, and some exciting work lies ahead!
To learn more, click on the book cover below to pre-order a copy of Project To Product:
And don’t forget to sign up for the Project to Product newsletter to receive articles directly into your inbox that cover some of the core concepts that lead to the creation of the Flow Framework, as well as highlighting conference presentations, book signings and other events I will be involved in.
A version of this article was originally published in the September/October 2017 issue of IEEE Software: M. Kersten, “Value Stream Architecture,” IEEE Software, vol. 34, no. 5, pp. 10-12, 2017. © 2017 IEEE doi: 10.1109/MS.2017.3571573 – Original article
