This blog was originally posted on the IT Revolution blog on December 18, 2018.
Project to Product tells a story of what we can learn about building software by walking down the BMW Group Leipzig plant production line. The other wonder of mass production that makes repeated appearances throughout the book is the Boeing 787 Dreamliner, which has long been an inspiration to me and whose silhouette graces the cover of Project to Product.
Not only is it a marvel of engineering and beauty, it is a marvel of supply chain management and a mind-bogglingly complex integration between business strategy, design, and production. In fact, the entire concept of Value Stream Networks in the Flow Framework™ introduced in Project to Product was inspired by the fact that the flow of value across software delivery teams is more similar to airline networks than car assembly lines.
With this as background, I was delighted to see my Twitter feed explode with former avionics engineers and many others chiming in on a passage from Project to Product that was posted by Wil Pannell and tells a sliver of the software journey of the 777 aircraft.
I heard this story in 1997 in the first Software Engineering course that I took, taught by Gail Murphy at the University of British Columbia (UBC). Gail and that course inspired me to pursue a career in software engineering. Six years after she told that story, I went back to UBC to learn more from Gail by working on my PhD with her, which culminated with our founding Tasktop as a commercial vehicle to better understand software delivery at scale. It is interesting to note that the lessons from that story are even more relevant today than they were when I first heard it two decades ago.
I have told this story more than once to colleagues and to my staff. For example, shortly after doing a major product release a few years ago, an unforeseen synchronization scenario was at risk of bringing a customer’s massive Jira instance to its knees, which would have frozen their ability to ship new releases. That day I told this story to one of our product development teams, explaining to them how important it was to both literally and figuratively get in the plane. The team had never been on site with a customer, and I did not tell them to go, but shortly after hearing the story they were across the country, sitting in the same room as the customer’s IT specialists. Here is the passage from Chapter 2 of Project to Product:
How is it that the BMW Group and Boeing can both manage existing production lines, transform their business to support new ones, and continually adapt to changes in technology, competition, and the market? The bottom line is that they are not stuck in the gridlocked puzzle pieces of project management. Instead, they have mastered a product-centric view of delivering value to their market.
My first exposure to this was a story told to me by Gail Murphy, then my professor in a third-year software engineering course, about the production of the Boeing 787’s predecessor, the 777. The 777 was Boeing’s first “fly-by-wire” plane. In other words, the software had to work, as it was purely software that was controlling the flaps and rudder and preventing the plane from falling out of the sky. Gail recounted that, due to the criticality of the software, Boeing decided to put all the heads of software engineering on the test flight. During the test flight, the plane started shaking, and the software engineers were able to implement a midflight fix via the turbulence control software.13 I have yet to find a better example of an organization putting software leaders’ skin in the game of high-stakes product development.
The goal of telling that story was to relate a tangible image of how critical the work of software leaders is, as well as the right mindset for the people building the software. While we may feel more removed from the safety of our users than a mechanical engineer working on landing gear, software is becoming increasingly critical to countless systems responsible for both our physical and digital safety. In 2018 alone, we saw software practitioners jailed (VW ‘Dieselgate’), blamed for massive security breaches (Equifax), and even for bringing down airline operations (BA and its IT system crashing five times in one year).
Gene Kim and I pondered each of these events over a long conversion. We realized that there is something wrong with the mindset that business leaders take with software delivery and IT with regards to blame and safety. I simply cannot imagine the head of a car company blaming a week-long outage on a factory floor worker tripping over an extension cord. Yet that was exactly the tone of the executives overseeing the events linked above. Gene went as far as writing an imaginary letter from an Airline CIO who understood the criticality of software. That 777 story epitomized for me a different way of thinking. One that as both a software developer and a business leader I have taken to heart.
The Twitter thread unearthed different accounts of this story, which I am still trying to get to the bottom of. I am continuing to work with my contacts and with some of the people on the thread to get more of the details on the original story. Even more interestingly, the thread has uncovered different instances of this story happening over the years:
@PAEONIA111: Roman bridge builders had to sleep under bridges they built.
@JimCownie: Sounds a bit like Napoleon’s solution to explosions at gunpowder factories: he just passed a law requiring that the owner and his family live on site.
@DJSnM: Neil Armstrong was one of the engineers integrating the MH-96 fly by wire system into the X-15, he flew the first 4 flights with it.
@wombatnation: A research lab I used to work at developed sonar systems. The engineering lead had to go on the sub for the maiden voyage for each system. Lot of pressure that your ice thickness sonar is going to work.
@paultag: Similarly, the US Army has riggers jump with a random (1 out of every 10) Parachute they packed to make sure no one was half-assing it
@mkorhonen12345: "Alle Leiter der Fluggesellschaften müssen am 1 Januar 2000 in der Luft sein", sagte Zhao Bo, der im chinesischen Ministerium für Informationsindustrie für die Lösung des Problems zuständig ist.
@Satcom_Guru: I can say that we semi-routinely reprogrammed the Thrust Management System inflight (757/767).
terminal onboard w/printer (no screen).
We worked in machine code; easy to change variables.
Very carefully!!!
Restored published config before landing.
1/2
I’ll ask around about 777 flight control story.
757 NA002, Bob Galliart dodged sailboats along Puget Sound at 500’, while I struggled loading finicky TMC for 10 minutes, to setup for a climb condition.
Every 30 seconds the anxious test director would call, ”done yet?”
2/2
The back-and-forth on the thread highlights that there are both good and bad conclusions to be drawn from these stories. For example, you can interpret it the way Napoleon might have and instill a culture of fear in your technology leadership ranks, focusing on blame. Or you can take the more enlightened approach and use this as a tool to elicit the feeling of everyone being in the same boat; the practitioners, leadership, and, most importantly, the customers, all focused on a productive and safe outcome. In order to help us gain a perspective on this, Gene Kim has been injecting thought leaders focused on safety culture in other domains into the DevOps Enterprise Summit program, which I encourage watching because our collective pace of learning on this front needs to increase.
A key point of Project to Product, and the underpinning of the Flow Framework™, is that business and technology leaders need to be in the same boat, sitting together at the oars and rowing the customer to the destination. That boat cannot be a Viking ship with leaders yelling commands and blame, but a boat where everyone take the perspective of the customer’s benefit and safety along the entire journey. Due to the complexity and scale of software delivery, perhaps a better analogy is that we need to think of ourselves as sitting on the same plane. The route to delivering customer value is not a straight line; it passes through a complex network similar to an airline network, where the routes are defined by creative work and the airports by teams. And all along that journey, we need to work with customer value and safety in mind.
For more on Project to Product, and a recording of my talk summarizing these topics, see ProjecttoProduct.org.
