A review of Itanium server sales forecasts made over seven years reveals the same bias year after year after year.
When the economist Kenneth Arrow was working as an air force weather forecaster during the Second World War, he and his colleagues found that their long-range weather predictions were no better than random. They informed the boss but were told, “The commanding general is well aware that the forecasts are no good. However, he needs them for planning purposes.”
David Orrell, The Science of Prediction, 2006
The first step in R&D portfolio management is to generate the list of projects that are under consideration and value them. Unfortunately, this is where many companies spend 99% of their effort, and too often this leads to countless wasted hours and benign neglect of the portfolio as a whole.
Let’s start out with a basic fact: Every project’s net present value (NPV) in every portfolio of every company in the world is wrong. Absolutely, dead wrong. The forecasts, be they cost or revenue, will not hit their projections. They may be higher or lower by one dollar or a billion dollars, but they will not match their projections. The forecasters be lucky if the sales forecasts used to generate the NPVs are within +/- 50 % of what will actually happen, yet they’re spending an inordinate amount of time on non-value-adding activities such as:
- Assessing forecasts to the second decimal place
- Explaining why a project’s NPV changed by 2%
- Trying to approximate the uncertainty in a forecast by picking the “right” point estimate instead of simply incorporating uncertainty into the forecast
Each year, the Energy Information Administration forecasts oil prices for the coming years. Should they bother?
These are all accepted practices in nearly every portfolio process we’ve seen. The irony is that the portfolio process itself has a glaring portfolio problem: the analysts we observe are spending FAR more time in the minutiae of project valuations than they are on activities that lead to actual decisions about the projects. These wasteful project-level tasks are being undertaken at the expense of value-adding portfolio activities. None of these activities lead to changes in decisions, and only serve to prevent decisions from being made.
Forecasts that are wrong, like the crude oil price and Itanium server examples shown in this post, still have an important purpose. The forecasting process helps us think through the factors that drive a forecast, and separate those within our control from those that are not. Finally, they lay bare our biases if we are disciplined enough to view the forecasts retrospectively, after events have transpired. Within portfolio management, the critical point is to acknowledge the limitations of a forecast, and use that to inform the amount of resources that are devoted to the forecasting exercise.
We will be delving into the causes–and remedies–of these issues in upcoming posts. In the meantime, do you have an interesting angle or experience you’d like to share?
You can find case studies on R&D portfolio management applications of the Enrich Analytics Platform on our website, or contact us.
See also:
Don’t use discount rates to account for risk
Building the “Good Enough” business model
Tornado diagrams 101
