Most of us use metrics or models to gauge progress and make business decisions. A fallacy of the practice is to treat our metrics like the truth and our models like crystal balls. Don’t fall into that trap. Remember that our numbers are abstractions of reality, not reality itself.
It was the famous statistician and co-inventor of the Box-Cox power transform, Sir David Cox, who said, “All models are wrong; some models are useful.” One of my friends and colleagues taught me that quote and I have forever been indebted to him for providing me with the important message it intones.
Whether we are performing process improvement, tracking business performance, or making business decisions, we all use models and metrics to aid in our decision-making. However, when we throw our common sense and intuition out the window, and let our numbers and models start making decisions for us, we make a terrible mistake.
Statistics has become the cherished mathematical science of our models and metrics. It is a powerful way of looking at our numbers and establishing our metrics. With statistics we can think in terms of distributions and probabilities, instead of just more or less or deceptive averages. Yet, as we have all been warned, if we have not learned first hand, statistics are a great tool for creating almost any portrayal we desire.
No doubt, there is great power in using statistics with metrics and models to monitor progress and make decisions. But, as Stan Lee made so famous with the Spiderman comic series, “With great power comes great responsibility.” With that power, comes a great temptation, and great danger.
The temptation is to let our models do the thinking for us or to trust statistical analyses more than intuitive insights. The danger is that those models and statistics are themselves prone to error, and do not take into account everything that a human mind can help to evaluate or judge.
Without going into statistical lessons or math, let’s consider the power and danger of statistics. Considering the changes in distributions and the probabilities of outcomes is extremely helpful. However, there are a couple of things in particular that can get us into trouble with our statistical uses.
The dangers of statistics can manifest in poorly trained or inexperienced analysts who don’t always recall or understand how to truly troubleshoot their analyses, but most often the problems come from a communication breakdown. Many times the individual tediously gathering and analyzing data is not the manager or leader who uses the analysis to make a decision.
Often, the full merit or risk of an analysis is not communicated to, or understood by the manager requesting the statistic. Executives are especially prone to requiring no more than a 30-second explanation of the analysis, leaving little room for important details. Unfortunately, details such as power and sample-size, sub-grouping assumptions, or the coefficient of determination can say a great deal about how much the analysis can be trusted or manipulated.
I’ve seen cases where changing the subgroup categories by a decimal place completely changed the obvious conclusions of the analysis. When it comes to building statistical models, most of us wouldn’t use a model of a production process or for process control that didn’t explain at least 90 percent of the variability within the data. However, when dealing with human behavior, it’s rare that a model will explain better than 60 percent of the data, and I’ve seen marketing and sales plans that based their strategies on models only 30 percent accurate because that was the best model that could be devised.
In fact, the recent demise of the loan-investment industry that triggered the 2008 recession is, not completely, but to a significant degree, to be blamed on loans issued using rather risky models that users believed could predict the general public’s investment behavior. There are a couple of fundamental truths about models, statistical or otherwise, that we tend to easily forget.
First, we must recall that our models are based on historical data. Our data is almost never complete, but a sampling. We can sample discriminately to reduce risk and error, but we cannot eliminate it unless we accurately collect every possible data point. Also, it is historical, meaning that our model tries to explain what has happened before. Any time we use a model to predict the future we are knowingly leaving the inference space of the model.
Just because the causes of outcomes in the past can be correlated, doesn’t mean that the performance of the future will be certain. If the process modeled is stable and nothing in the environment changes, then the outcomes we predict will probably come to pass. But if anything changes, our model is no longer valid.
That is why modeling human behavior, especially human investment behavior, is a very risky proposition. People change their minds and their behavior on a whim.
Metrics are equally dangerous. First, whether we mean them to or not, metrics drive behavior. Even if we offer no reward or incentive to move a metric, knowledge about what is better performance, sighs of relief, or smiles of pride will be enough to cause people to endeavor to improve the metric. Unfortunately, improving the metric, while it probably means that overall performance is improving, does not necessarily mean that overall performance is working the way we intended.
If we have metrics around quality, our throughput might drop as personnel take more time or put forth more effort to improve quality. This is why we utilize conflicting metrics. We measure quality and throughput at the same time so that one will not be sacrificed. It’s a good idea, but not foolproof. There are always means of hiding errors or satisfying metrics when the process is not as good as we want it to be.
I know of no cure for metrics-driven behavior. It’s unavoidable. What we must remember is that metrics are only indicators; they are not the truth. To find the truth we must go forth and observe it for ourselves. If we witness behavior driven by the metrics, we must point it out and put a stop to it. Our purpose should always be to drive proper behavior and practice. It should not be a quest to achieve some arbitrary, desirable number on a digital dashboard.
By all means, continue to use conflicting metrics to balance the behaviors striving for the right, desired outcomes. Use lagging and leading metrics the same way. Lagging metrics are typically the final output and are collected after the results we were trying to control have taken place. Leading metrics are measures of in-process performance that help us predict what the final outcome might be.
Behaviors on leading metrics are especially dangerous, because sometimes our desire to improve them, lead to behaviors or decisions that ultimately hurt the final outcome or the business. A basic example is the throughput of a single operation in a production line. We might use that throughput as a predictor of the final output of the production process. However, if striving to improve the throughput of a single operation causes that operation to expedite material, steal material from other operations, or sacrifice an important element for another downstream operation, then the final outcome on piece-part cost or quality, or even the final production throughput can be harmed.
Don’t misunderstand me. I’m not saying that we should stop making models, tracking metrics, or using statistics to drive decisions. I’m a big advocate of doing all three. In fact, every business should absolutely use models, metrics, and statistics in my opinion. I’m trying to warn us all against the temptation of relying too much on our numbers and not enough on our observations and common sense.
Every model has error and since no model contains data from the future, using any model to predict the future is a gamble. Metrics are indicators, not the truth, and the longer a metric has been in place the more likely it has been behaviorally manipulated. Statistics can fool us as easily as they can reveal profound insight if we don’t take the time to understand them carefully.
Use numbers and models to run your business or your team more wisely, but don’t throw wisdom out the window by letting your numbers make decisions for you. Continuously investigate your numbers, observe behaviors, and question how analyses were conducted. Our human investigation and intervention is the best protection against faerie tale fallacy.
Stay wise, friends.
If you like what you just read, see more of Alan’s thoughts at www.bizwizwithin.com.
