At LNS Research we believe in the expression “you can’t control what you don’t measure.” With that, we’ve focused a significant part of our research on identifying the “Metrics That Matter.” And one of the topics of we find of most interest to our Global Executive Council is benchmarking and understanding how to measure performance.
The Danger of Metrics for Metrics’ Sake
Very early in my career I was the lead process control engineer at a brand new state-of-the-art sawmill that was just going through its start-up. This particular mill had a brand new downtime measurement system that would detect when a machine center was down and prompt the operator to enter a downtime cause code. To ensure that the system could capture control system failures it was decided to install redundant sensors to detect machine center stoppages.
One of the machine centers in a sawmill is a debarking station. In the debarking station, tree-length logs from the forest are run through a machine that scrapes the bark from the tree. On one instance a tree had an undetected soft spot that caught one of the rotating scrapers. It ended up thrashing around, knocking all of the photocells that controlled and synchronized the machine out of alignment, as well as the downtime system photocells.
Amazingly, the mill manager insisted we fix the downtime system cells first so we knew how long the machine center was down instead of fixing the system so we could resume production. The metric had become king and we lost valuable production time for the sake of a metric that was going to be inaccurate, since there was no way to capture the real reason we were down for part of the time—fixing the metric system!
The lesson here is not to let the measurements of the system degrade system performance just so you can gauge performance, since you aren’t really measuring what the system is truly capable of delivering.
The Danger of Miscalculated Metrics
As I’ve spoken with regard to OEE, it’s a metric that was specifically designed to measure performance at a machine (or machine center), yet many organizations are trying to use it to measure the performance of an entire production line or plant. Organizations do this by taking the total productivity of the plant and crossing it with the plant’s average availability and the aggregate quality number. These measurements are so broad that while the number is calculable, it most likely gives you no insight as to what is actually occurring in the plant.
You can calculate a total plant or line OEE, but you should treat it as a network calculation much like you would calculate resistance in a complex series-parallel circuit or fluid flow in a complex hydraulic circuit. This way you can see what is happening at each element in the plant and begin to determine where problems actually begin as they may in reality be cascading and making things look worse than they really are.
The lesson here is that over-aggregation of metrics can mask or even exaggerate the initial problem in a process.
The Danger of People and Metrics
One of the greatest dangers with metrics is that people can often influence them, especially when those metrics are used to reward (or punish) those same people.
Cem Kaner is a noted author and researcher on the subject of software testing. In a report by Doug Hoffman that heavily cited Kaner’s work, particularly with the Los Altos Workshop on Software Testing, there were numerous examples where the metrics tended to match the predicted targets because people were manipulating the data.
One such metric was defect rates during testing. If the number was too high, the developers were “punished” for writing bad code, and if the number was too low the testers were “punished” for not doing a good enough job of testing. With this lose-lose situation it’s no wonder the defects found during testing exactly matched the target.
The lesson here is that if you reward or punish people based on a metric, and they have direct influence over that metric, you can expect the measurement to pretty closely match the target, regardless of what the real situation is.
A Metric Misunderstood Is Most Likely a Metric Misused
In a recent article here at LNS Research, we discussed 6 Myths of OEE, including many of its misuses. Unfortunately, most metrics suffer from the same risks as OEE. If they are misunderstood, they’re most likely misused.
Before organizations start managing by metrics and using those metrics as the primary means of managing the business, they need to understand exactly what those metrics mean, how they are calculated, and the risks associated with misusing those metrics.
Say you’re driving to work and the speed limit sign says 60 MPH—your speedometer says you’re only doing 40 MPH; it might be wise to consider that with all the snow on the road maybe things aren’t as slow as they seem.
Top 3 Ways to Misuse Metrics:
1. Have too many metrics – Most likely people will end up spending too much time measuring, taking them away from doing. Even if the metrics are all collected automatically, too many metrics will make it impossible to see the forest for the trees.
2. Choose arbitrary metrics – The metrics must match the process and actually provide meaningful insight into performance. If the metrics provide no insight as to what direction to take to improve, they are near meaningless.
3. Never change or adjust metrics – Businesses do not operate in a static world. At the very least you should adjust the target values of your metrics based on the changing environment. You should also be prepared to use different metrics if the environmental changes are significant enough. Twenty years ago when oil was cheap and climate change was virtually unknown, a process’ carbon footprint was not a meaningful metric. As we know, today that is far from the current situation.
Identifying the Metrics that Matter Today
So, which are the metrics that are really proving to move the needle for manufacturers today? In conjunction with MESA International, LNS Research conducted a five-month research study to determine just that. Broken down into eight categories of 28 specific metrics, the results of that study can be found by downloading the “2013-2014 Manufacturing Metrics that Really Matter” eBook.