It’s not that they’re measuring the wrong variables, it’s most likely that those organizations have already made the decisions based on variables they already measure. In the “Function Points” example, I would bet there were a few obvious learnings early on that spread throughout the organizations, and once the culture had changed any further effort didn’t help at all.
Another example: I took statistics on how my friends played games that involved bidding, such as Liar’s Poker. I found that they typically would bid too much. Therefore a measurement of how many times someone had the winning bid was a high predictor of how they would perform in the game-people who bid high would typically lose.
Once I shared this information, behavior changed and people started using a much more rational bidding scheme. And the old measurement of “how often someone bid high” was no longer very predictive. It simply meant that they’d had more opportunities where bidding high made sense. Other variables such as “the player to your left” started becoming much more predictive.
It’s not that they’re measuring the wrong variables, it’s most likely that those organizations have already made the decisions based on variables they already measure. In the “Function Points” example, I would bet there were a few obvious learnings early on that spread throughout the organizations, and once the culture had changed any further effort didn’t help at all.
Another example: I took statistics on how my friends played games that involved bidding, such as Liar’s Poker. I found that they typically would bid too much. Therefore a measurement of how many times someone had the winning bid was a high predictor of how they would perform in the game-people who bid high would typically lose.
Once I shared this information, behavior changed and people started using a much more rational bidding scheme. And the old measurement of “how often someone bid high” was no longer very predictive. It simply meant that they’d had more opportunities where bidding high made sense. Other variables such as “the player to your left” started becoming much more predictive.