This is a very important concern that I have too. I have not read the book, and it might be a very interesting read, but when it starts with:
No matter how “fuzzy” the measurement is, it’s still a measurement if it tells you more than you knew before.
It concerns me. Because business is already full of dubious metrics that actually do harm. For instance, in programming, source lines of code (SLOC) per month is one metric that is used to gauge ‘programmer productivity’, but has come under extreme and rightful skepticism.
Scientific methods are powerful when used properly, but a little knowledge can be a dangerous thing.
In the TQM world this comes under the heading “any metric used to reward people will become corrupt”. H Edwards Deming was writing about this issue in the 1960s or earlier. For this reason he advocated separating data collection used to run the business from data collection used to reward people. Too often, people decide this is “inefficient” and combine the two, with predictable results. Crime statistics in the US is one terrible example of this.
From my recollection of the book I think he would say that SLOC is not actually a terrible metric and can be quite useful. I personally use it myself on my own projects—but I have no incentive to game the system. If you start paying people for SLOC you are going to get a lot of SLOCs!
Because of the history, you need to go overboard to reassure people you will not use metrics against them. They are going to assume you will use them against them, until proven otherwise.
Not a dedicated section; this advice is scattered throughout the book. E.g. there’s a section (p. 174-176) explaining why p-value hypothesis testing doesn’t measure what the reader might think it measures (and thus Hubbard doesn’t use p-value hypothesis testing).
Is there a section on “How To Not Fool Yourself That You’re Measuring X When You’re Actually Measuring Z”?
This is a very important concern that I have too. I have not read the book, and it might be a very interesting read, but when it starts with:
It concerns me. Because business is already full of dubious metrics that actually do harm. For instance, in programming, source lines of code (SLOC) per month is one metric that is used to gauge ‘programmer productivity’, but has come under extreme and rightful skepticism.
Scientific methods are powerful when used properly, but a little knowledge can be a dangerous thing.
Yes he is all over this.
In the TQM world this comes under the heading “any metric used to reward people will become corrupt”. H Edwards Deming was writing about this issue in the 1960s or earlier. For this reason he advocated separating data collection used to run the business from data collection used to reward people. Too often, people decide this is “inefficient” and combine the two, with predictable results. Crime statistics in the US is one terrible example of this.
From my recollection of the book I think he would say that SLOC is not actually a terrible metric and can be quite useful. I personally use it myself on my own projects—but I have no incentive to game the system. If you start paying people for SLOC you are going to get a lot of SLOCs!
Because of the history, you need to go overboard to reassure people you will not use metrics against them. They are going to assume you will use them against them, until proven otherwise.
Not a dedicated section; this advice is scattered throughout the book. E.g. there’s a section (p. 174-176) explaining why p-value hypothesis testing doesn’t measure what the reader might think it measures (and thus Hubbard doesn’t use p-value hypothesis testing).