Getting back to trying to propose practical mitigation strategies for goodhart’s law, I propose a fairly simple solution: Choose a G*, evaluate performance based on it, but KEEP IT SECRET. This of course wouldn’t really work for national scale, GDP-esque kind of situations, but for corporate management situations it seems like it could work well enough. If only upper management knows what G* is, it becomes impossible to optimize for it, and everyone has to just keep working under the assumption they’re being evaluated on G.
Taking it a step further, to hedge against employees eventually figuring out G* and surreptitiously optimizing for it, you could have a bounty on guessing G* - the first employee who figures out what the mystery metric G* really is gets a prize, and as soon as it’s claimed, you switch to using G**
The hedge is absolutely necessary, elsewise, a manager will just tell subordinates what G* is in order to look impressive for managing a high-performing group.
Andrew Grove (of Intel fame) wrote a book, High Output Management, suggesting that management needs two opposing metrics to avoid this problem. For example, measure productivity and number of defects, and score people on the combined results.
BTW the large nail/little nail joke has a third part. Soviet management eventually got a clue and started measuring by the value of the nails produced… and the result was the world’s first solid-gold-nail factory.
Presumably finding arbitrarily many basic G*‘s will be hard. Two ideas for dealing with this: 1. Even if you only have finitely many and they’re all known, you could select one at random each time there’s a switch. 2. Each time there’s a switch, select a somehow-random linear (or some other sort, if you like) combination of your basic G*’s. (That would make guessing it in the first place quite hard, actually...)
if management are doing that then are neglecting a powerful tool in their tool-kit, because announcing a G will surely cause G to fall, and experience says that to begin with a well-chosen G and G remain correlated (because many of the things to do to reduce G also reduce G). It is only over time that G* and G detach.
Getting back to trying to propose practical mitigation strategies for goodhart’s law, I propose a fairly simple solution: Choose a G*, evaluate performance based on it, but KEEP IT SECRET. This of course wouldn’t really work for national scale, GDP-esque kind of situations, but for corporate management situations it seems like it could work well enough. If only upper management knows what G* is, it becomes impossible to optimize for it, and everyone has to just keep working under the assumption they’re being evaluated on G.
Taking it a step further, to hedge against employees eventually figuring out G* and surreptitiously optimizing for it, you could have a bounty on guessing G* - the first employee who figures out what the mystery metric G* really is gets a prize, and as soon as it’s claimed, you switch to using G**
The hedge is absolutely necessary, elsewise, a manager will just tell subordinates what G* is in order to look impressive for managing a high-performing group.
Andrew Grove (of Intel fame) wrote a book, High Output Management, suggesting that management needs two opposing metrics to avoid this problem. For example, measure productivity and number of defects, and score people on the combined results.
BTW the large nail/little nail joke has a third part. Soviet management eventually got a clue and started measuring by the value of the nails produced… and the result was the world’s first solid-gold-nail factory.
Presumably finding arbitrarily many basic G*‘s will be hard. Two ideas for dealing with this: 1. Even if you only have finitely many and they’re all known, you could select one at random each time there’s a switch. 2. Each time there’s a switch, select a somehow-random linear (or some other sort, if you like) combination of your basic G*’s. (That would make guessing it in the first place quite hard, actually...)
if management are doing that then are neglecting a powerful tool in their tool-kit, because announcing a G will surely cause G to fall, and experience says that to begin with a well-chosen G and G remain correlated (because many of the things to do to reduce G also reduce G). It is only over time that G* and G detach.