IMO, it’s better to identify the set of metrics you want to optimize on various dimensions. Trying to collapse it into one loses a lot of information and accelerates Goodhart.
GDP is a fine component to continue to use—it’s not sufficient, but it does correlate fairly well with commercial production of a region. Adding in QALs (it’s annual, so automatically is years. you only need to report quality-adjusted lives to compare year-to-year) doesn’t seem wrong to me.
Another option is to just try to optimize for the thing directly, without specifying a measurement. Then come up with ad-hoc measurements that make sense for any given situation to make sure you’re on track.
There’s obviously a cost to doing this, but also benefits.
Without measurements, it’s hard to know that “the thing” you’re optimizing for is actually a thing at all, and nearly impossible to know if it’s the same thing as someone else is optimizing for.
Agreed that you shouldn’t lose sight that the measurements are usually proxies and reifications of what you want, and you need to periodically re-examine which measures should be replaced when they’re no longer useful for feedback purposes. But disagreed that you can get anywhere with no measurements. Note that I think of “measurement” as a fairly broad term—any objective signal of the state of the world.
Surely you can use your own intuitions? That will capture much more data than any individual measurement.
I do agree that it’s harder to make sure people are optimizing for the same things in this case, especially as an organization goes over successive Dunbar Numbers.
Of course. Understanding and refining your intuitions is a critical part of this (“this” being rational goal definition and pursuit). And the influence goes in both directions—measurements support or refute your intuitions, and your intuitions guide what to measure and how precisely. I’ll argue that this is true intrapersonally (you’ll have conflicting intuitions, and it’ll require measurement and effort to understand their limits), as well as for sub- and super-dunbar groups.
I don’t think I understand “vibing” well enough to know if it’s any different than simply discussing things at multiple different levels of abstraction.
- measurements support or refute your intuitions, and your intuitions guide what to measure and how precisely. I’ll argue that this is true intrapersonally (you’ll have conflicting intuitions, and it’ll require measurement and effort to understand their limits), as well as for sub- and super-dunbar groups.
Yes, my point being, a valid approach for certain projects is to use your intuitions to guide you, and then use ad-hoc measurements at various points to ensure your intuitions are doing well.
IMO, it’s better to identify the set of metrics you want to optimize on various dimensions. Trying to collapse it into one loses a lot of information and accelerates Goodhart.
GDP is a fine component to continue to use—it’s not sufficient, but it does correlate fairly well with commercial production of a region. Adding in QALs (it’s annual, so automatically is years. you only need to report quality-adjusted lives to compare year-to-year) doesn’t seem wrong to me.
Another option is to just try to optimize for the thing directly, without specifying a measurement. Then come up with ad-hoc measurements that make sense for any given situation to make sure you’re on track.
There’s obviously a cost to doing this, but also benefits.
Without measurements, it’s hard to know that “the thing” you’re optimizing for is actually a thing at all, and nearly impossible to know if it’s the same thing as someone else is optimizing for.
Agreed that you shouldn’t lose sight that the measurements are usually proxies and reifications of what you want, and you need to periodically re-examine which measures should be replaced when they’re no longer useful for feedback purposes. But disagreed that you can get anywhere with no measurements. Note that I think of “measurement” as a fairly broad term—any objective signal of the state of the world.
Surely you can use your own intuitions? That will capture much more data than any individual measurement.
I do agree that it’s harder to make sure people are optimizing for the same things in this case, especially as an organization goes over successive Dunbar Numbers.
This is one reason that vibing is important.
Of course. Understanding and refining your intuitions is a critical part of this (“this” being rational goal definition and pursuit). And the influence goes in both directions—measurements support or refute your intuitions, and your intuitions guide what to measure and how precisely. I’ll argue that this is true intrapersonally (you’ll have conflicting intuitions, and it’ll require measurement and effort to understand their limits), as well as for sub- and super-dunbar groups.
I don’t think I understand “vibing” well enough to know if it’s any different than simply discussing things at multiple different levels of abstraction.
Yes, my point being, a valid approach for certain projects is to use your intuitions to guide you, and then use ad-hoc measurements at various points to ensure your intuitions are doing well.