Mo Putera comments on You Are Not Measuring What You Think You Are Measuring

[Mo Putera]

I think your ‘Towards a coherent process for metric design’ section alone is worth its weight in gold. Since most LW readers aren’t going to click on your linked paper (click-through rates being as low in general as they are, from my experience in marketing analytics), let me quote that section wholesale:

Given the various strategies and considerations discussed in the paper, as well as failure modes and limitations, it is useful to lay out a simple and coherent outline of a process for metric design. While this will by necessity be far from complete, and will include items that may not be relevant for a particular application, it should provide at least an outline that can be adapted to various metric design processes. Outside of the specific issues discussed earlier, there is a wide breadth of expertise and understanding that may be needed for metric design. Citations in this section will also provide a variety of resources for at least introductory further reading on those topics.

1. Understand the system being measured, including both technical (Blanchard & Fabrycky, 1990) and organizational (Berry & Houston, 1993) considerations.

   1. Determine scope

      1. What is included in the system?

      2. What will the metrics be used for?

   2. Understand the causal structure of the system

      1. What is the logic model or theory? (Rogers, Petrosino, Huebner, & Hacsi, 2000)

      2. Is there formal analysis (Gelman, 2010) or expert opinion (van Gelder, Vodicka, & Armstrong, 2016) that can inform this?

   3. Identify stakeholders (Kenny, 2014)

      1. Who will be affected?

      2. Who will use the metrics?

      3. Whose goals are relevant?

2. Identify the Goals

   1. What immediate goals are being served by the metric(s)? How are individual impacts related to performance more broadly? (Ruch, 1994)

   2. What longer term or broader goals are implicated?

3. Identify Relevant Desiderata

   1. Availability

   2. Cost

   3. Immediacy

   4. Simplicity

   5. Transparency

   6. Fairness

   7. Corruptibility

4. Brainstorm potential metrics

   1. What outcomes important to capture?

   2. What data sources exist?

   3. What methods can be used to capture additional data?

   4. What measurements are easy to capture?

   5. What is the relationship between the measurements and the outcomes?

   6. What isn’t captured by the metrics?

5. Consider and Plan

   1. Understand why and how the metric is useful. (Manheim, 2018)

   2. Consider how the metrics will be used to diagnose issues or incentify people.(Dai, Dietvorst, Tuckfield, Milkman, & Schweitzer, 2017)

   3. Plan how to use the metrics to develop the system, avoiding the “reward /​ punish” dichotomy. (Wigert & Harter, 2017)

   4. Perform a pre-mortem (Klein, 2007)

6. Plan to routinely revisit the metrics (Atkins, Wanick, & Wills, 2017)

[Davidmanheim]

Thanks!