At some point I hoped that CFAR would come up with “rationality trials”, toy challenges that are difficult to game and transfer well to some subset of real world situations. Something like boxing, or solving math problems. But a new entry in that row.
IMO standardized tests of this form are hard; I was going to say “mainstream academia hasn’t done much better” but Stanovich published something in 2016 that I’m guessing no one at CFAR has read (except maybe Dan?). I am not aware of any sustained research attempts on CFAR’s part to do this. [My sense is lots of people looked at it for a little bit, thought “this is hard”, and then dug in ground that seemed more promising.]
I think there are more labor-intensive / less clean alternatives that could have worked. We could have, say, just made the equivalent of Bridgewater Baseball Cards for rationality, and had people rate each other. This is sadly still a little ‘marketing’ instead of ‘object-level’ (the metric of “am I open-minded?” grounds in “do other people think I’m open-minded?” instead of just pointing at my brain and the environment), and maybe is painful for the people involved / gets them doing weird mental patterns instead of healthy mental patterns. But I think the visibility would have been good / it would have been easier to tell when someone is making ‘real progress’ vs. ‘the perception of progress’.
At some point I hoped that CFAR would come up with “rationality trials”, toy challenges that are difficult to game and transfer well to some subset of real world situations. Something like boxing, or solving math problems. But a new entry in that row.
IMO standardized tests of this form are hard; I was going to say “mainstream academia hasn’t done much better” but Stanovich published something in 2016 that I’m guessing no one at CFAR has read (except maybe Dan?). I am not aware of any sustained research attempts on CFAR’s part to do this. [My sense is lots of people looked at it for a little bit, thought “this is hard”, and then dug in ground that seemed more promising.]
I think there are more labor-intensive / less clean alternatives that could have worked. We could have, say, just made the equivalent of Bridgewater Baseball Cards for rationality, and had people rate each other. This is sadly still a little ‘marketing’ instead of ‘object-level’ (the metric of “am I open-minded?” grounds in “do other people think I’m open-minded?” instead of just pointing at my brain and the environment), and maybe is painful for the people involved / gets them doing weird mental patterns instead of healthy mental patterns. But I think the visibility would have been good / it would have been easier to tell when someone is making ‘real progress’ vs. ‘the perception of progress’.