David Roodman, who Holden Karnofsky once called “the gold standard for in-depth quantitative (generalist) research”, explained a central motivation underlying his style like so (emphasis mine) back in 2018:
Robert Wiblin: So, you’re someone who a lot of people trust to do kind of the most difficult empirical research, in effect [inaudible 00:07:57], where there’s either a lot of contentious evidence that has to be pulled together and reach a conclusion, or perhaps there’s very little evidence and so we need to get kind of as much juice out of it as we can. I think Holden has called you, “The gold standard for in-depth quantitative research.”
So how do you think you got to be that good?
David Roodman: Well, I won’t argue … and I won’t agree or disagree about whether I am that good. But I think that goes back even earlier in my life. Maybe I was just born with a certain sense of responsibility to the truth, as it were. But I am the child of a bitter divorce. My parents split when I was ten, and I grew up from that point on with the experience of there being these two gods in my life. They collided, and I couldn’t make sense of who was right and who was wrong when. But also felt a lot of fear about what happen if I chose sides? Alienating and losing a parent.
So I felt this very strong compulsion to go down the center, and if I ever strayed from the center, to be really well prepared to explain why I was doing it. And I think that actually drives my approach to researching. I’m really afraid of being wrong and so I always want to dig down the next level. And that’s part of how I’ve developed my style.
And this is a style that I feel like I’ve discovered. There was no grand plan, especially working for GiveWell and Open Philanthropy in the last three or four years. What I do that’s unusual is to review empirical research, mostly in the social sciences, and as much as possible, re-run the studies that I read for myself. I’ve hardly ever done new research, but I will go back to original data sources, try to understand the methods that were applied, re-do them, and then think critically about whether I agree with those methods or I want to apply alternatives.
And that arises both from my personality, as I already said, and also I think the fact that I don’t have formal training. I never did get a PhD, and I think probably people who come through PhD programs pick up a different kind of culture and maybe face different incentives, which discourages the kind of work I do. So I’ve kind of stumbled into this.
I thought about Roodman’s remarks (on the terror of being wrong) when I saw Wei Dai’s reflections on his cognitive style, and about Roodman’s preference to redo things for himself when I saw Leo Gao’s take re: “a big part of doing a job well is just being willing to spend a huge amount of time absorbing all of the relevant context and building a mental model of the problem… looking into the reasoning behind claims that other people just take at face value”. This fear of being wrong → rederive claims others take for granted → think about whether I agree or want to apply alternatives has been an unexpected repeat source of alpha for me, even at my “capability level” a few stdevs below these folks.
I’ve also been trying to get frontier models to be “terrified of being wrong” and to “pull a Roodman” when needed with bespoke iteratively-refined Claude skills, templates, etc. At least for now I’ve been surprised, even with the likes of Fable (max) and Sol (max), how much they still fall short for the thing I specialise in (which involves desk research-based valuation-oriented quant modelling, and requires something like respect for the messy truth over slick reassuring persuasiveness).
David Roodman, who Holden Karnofsky once called “the gold standard for in-depth quantitative (generalist) research”, explained a central motivation underlying his style like so (emphasis mine) back in 2018:
I thought about Roodman’s remarks (on the terror of being wrong) when I saw Wei Dai’s reflections on his cognitive style, and about Roodman’s preference to redo things for himself when I saw Leo Gao’s take re: “a big part of doing a job well is just being willing to spend a huge amount of time absorbing all of the relevant context and building a mental model of the problem… looking into the reasoning behind claims that other people just take at face value”. This fear of being wrong → rederive claims others take for granted → think about whether I agree or want to apply alternatives has been an unexpected repeat source of alpha for me, even at my “capability level” a few stdevs below these folks.
I’ve also been trying to get frontier models to be “terrified of being wrong” and to “pull a Roodman” when needed with bespoke iteratively-refined Claude skills, templates, etc. At least for now I’ve been surprised, even with the likes of Fable (max) and Sol (max), how much they still fall short for the thing I specialise in (which involves desk research-based valuation-oriented quant modelling, and requires something like respect for the messy truth over slick reassuring persuasiveness).