Completely agree that this is a crucial topic that is under discussed for research. The sort of knowledge/skill you’re pointing out also fits really well with the concept of Tacit Knowledge: the sort of expertise that is really hard to verbalize and explain.
That being said, I think some of your proposals are confusing different skills for research.
In fact, we do have this, in the form of blogs. The right type of blogs can provide a valuable form of “film study”. I personally learned a lot about statistics from Andrew Gelman’s blog. Often, someone sends him a paper and he just gives his off-the-cuff reactions to it: what he liked and didn’t, what was convincing, what parts seem sketchy. I probably learned more from reading his blog than from statistics classes (of which I’ve taken embarrassingly few, yet somehow managed to get hired by a Statistics department; I’ll credit Gelman for this). Scott Aaronson’s blog is good in the same way for theoretical computer science. Many posts on the GiveWell and Open Philanthropy blogs are good in this way, too. In all cases, I’d look at the earlier rather than later posts (though not the very earliest); the reason is that once blogs have too large an audience, writers start to feel constrained to write more “professionally” and you get less of the valuable off-the-cuff thinking.
While academic blogs can be analogous to film study, it’s really important to note that the expertise displayed is very often that of judging or understanding work in the field. From my experience reading Aaronson’s blog, that’s what he shares best, especially with regard to the limitations of quantum computing. I haven’t really digged into Gelman’s blog, but the format you mention is a perfect example of the expertise of understanding some research. Very important skill, but not the same as actually conducting the research that goes into a paper.
Pair programming or programming live on twitch are better examples of film study analogues for the latter skill. For research, I haven’t found much better than the tacit knowledge method of spending some time with an expert and actually looking for what they do in each context. That’s similar to your advisor point, except it scales just a little better, because you don’t need as strong a relationship.
Biographies of researchers can also prove a great variant of your “read history” proposal. You have to select carefully, but a well-research biography can put in context a lot of the thinking that the research did, and how they move between different ideas.
I also believe there is some slight issue with the analogy with film studies:
The first habit is film study. Almost every high-level athlete watches films of other players of the same sport, including historical greats, contemporary rivals, and themselves. This allows them to incorporate good ideas from other players’ games as well as to catch and eliminate flaws in their own game. Even the very best players benefit from watching film of themselves and others.
In research, especially in a weird new field like alignment, it’s rare to find another researcher who want to conduct precisely the same research. But that’s the basis of every sport and game: people want to win the same game. It make the whole “learning from other” slightly more difficult IMO. You can’t just look for what works, you constantly have to repurpose ideas that work in slightly different field and/or approaches and check for the loss in translation.
A better analogy might be to the sort of study Bruce Lee did to create a whole new martial art. He studied specific techniques, sure, but mostly for adapting them to his purpose, instead of playing the “game” of the original martial art.
I haven’t really digged into Gelman’s blog, but the format you mention is a perfect example of the expertise of understanding some research. Very important skill, but not the same as actually conducting the research that goes into a paper.
Research consists of many skills put together. Understanding prior work and developing the taste to judge it is one of the more important individual skills in research (moreso than programming, at least in most fields). So I think the blog example is indeed a central one.
In research, especially in a weird new field like alignment, it’s rare to find another researcher who want to conduct precisely the same research. But that’s the basis of every sport and game: people want to win the same game. It make the whole “learning from other” slightly more difficult IMO. You can’t just look for what works, you constantly have to repurpose ideas that work in slightly different field and/or approaches and check for the loss in translation.
I agree with this, although I think creative new ideas often come from people who have also mastered the “standard” skills. And indeed, most research is precisely about coming up with new ideas, which is a skill that you can cultivate my studying how others generate ideas.
More tangentially, you may be underestimating the amount of innovation in sports. Harden and Jokic both innovate in basketball (among others), but I am pretty sure they also do lots of film study. Jokic’s innovation probably comes from having mastered other sports like water polo and the resulting skill transfer. I would guess that mastery of fruitfully adjacent fields is a productive way to generate ideas.
Research consists of many skills put together. Understanding prior work and developing the taste to judge it is one of the more important individual skills in research (moreso than programming, at least in most fields). So I think the blog example is indeed a central one.
I completely agree that it is a relevant and important skill, but there are many people with good understanding of prior work who are completely unable of producing interesting new research. Non-exhaustively, this includes being able to have new ideas, to develop them, to test them, to get feedback and adapt to the feedback. And given that understanding prior work emerges pretty naturally once you read a lot of papers, I’m personally more interested in training for these other skills. My argument was that blogs don’t really help for that.
I agree with this, although I think creative new ideas often come from people who have also mastered the “standard” skills. And indeed, most research is precisely about coming up with new ideas, which is a skill that you can cultivate my studying how others generate ideas.
Difference of opinion: for me, coming with ideas is incredibly cheap. I also have piles of promising ideas that I will never have the time to explore, and I keep having new ideas. I never needed any help in that, and so I am completely uninterested in any way to generate more ideas. The other skills of research require way more effort to me (not even sure how to disentangle them TBH), so I focus on those. And I have trouble finding any actual standard skills that translate directly between research field: even things like doing experiments have very different meaning and related skills depending on the field.
More tangentially, you may be underestimating the amount of innovation in sports. Harden and Jokic both innovate in basketball (among others), but I am pretty sure they also do lots of film study. Jokic’s innovation probably comes from having mastered other sports like water polo and the resulting skill transfer. I would guess that mastery of fruitfully adjacent fields is a productive way to generate ideas.
Didn’t want to imply that athletes never innovate. And that’s an interesting example of the innovation from adjacent field. That’s definitely how I get a lot of ideas. But that’s still made incredibly more potent by being able to study and master the skills from your actual field. Which is really hard to do when there is no film study analogy for it.
Completely agree that this is a crucial topic that is under discussed for research. The sort of knowledge/skill you’re pointing out also fits really well with the concept of Tacit Knowledge: the sort of expertise that is really hard to verbalize and explain.
That being said, I think some of your proposals are confusing different skills for research.
While academic blogs can be analogous to film study, it’s really important to note that the expertise displayed is very often that of judging or understanding work in the field. From my experience reading Aaronson’s blog, that’s what he shares best, especially with regard to the limitations of quantum computing. I haven’t really digged into Gelman’s blog, but the format you mention is a perfect example of the expertise of understanding some research. Very important skill, but not the same as actually conducting the research that goes into a paper.
Pair programming or programming live on twitch are better examples of film study analogues for the latter skill. For research, I haven’t found much better than the tacit knowledge method of spending some time with an expert and actually looking for what they do in each context. That’s similar to your advisor point, except it scales just a little better, because you don’t need as strong a relationship.
Biographies of researchers can also prove a great variant of your “read history” proposal. You have to select carefully, but a well-research biography can put in context a lot of the thinking that the research did, and how they move between different ideas.
I also believe there is some slight issue with the analogy with film studies:
In research, especially in a weird new field like alignment, it’s rare to find another researcher who want to conduct precisely the same research. But that’s the basis of every sport and game: people want to win the same game. It make the whole “learning from other” slightly more difficult IMO. You can’t just look for what works, you constantly have to repurpose ideas that work in slightly different field and/or approaches and check for the loss in translation.
A better analogy might be to the sort of study Bruce Lee did to create a whole new martial art. He studied specific techniques, sure, but mostly for adapting them to his purpose, instead of playing the “game” of the original martial art.
Thanks for the feedback!
Research consists of many skills put together. Understanding prior work and developing the taste to judge it is one of the more important individual skills in research (moreso than programming, at least in most fields). So I think the blog example is indeed a central one.
I agree with this, although I think creative new ideas often come from people who have also mastered the “standard” skills. And indeed, most research is precisely about coming up with new ideas, which is a skill that you can cultivate my studying how others generate ideas.
More tangentially, you may be underestimating the amount of innovation in sports. Harden and Jokic both innovate in basketball (among others), but I am pretty sure they also do lots of film study. Jokic’s innovation probably comes from having mastered other sports like water polo and the resulting skill transfer. I would guess that mastery of fruitfully adjacent fields is a productive way to generate ideas.
Sorry for taking so long to answer!
I completely agree that it is a relevant and important skill, but there are many people with good understanding of prior work who are completely unable of producing interesting new research. Non-exhaustively, this includes being able to have new ideas, to develop them, to test them, to get feedback and adapt to the feedback. And given that understanding prior work emerges pretty naturally once you read a lot of papers, I’m personally more interested in training for these other skills. My argument was that blogs don’t really help for that.
Difference of opinion: for me, coming with ideas is incredibly cheap. I also have piles of promising ideas that I will never have the time to explore, and I keep having new ideas. I never needed any help in that, and so I am completely uninterested in any way to generate more ideas. The other skills of research require way more effort to me (not even sure how to disentangle them TBH), so I focus on those. And I have trouble finding any actual standard skills that translate directly between research field: even things like doing experiments have very different meaning and related skills depending on the field.
Didn’t want to imply that athletes never innovate. And that’s an interesting example of the innovation from adjacent field. That’s definitely how I get a lot of ideas. But that’s still made incredibly more potent by being able to study and master the skills from your actual field. Which is really hard to do when there is no film study analogy for it.