rather than actually talking about the details, which is what I would usually find useful about reviews.
I’m interested in details about what you find useful about the prospect of reviews that talk about the details. I share a sense that it’d be helpful, but I’m not sure I could justify that belief very strongly (when it comes to the opportunity cost of the people qualified to do the job)
In general, I’m legit fairly uncertain whether “effort-reviews”(whether detail-focused or big-picture focused) are worthwhile. It seems plausible to me that detail-focused-reviews are more useful soon after a work is published than 2 years later, and that big-picture-reviews are more useful in the “two year retrospective” sense (and maybe we should figure out some way to get detail-oriented reviews done more frequently, faster).
It does seem to me that, by the time a book is being considered for “long-term-valuable’, I would like someone, at some point, to have done a detail-oriented review examining all of the fiddly pieces. In some cases, that review has been done before the post was even published, in a private google doc.
It’s far easier for me to figure out how much to update on evidence when someone else has looked at the details and highlighted ways in which the evidence is stronger or weaker than a reader might naively take away from the paper. (At least, assuming the reviewer did a good job.)
This doesn’t apply to big-picture reviews because such reviews are typically a rehash of old arguments I already know.
This is similar to the general idea in AI safety via debate—when you have access to a review you are more like a judge; without a review you are more like the debate opponent.
Having someone else explain the paper from their perspective can surface other ways of thinking about the paper that can help with understanding it.
This sometimes does happen with big-picture reviews, though I think it’s less common.
Tbc, I’m not necessarily saying it is worth the opportunity cost of the reviewer’s time; I haven’t thought much about it.
I’m interested in details about what you find useful about the prospect of reviews that talk about the details. I share a sense that it’d be helpful, but I’m not sure I could justify that belief very strongly (when it comes to the opportunity cost of the people qualified to do the job)
In general, I’m legit fairly uncertain whether “effort-reviews”(whether detail-focused or big-picture focused) are worthwhile. It seems plausible to me that detail-focused-reviews are more useful soon after a work is published than 2 years later, and that big-picture-reviews are more useful in the “two year retrospective” sense (and maybe we should figure out some way to get detail-oriented reviews done more frequently, faster).
It does seem to me that, by the time a book is being considered for “long-term-valuable’, I would like someone, at some point, to have done a detail-oriented review examining all of the fiddly pieces. In some cases, that review has been done before the post was even published, in a private google doc.
A couple of reasons:
It’s far easier for me to figure out how much to update on evidence when someone else has looked at the details and highlighted ways in which the evidence is stronger or weaker than a reader might naively take away from the paper. (At least, assuming the reviewer did a good job.)
This doesn’t apply to big-picture reviews because such reviews are typically a rehash of old arguments I already know.
This is similar to the general idea in AI safety via debate—when you have access to a review you are more like a judge; without a review you are more like the debate opponent.
Having someone else explain the paper from their perspective can surface other ways of thinking about the paper that can help with understanding it.
This sometimes does happen with big-picture reviews, though I think it’s less common.
Tbc, I’m not necessarily saying it is worth the opportunity cost of the reviewer’s time; I haven’t thought much about it.