Thanks for sharing. I definitely appreciate it all as user-feedback.
I think I have some high-level thoughts that don’t depend much on the details of this particular post and these particular reviews, and then some object-level thoughts.
At a high level:
By default, serious in-depth reviews are a lot of work, and AFAICT fairly unrewarding. A lot of what I was trying to do with this post and prizes is correct an ecosystem incentives-issue where people aren’t rewarded for doing a sort of “intellectual grunt work” that’s important but underappreciated. (Part of what I appreciated about Shimi-et-al was them initiating a process for peer review in general, not just for this one particular post)
In general my posting a review here means I got something out of it, but not that I endorse everything in it. I’m also doing all this with a bit of limited time and trying to cover a lot of breadth, so I’m not too surprised if there are significant criticisms to be made of some reviews.
I also think, well, if the system is working, reviewers should sometimes say things the authors don’t like, and that’s okay. I wouldn’t argue the current system is that great (including this post and prizes, and my current approach to aggregating them). But I don’t currently think anything necessarily went wrong here.
But, being misunderstood sucks, and I do empathize/sympathize. I’ve appreciated your work on the review this year and I definitely appreciate +12 OOMs as a post. (I noticed +12 OOMs getting a disproportionate amount of review attention, and in the culture-I-hope-for this feels like a compliment, even if parts of the process are frustrating)
Some object-level thoughts:
I agree that Shimi-et-al’s argument about “The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame” isn’t a fair characterization of what you wrote. (In an ideal world I’d have read more of the back-and-forth-between you and Shimi on their review, and incorporated that into my commentary here)
I think I mostly appreciated their review for digging into the details of the examples in the second half.
I had stated that Zach Perlman’s review made a similar point to Nostalgebraist’s. Looking back, I’m not sure whether I stand by that. I don’t think I’d have derived Nostalgebraist’s point of “The impetus to ask “what does future compute enable?” rather than “how much compute might TAI require?” influenced my own view of Bio Anchors” from Zach’s if that’s all I had to go on.
I said, reading Nostalgebraist’s review “I feel like I understand the point for the first time.” I did notice that he didn’t frame it the same way you did, and I’m not sure whether I endorse my phrasing. Maybe Nostalebraist’s interpretation is more of it’s own thing than a point you made. But, I did feel like it added another layer to your post, and somehow made things feel more crisp to me as a useful meta-level-insight than Zach’s (or your) summary.
I may have more thoughts, but wanted to post this for now.
Just chiming in to say huge +1 to the idea of rewarding people for doing reviews, it’s an awesome and very pro-social thing to do and I’m honored that so many people chose my post to review. I endorse rewarding Shimi et al, and Nostalgebraist, in particular.
Also: I happen to be having a related conversation that also gives some context on how I conceived of the OP at least & what I hoped to accomplish with it.
Thanks for sharing. I definitely appreciate it all as user-feedback.
I think I have some high-level thoughts that don’t depend much on the details of this particular post and these particular reviews, and then some object-level thoughts.
At a high level:
By default, serious in-depth reviews are a lot of work, and AFAICT fairly unrewarding. A lot of what I was trying to do with this post and prizes is correct an ecosystem incentives-issue where people aren’t rewarded for doing a sort of “intellectual grunt work” that’s important but underappreciated. (Part of what I appreciated about Shimi-et-al was them initiating a process for peer review in general, not just for this one particular post)
In general my posting a review here means I got something out of it, but not that I endorse everything in it. I’m also doing all this with a bit of limited time and trying to cover a lot of breadth, so I’m not too surprised if there are significant criticisms to be made of some reviews.
I also think, well, if the system is working, reviewers should sometimes say things the authors don’t like, and that’s okay. I wouldn’t argue the current system is that great (including this post and prizes, and my current approach to aggregating them). But I don’t currently think anything necessarily went wrong here.
But, being misunderstood sucks, and I do empathize/sympathize. I’ve appreciated your work on the review this year and I definitely appreciate +12 OOMs as a post. (I noticed +12 OOMs getting a disproportionate amount of review attention, and in the culture-I-hope-for this feels like a compliment, even if parts of the process are frustrating)
Some object-level thoughts:
I agree that Shimi-et-al’s argument about “The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame” isn’t a fair characterization of what you wrote. (In an ideal world I’d have read more of the back-and-forth-between you and Shimi on their review, and incorporated that into my commentary here)
I think I mostly appreciated their review for digging into the details of the examples in the second half.
I had stated that Zach Perlman’s review made a similar point to Nostalgebraist’s. Looking back, I’m not sure whether I stand by that. I don’t think I’d have derived Nostalgebraist’s point of “The impetus to ask “what does future compute enable?” rather than “how much compute might TAI require?” influenced my own view of Bio Anchors” from Zach’s if that’s all I had to go on.
I said, reading Nostalgebraist’s review “I feel like I understand the point for the first time.” I did notice that he didn’t frame it the same way you did, and I’m not sure whether I endorse my phrasing. Maybe Nostalebraist’s interpretation is more of it’s own thing than a point you made. But, I did feel like it added another layer to your post, and somehow made things feel more crisp to me as a useful meta-level-insight than Zach’s (or your) summary.
I may have more thoughts, but wanted to post this for now.
Just chiming in to say huge +1 to the idea of rewarding people for doing reviews, it’s an awesome and very pro-social thing to do and I’m honored that so many people chose my post to review. I endorse rewarding Shimi et al, and Nostalgebraist, in particular.
Also: I happen to be having a related conversation that also gives some context on how I conceived of the OP at least & what I hoped to accomplish with it.