Warning: Long rant incoming, one you probably won’t benefit from reading unless you are Raemon, and in fact I’m a bit embarrassed to have written it:
I admit I feel some dismay at seeing Nostalgebraist’s review and especially Shimi/Collman/Gyrodiot’s reviews appear on this list. I respect all of these people as thinkers and upvoted their reviews, IIRC, and also I am genuinely honored and flattered that they not only read my post but took the time to review it. I won’t object if you pay them money for their reviews; I wish them well. In fact I’ll feel guilty if this comment of mine gets in the way of their reward, and I hope that it doesn’t.
But am having to do some serious soul-searching upon receiving the evidence that their reviews have stood the test of time and helped you understand my original post—because I think they both miss the point of the original post. Now I’m wondering what I did wrong, how I could have been so unclear in the OP, that so many people misunderstood...
Quoting from the original post:
I describe a hypothetical scenario that concretizes the question “what could be built with 2020’s algorithms/ideas/etc. but a trillion times more compute?” Then I give some answers to that question. Then I ask: How likely is it that some sort of TAI would happen in this scenario? This second question is a useful operationalization of the (IMO) most important, most-commonly-discussed timelines crux: “Can we get TAI just by throwing more compute at the problem?” I consider this operationalization to be the main contribution of this post; it directly plugs into Ajeya’s timelines model and is quantitatively more cruxy than anything else I know of. The secondary contribution of this post is my set of answers to the first question: They serve as intuition pumps for my answer to the second, which strongly supports my views on timelines.
I literally said right at the front (admittedly behind spoiler screen) what the main and secondary points of the post were. And the subtitle said it too: “Big Timelines Crux Operationalized.”
the Shimi/Collman/Gyrodiot review most seriously misunderstands the OP; see this quote from the review:
The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame. If not, then the arguments made by Daniel wouldn’t have the consequence of making people have shorter timelines.
The main point of the post was to focus the discussion on the big crux, not to argue for short timelines. The secondary point was an intuition pump for short timelines—but it does NOT depend on it being at all plausible for us to achieve +12 OOMs in the real world anytime soon! I said very clearly that the +12 OOMs thing was a hypothetical, involving magic! I brought this up in the comments; see discussion. You quote a passage that seems to be making the same mistake:
Another issue with this hypothesis is that it assumes, under the hood, exactly the kind of breakthrough that Daniel is trying so hard to remove from the software side. Our cursory look at Ajeya’s report (focused on the speed-up instead of the cost reduction) showed that almost all the hardware improvement forecasted came from breakthrough into currently not working (or not scalable) hardware. Even without mentioning the issue that none of these technologies look like they can provide anywhere near the improvement expected, there is still the fact that getting these orders of magnitude of compute requires many hardware breakthroughs, which contradicts Daniel’s stance on not needing new technology or ideas, just scaling.
To be fair to the authors, I didn’t spell out as much as I could have why it doesn’t matter if we ever achieve +12 OOMs in real life anytime soon. I mean I did spell it out, but I didn’t spell it out in as much detail as I could have—I relied on the readers being somewhat familiar with Ajeya’s model I guess. In response to a conversation with Adam Shimi after the review went up, I wrote the “Master Argument” google doc which you may have seen by now. It explains Ajeya’s model and then explains how having 80% by +12 gets you t much shorter timelines than just 50%. The key, I guess, is that if you move 30% of your mass from above 12 to below 12, unless you are crazy you will move a bunch of it to the 0-6 OOM range. You won’t pile it all up in the 6-12 OOM range. In retrospect I should have said more about that in the OP.
Anyhow. On to Nostalgebraist’s review:
...to be honest I’m not sure I understand it. The part of it where it’s talking about what the main point of Fun With +12 OOMs is… well, maybe it’s something interesting that I said, and maybe it’s equivalent to the main point under some transformation, but it’s certainly not how I think of the main point. I think the main point is “here’s this big timelines crux we all should be debating: What is the probability that +12 OOMs would be enough?” and the secondary point is “Here are some intuition pumps that +12 OOMs would be enough.”
Part of Nostalgebraist’s review was a critique of my secondary point. That part I agree with; there’s a LOT more that needs to be said (and a lot more I could have said, believe me!) about why +12 OOMs is probably enough, than just the 5 intuition pumps I gave. There’s a lot more I could do to make those 5 pumps pump harder, too. I hope someone one day finds the time to write all that stuff.
Side note: Zach Stein-Perlman’s review of Fun with +12 OOMs is great, I think he understood the original post quite well. The others… again, I appreciate them, they said some interesting things and some useful things, but it annoys me that they don’t seem to have understood the main point. And, as I said at the beginning, it makes me a bit defensive and soul-searchy. What did I do wrong? I thought I was being so clear, signposting everything, etc.!?! Yet multiple smart people I respect read it closely enough that they were motivated to review it, and came away with a different impression!
I think Nostalgebraist’s review might not deserve this reaction from me, actually. Like I said, maybe what they think the main takeaway is, is also what I thought it was, just described differently. And anyhow it’s possible that they understood perfectly what I thought the main takeaway was, and just disagreed with me about it—maybe they think that the most interesting and novel contribution isn’t what I thought it was! Fair enough. I may be making a mistake by dragging them into this. I probably shouldn’t be wasting time writing this anyway. But their review of Ajeya’s Bio Anchors report also rankled me in the same way, but more so—I think it misunderstood the whole point of the report, and I feel more confident in this claim than in the claims I made above.
Thanks for sharing. I definitely appreciate it all as user-feedback.
I think I have some high-level thoughts that don’t depend much on the details of this particular post and these particular reviews, and then some object-level thoughts.
At a high level:
By default, serious in-depth reviews are a lot of work, and AFAICT fairly unrewarding. A lot of what I was trying to do with this post and prizes is correct an ecosystem incentives-issue where people aren’t rewarded for doing a sort of “intellectual grunt work” that’s important but underappreciated. (Part of what I appreciated about Shimi-et-al was them initiating a process for peer review in general, not just for this one particular post)
In general my posting a review here means I got something out of it, but not that I endorse everything in it. I’m also doing all this with a bit of limited time and trying to cover a lot of breadth, so I’m not too surprised if there are significant criticisms to be made of some reviews.
I also think, well, if the system is working, reviewers should sometimes say things the authors don’t like, and that’s okay. I wouldn’t argue the current system is that great (including this post and prizes, and my current approach to aggregating them). But I don’t currently think anything necessarily went wrong here.
But, being misunderstood sucks, and I do empathize/sympathize. I’ve appreciated your work on the review this year and I definitely appreciate +12 OOMs as a post. (I noticed +12 OOMs getting a disproportionate amount of review attention, and in the culture-I-hope-for this feels like a compliment, even if parts of the process are frustrating)
Some object-level thoughts:
I agree that Shimi-et-al’s argument about “The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame” isn’t a fair characterization of what you wrote. (In an ideal world I’d have read more of the back-and-forth-between you and Shimi on their review, and incorporated that into my commentary here)
I think I mostly appreciated their review for digging into the details of the examples in the second half.
I had stated that Zach Perlman’s review made a similar point to Nostalgebraist’s. Looking back, I’m not sure whether I stand by that. I don’t think I’d have derived Nostalgebraist’s point of “The impetus to ask “what does future compute enable?” rather than “how much compute might TAI require?” influenced my own view of Bio Anchors” from Zach’s if that’s all I had to go on.
I said, reading Nostalgebraist’s review “I feel like I understand the point for the first time.” I did notice that he didn’t frame it the same way you did, and I’m not sure whether I endorse my phrasing. Maybe Nostalebraist’s interpretation is more of it’s own thing than a point you made. But, I did feel like it added another layer to your post, and somehow made things feel more crisp to me as a useful meta-level-insight than Zach’s (or your) summary.
I may have more thoughts, but wanted to post this for now.
Just chiming in to say huge +1 to the idea of rewarding people for doing reviews, it’s an awesome and very pro-social thing to do and I’m honored that so many people chose my post to review. I endorse rewarding Shimi et al, and Nostalgebraist, in particular.
Also: I happen to be having a related conversation that also gives some context on how I conceived of the OP at least & what I hoped to accomplish with it.
Warning: Long rant incoming, one you probably won’t benefit from reading unless you are Raemon, and in fact I’m a bit embarrassed to have written it:
I admit I feel some dismay at seeing Nostalgebraist’s review and especially Shimi/Collman/Gyrodiot’s reviews appear on this list. I respect all of these people as thinkers and upvoted their reviews, IIRC, and also I am genuinely honored and flattered that they not only read my post but took the time to review it. I won’t object if you pay them money for their reviews; I wish them well. In fact I’ll feel guilty if this comment of mine gets in the way of their reward, and I hope that it doesn’t.
But am having to do some serious soul-searching upon receiving the evidence that their reviews have stood the test of time and helped you understand my original post—because I think they both miss the point of the original post. Now I’m wondering what I did wrong, how I could have been so unclear in the OP, that so many people misunderstood...
Quoting from the original post:
I literally said right at the front (admittedly behind spoiler screen) what the main and secondary points of the post were. And the subtitle said it too: “Big Timelines Crux Operationalized.”
the Shimi/Collman/Gyrodiot review most seriously misunderstands the OP; see this quote from the review:
The main point of the post was to focus the discussion on the big crux, not to argue for short timelines. The secondary point was an intuition pump for short timelines—but it does NOT depend on it being at all plausible for us to achieve +12 OOMs in the real world anytime soon! I said very clearly that the +12 OOMs thing was a hypothetical, involving magic! I brought this up in the comments; see discussion. You quote a passage that seems to be making the same mistake:
To be fair to the authors, I didn’t spell out as much as I could have why it doesn’t matter if we ever achieve +12 OOMs in real life anytime soon. I mean I did spell it out, but I didn’t spell it out in as much detail as I could have—I relied on the readers being somewhat familiar with Ajeya’s model I guess. In response to a conversation with Adam Shimi after the review went up, I wrote the “Master Argument” google doc which you may have seen by now. It explains Ajeya’s model and then explains how having 80% by +12 gets you t much shorter timelines than just 50%. The key, I guess, is that if you move 30% of your mass from above 12 to below 12, unless you are crazy you will move a bunch of it to the 0-6 OOM range. You won’t pile it all up in the 6-12 OOM range. In retrospect I should have said more about that in the OP.
Anyhow. On to Nostalgebraist’s review:
...to be honest I’m not sure I understand it. The part of it where it’s talking about what the main point of Fun With +12 OOMs is… well, maybe it’s something interesting that I said, and maybe it’s equivalent to the main point under some transformation, but it’s certainly not how I think of the main point. I think the main point is “here’s this big timelines crux we all should be debating: What is the probability that +12 OOMs would be enough?” and the secondary point is “Here are some intuition pumps that +12 OOMs would be enough.”
Part of Nostalgebraist’s review was a critique of my secondary point. That part I agree with; there’s a LOT more that needs to be said (and a lot more I could have said, believe me!) about why +12 OOMs is probably enough, than just the 5 intuition pumps I gave. There’s a lot more I could do to make those 5 pumps pump harder, too. I hope someone one day finds the time to write all that stuff.
Side note: Zach Stein-Perlman’s review of Fun with +12 OOMs is great, I think he understood the original post quite well. The others… again, I appreciate them, they said some interesting things and some useful things, but it annoys me that they don’t seem to have understood the main point. And, as I said at the beginning, it makes me a bit defensive and soul-searchy. What did I do wrong? I thought I was being so clear, signposting everything, etc.!?! Yet multiple smart people I respect read it closely enough that they were motivated to review it, and came away with a different impression!
I think Nostalgebraist’s review might not deserve this reaction from me, actually. Like I said, maybe what they think the main takeaway is, is also what I thought it was, just described differently. And anyhow it’s possible that they understood perfectly what I thought the main takeaway was, and just disagreed with me about it—maybe they think that the most interesting and novel contribution isn’t what I thought it was! Fair enough. I may be making a mistake by dragging them into this. I probably shouldn’t be wasting time writing this anyway. But their review of Ajeya’s Bio Anchors report also rankled me in the same way, but more so—I think it misunderstood the whole point of the report, and I feel more confident in this claim than in the claims I made above.
Thanks for sharing. I definitely appreciate it all as user-feedback.
I think I have some high-level thoughts that don’t depend much on the details of this particular post and these particular reviews, and then some object-level thoughts.
At a high level:
By default, serious in-depth reviews are a lot of work, and AFAICT fairly unrewarding. A lot of what I was trying to do with this post and prizes is correct an ecosystem incentives-issue where people aren’t rewarded for doing a sort of “intellectual grunt work” that’s important but underappreciated. (Part of what I appreciated about Shimi-et-al was them initiating a process for peer review in general, not just for this one particular post)
In general my posting a review here means I got something out of it, but not that I endorse everything in it. I’m also doing all this with a bit of limited time and trying to cover a lot of breadth, so I’m not too surprised if there are significant criticisms to be made of some reviews.
I also think, well, if the system is working, reviewers should sometimes say things the authors don’t like, and that’s okay. I wouldn’t argue the current system is that great (including this post and prizes, and my current approach to aggregating them). But I don’t currently think anything necessarily went wrong here.
But, being misunderstood sucks, and I do empathize/sympathize. I’ve appreciated your work on the review this year and I definitely appreciate +12 OOMs as a post. (I noticed +12 OOMs getting a disproportionate amount of review attention, and in the culture-I-hope-for this feels like a compliment, even if parts of the process are frustrating)
Some object-level thoughts:
I agree that Shimi-et-al’s argument about “The relevance of this work appears to rely mostly on the hypothesis that the +12 OOMs of magnitude of compute and all relevant resources could plausibly be obtained in a short time frame” isn’t a fair characterization of what you wrote. (In an ideal world I’d have read more of the back-and-forth-between you and Shimi on their review, and incorporated that into my commentary here)
I think I mostly appreciated their review for digging into the details of the examples in the second half.
I had stated that Zach Perlman’s review made a similar point to Nostalgebraist’s. Looking back, I’m not sure whether I stand by that. I don’t think I’d have derived Nostalgebraist’s point of “The impetus to ask “what does future compute enable?” rather than “how much compute might TAI require?” influenced my own view of Bio Anchors” from Zach’s if that’s all I had to go on.
I said, reading Nostalgebraist’s review “I feel like I understand the point for the first time.” I did notice that he didn’t frame it the same way you did, and I’m not sure whether I endorse my phrasing. Maybe Nostalebraist’s interpretation is more of it’s own thing than a point you made. But, I did feel like it added another layer to your post, and somehow made things feel more crisp to me as a useful meta-level-insight than Zach’s (or your) summary.
I may have more thoughts, but wanted to post this for now.
Just chiming in to say huge +1 to the idea of rewarding people for doing reviews, it’s an awesome and very pro-social thing to do and I’m honored that so many people chose my post to review. I endorse rewarding Shimi et al, and Nostalgebraist, in particular.
Also: I happen to be having a related conversation that also gives some context on how I conceived of the OP at least & what I hoped to accomplish with it.