On the foom side, Paul Christiano brings up Eliezer Yudkowsky’s past expectation that ASI “would likely emerge from a small group rather than a large industry” as a failed prediction here [disagreement 12] and as “improbable and crazy” here.
Actually, I don’t think Paul says this is a failed prediction in the linked text. He says:
The Eliezer predictions most relevant to “how do scientific disciplines work” that I’m most aware of are incorrectly predicting that physicists would be wrong about the existence of the Higgs boson (LW bet registry) and expressing the view that real AI would likely emerge from a small group rather than a large industry (pg 436 but expressed many places).
My understanding is that this is supposed to be read as “[incorrectly predicting that physicists would be wrong about the existence of the Higgs boson (LW bet registry)] and [expressing the view that real AI would likely emerge from a small group rather than a large industry]”, Paul isn’t claiming that the view that real AI would likely emerge from a small group is a failed prediction!
On “improbable and crazy”, Paul says:
The debate was about whether a small group could quickly explode to take over the world. AI development projects are now billion-dollar affairs and continuing to grow quickly, important results are increasingly driven by giant projects, and 9 people taking over the world with AI looks if anything even more improbable and crazy than it did then. Now we’re mostly talking about whether a $10 trillion company can explosively grow to $300 trillion as it develops AI, which is just not the same game in any qualitative sense. I’m not sure Eliezer has many precise predictions he’d stand behind here (setting aside the insane pre-2002 predictions), so it’s not clear we can evaluate his track record, but I think they’d look bad if he’d made them.
Note that Paul says “looks if anything even more improbable and crazy than it did then”. I think your quotation is reasonable, but it’s unclear if Paul thinks this is “crazy” or if he thinks it’s just more incorrect and crazy-looking than it was in the past.
I just reworded from “as a failed prediction” to “as evidence against Eliezer’s judgment and expertise”. I agree that the former was not a good summary, but am confident that the latter is what Paul intended to convey and expected his readers to understand, based on the context of disagreement 12 (which you quoted part but not all of). Sorry, thanks for checking.
Actually, I don’t think Paul says this is a failed prediction in the linked text. He says:
My understanding is that this is supposed to be read as “[incorrectly predicting that physicists would be wrong about the existence of the Higgs boson (LW bet registry)] and [expressing the view that real AI would likely emerge from a small group rather than a large industry]”, Paul isn’t claiming that the view that real AI would likely emerge from a small group is a failed prediction!
On “improbable and crazy”, Paul says:
Note that Paul says “looks if anything even more improbable and crazy than it did then”. I think your quotation is reasonable, but it’s unclear if Paul thinks this is “crazy” or if he thinks it’s just more incorrect and crazy-looking than it was in the past.
I just reworded from “as a failed prediction” to “as evidence against Eliezer’s judgment and expertise”. I agree that the former was not a good summary, but am confident that the latter is what Paul intended to convey and expected his readers to understand, based on the context of disagreement 12 (which you quoted part but not all of). Sorry, thanks for checking.