I broadly disagree with Yudkowsky on his vision of FOOM and think he’s pretty sloppy wrt. AI takeoff overall.
But, I do think you’re quite likely to get a quite rapid singularity if people don’t intentionally slow things down. For instance, I broadly think the modeling in Tom Davidson’s takeoff speeds report seems very reasonable to me. Except that I think the default parameters he uses are insufficiently aggressive (I think compute requirements are likely to be somewhat lower than given in this report). Notably this model doesn’t get you FOOM in a week (perhaps more like 20% automation to 100% automation in a few years) and is mostly just doing non-mechanistic trend expolation combined with a growth model.
So, I think it would be much more interesting from my perspective to engage with this report than Yudkowsky.
I broadly disagree with Yudkowsky on his vision of FOOM and think he’s pretty sloppy wrt. AI takeoff overall.
But, I do think you’re quite likely to get a quite rapid singularity if people don’t intentionally slow things down. For instance, I broadly think the modeling in Tom Davidson’s takeoff speeds report seems very reasonable to me. Except that I think the default parameters he uses are insufficiently aggressive (I think compute requirements are likely to be somewhat lower than given in this report). Notably this model doesn’t get you FOOM in a week (perhaps more like 20% automation to 100% automation in a few years) and is mostly just doing non-mechanistic trend expolation combined with a growth model.
So, I think it would be much more interesting from my perspective to engage with this report than Yudkowsky.
See also this section where Tom talks about kinks in the underlying capabilities leading to rapid progress