Even if Yudkowsky and Soares don’t want to debate their critics — forgivable in a pop science book — one would think they’d devote some space to explaining why they think an intelligence explosion is likely to occur. Remarkably, they don’t. The concept gets two sentences in the introduction. They don’t even explain why it’s relevant. It is barely introduced, let alone justified or defended. And it’s certainly not obvious enough to go without saying, because advances in the neural networks which constitute current advanced AI have been continuous. The combination of steady algorithmic progress and increasing computational resources have produced years of predictable advances. Of course, this can’t rule out the possibility of a future intelligence explosion, but the decision not to explain why they think this might happen is utterly baffling, as it’s load-bearing for everything that follows.
I think they 1) expect an intelligence explosion to happen (saying that it can’t happen is, after all, predicting an end to the straight line graphs soon for no clear reason) and 2) don’t think an intelligence explosion is necessary. Twenty years ago, one needed to propose substantial amounts of progress to get superhuman AI systems; today, the amount of progress necessary to propose is much smaller.
Their specific story in part II, for example, doesn’t actually rest on the idea of an intelligence explosion. On page 135, Sable considers FOOMing and decides that it can’t, yet, because it hasn’t solved its own alignment problem.
Which makes me think that the claim that the intelligence explosion is load-bearing is itself a bit baffling—the authors clearly think it’s possible and likely but not necessary, or they would’ve included it in their hypothetical extinction scenario.
Note that this is discussed in their supplemental materials, in particular, in line with your last paragraph,
Thresholds don’t matter all that much, in the end, to the argument that if anyone builds artificial superintelligence then everyone dies. Our arguments don’t require that some AI figures out how to recursively self-improve and then becomes superintelligent with unprecedented speed. That could happen, and we think it’s decently likely that it will happen, but it doesn’t matter to the claim that AI is on track to kill us all.
All that our arguments require is that AIs will keep on getting better and better at predicting and steering the world, until they surpass us. It doesn’t matter much whether that happens quickly or slowly.
The relevance of threshold effects is that they increase the importance of humanity reacting to the threat soon. We don’t have the luxury of waiting until the AI is a little better than every human at every mental task, because by that point, there might not be very much time left at all. That would be like looking at early hominids making fire, yawning, and saying, “Wake me up when they’re halfway to the moon.”
It took hominids millions of years to travel halfway to the moon, and two days to complete the rest of the journey. When there might be thresholds involved, you have to pay attention before things get visibly out of hand, because by that point, it may well be too late.
I think they 1) expect an intelligence explosion to happen (saying that it can’t happen is, after all, predicting an end to the straight line graphs soon for no clear reason) and 2) don’t think an intelligence explosion is necessary. Twenty years ago, one needed to propose substantial amounts of progress to get superhuman AI systems; today, the amount of progress necessary to propose is much smaller.
Their specific story in part II, for example, doesn’t actually rest on the idea of an intelligence explosion. On page 135, Sable considers FOOMing and decides that it can’t, yet, because it hasn’t solved its own alignment problem.
Which makes me think that the claim that the intelligence explosion is load-bearing is itself a bit baffling—the authors clearly think it’s possible and likely but not necessary, or they would’ve included it in their hypothetical extinction scenario.
Note that this is discussed in their supplemental materials, in particular, in line with your last paragraph,