Also, this is reference class tennis. If the rules allow for changing the metric in the middle of the debate, I shoot back with “the first telegraph cable improved transatlantic communication latency more than ten-million-fold the instant it was turned on; how’s that for a discontinuity”.
To be clear, I’m not saying “there’s this iron law about technology and you might thing nuclear weapons disprove it, but they don’t because <reasons>” (I’m not claiming there are any laws or hard rules about anything at all). What I’m saying is that there’s a thing that usually happens, but it didn’t happen with nuclear weapons, and I think we can see why. Nuclear weapons absolutely do live in the relevant reference class, and I think the way their development happened should make us more worried about AGI.
To be clear, I’m not saying “there’s this iron law about technology and you might thing nuclear weapons disprove it, but they don’t because <reasons>” (I’m not claiming there are any laws or hard rules about anything at all). What I’m saying is that there’s a thing that usually happens, but it didn’t happen with nuclear weapons, and I think we can see why. Nuclear weapons absolutely do live in the relevant reference class, and I think the way their development happened should make us more worried about AGI.