What do you think I am wrong about here? What considerations am I missing? What should I focus more attention on?
I mean… I really hope this isn’t the list of considerations you consider most load-bearing for your estimates of AI risk. The biggest determinants are of course how hard it is to control system much smarter than us, and whether that will happen any time soon. It seems like the answer to the first one is quite solidly “very hard” (though not like arbitrarily hard) and the answer to the second one seems very likely “yes, quite soon, sometime in the coming decades, unless something else seriously derails humanity”.
After that, the questions are “can humanity coordinate to have more time, or somehow get much better at controlling systems smarter than us?”. The answer to the former is “probably not, but it’s really hard to say” and the answer to the latter is IMO “very likely not, but it’s still worth a shot”.
Another relevant crux/consideration would be: “will it turn out to be the case that controlling AI systems scales very smoothly, i.e. you can use a slightly superhuman system to control an even smarter system, etc. in an inductive fashion?”
My current answer to that is “no”, and it seems quite hard to learn more before it becomes a load-bearing guess. Digging into that requires building pretty detailed models.
I think you basically don’t talk about any of these in your post above? I also don’t know where your 5% comes from, so I don’t really know how to interface with it. Like do you actually mean that with 95% probability we will end up with an aligned superintelligence that will allow us to genuinely capture most of our cosmic endowment? If so, I have no idea why you believe that. Maybe you mean that with 90% probability AI will not really matter and never reach superhuman capabilities? That seems very unlikely to me, but seems fine to write about.
I mean… I really hope this isn’t the list of considerations you consider most load-bearing for your estimates of AI risk. The biggest determinants are of course how hard it is to control system much smarter than us, and whether that will happen any time soon. It seems like the answer to the first one is quite solidly “very hard” (though not like arbitrarily hard) and the answer to the second one seems very likely “yes, quite soon, sometime in the coming decades, unless something else seriously derails humanity”.
After that, the questions are “can humanity coordinate to have more time, or somehow get much better at controlling systems smarter than us?”. The answer to the former is “probably not, but it’s really hard to say” and the answer to the latter is IMO “very likely not, but it’s still worth a shot”.
Another relevant crux/consideration would be: “will it turn out to be the case that controlling AI systems scales very smoothly, i.e. you can use a slightly superhuman system to control an even smarter system, etc. in an inductive fashion?”
My current answer to that is “no”, and it seems quite hard to learn more before it becomes a load-bearing guess. Digging into that requires building pretty detailed models.
I think you basically don’t talk about any of these in your post above? I also don’t know where your 5% comes from, so I don’t really know how to interface with it. Like do you actually mean that with 95% probability we will end up with an aligned superintelligence that will allow us to genuinely capture most of our cosmic endowment? If so, I have no idea why you believe that. Maybe you mean that with 90% probability AI will not really matter and never reach superhuman capabilities? That seems very unlikely to me, but seems fine to write about.