I think one of the points of this post is that you shouldn’t have or communicate one single number. These are different things, and you at the very least need to quantify “avoidable”, and figure out the correlations between them (like “human-caused degradation would reduce the world population by 90% if AI didn’t extend our ability to cope by 30 years, but then another part of fragility of human expectations makes civilization collapse anyway”).
At some point (which we’re well past in most discussions around here), it becomes too complex and FAR too dependent on assumptions with very large error bars (and very large conditionals on surprising levels of human coordination), that there is no way to predict the outcomes. About the best you can do is very large buckets of “somewhat likely”, “rather unlikely”, and “not gonna worry about it”, with another dimension of “how much, if any, will my actions change things”, also focused on paths wide enough that you’re not basing it on insane multiplication of very small/large made-up numbers.
‘Avoidable’ in above toy numbers are purely that 1 - avoidable doom directly caused by AI is in fact avoided if we destroy all (relevantly capable?) AI when testing for doom. 2 - avoidable doom directly caused by humans or nature is in fact avoided by AI technology we possess when testing for doom.
Still not sure I follow. “testing for doom” is done by experiencing the doom or non-doom-yet future at some point in time, right? And we can’t test under conditions that don’t actually obtain. Or do you have some other test that works on counterfactual (or future-unknown-maybe-factual) worlds?
I think one of the points of this post is that you shouldn’t have or communicate one single number. These are different things, and you at the very least need to quantify “avoidable”, and figure out the correlations between them (like “human-caused degradation would reduce the world population by 90% if AI didn’t extend our ability to cope by 30 years, but then another part of fragility of human expectations makes civilization collapse anyway”).
At some point (which we’re well past in most discussions around here), it becomes too complex and FAR too dependent on assumptions with very large error bars (and very large conditionals on surprising levels of human coordination), that there is no way to predict the outcomes. About the best you can do is very large buckets of “somewhat likely”, “rather unlikely”, and “not gonna worry about it”, with another dimension of “how much, if any, will my actions change things”, also focused on paths wide enough that you’re not basing it on insane multiplication of very small/large made-up numbers.
‘Avoidable’ in above toy numbers are purely that
1 - avoidable doom directly caused by AI is in fact avoided if we destroy all (relevantly capable?) AI when testing for doom.
2 - avoidable doom directly caused by humans or nature is in fact avoided by AI technology we possess when testing for doom.
Still not sure I follow. “testing for doom” is done by experiencing the doom or non-doom-yet future at some point in time, right? And we can’t test under conditions that don’t actually obtain. Or do you have some other test that works on counterfactual (or future-unknown-maybe-factual) worlds?
Yeah, the test is just if doom is experienced (and I have no counterfactual world testing, useful as that would be).