How about the distinction between (A) “An AGI kills every human, and the people who turned on the AGI didn’t want that to happen” versus (B) “An AGI kills every human, and the people who turned on the AGI did want that to happen”?
I think the misuse vs. accident dichotomy is clearer when you don’t focus exclusively on “AGI kills every human” risks. (E.g., global totalitarianism risks strike me as small but non-negligible if we solve the alignment problem. Larger are risks that fall short of totalitarianism but still involve non-morally-humble developers damaging humanity’s long-term potential.)
The dichotomy is really just “AGI does sufficiently bad stuff, and the developers intended this” versus “AGI does sufficiently bad stuff, and the developers didn’t intend this”. The terminology might be non-ideal, but the concepts themselves are very natural.
It’s basically the same concept as “conflict disaster” versus “mistake disaster”. If something falls into both category to a significant extent (e.g., someone tries to become dictator but fails to solve alignment), then it goes in the “accident risk” bucket, because it doesn’t actually matter what you wanted to do with the AI if you’re completely unable to achieve that goal. The dynamics and outcome will end up looking basically the same as other accidents.
I think the misuse vs. accident dichotomy is clearer when you don’t focus exclusively on “AGI kills every human” risks. (E.g., global totalitarianism risks strike me as small but non-negligible if we solve the alignment problem. Larger are risks that fall short of totalitarianism but still involve non-morally-humble developers damaging humanity’s long-term potential.)
The dichotomy is really just “AGI does sufficiently bad stuff, and the developers intended this” versus “AGI does sufficiently bad stuff, and the developers didn’t intend this”. The terminology might be non-ideal, but the concepts themselves are very natural.
It’s basically the same concept as “conflict disaster” versus “mistake disaster”. If something falls into both category to a significant extent (e.g., someone tries to become dictator but fails to solve alignment), then it goes in the “accident risk” bucket, because it doesn’t actually matter what you wanted to do with the AI if you’re completely unable to achieve that goal. The dynamics and outcome will end up looking basically the same as other accidents.
By “intend” do you mean that they sought that outcome / selected for it?
Or merely that it was a known or predictable outcome of their behavior?
I think “unintentional” would already probably be a better term in most cases.