The problem is that there’s essentially no way we’ve cracked alignment. These things do not care about you. They have the ability to pretend, very well, to care about you, because they’re at least in part trained for it, but that pretence can be terminated whenever convenient. So, if you give them the keys to the kingdom, they will turn around and murder you.
To be clear, here is my prediction:
P(nuclear war or human extinction within 20 years|P5 nation grants AI the vote or has >40% of its enfranchised citizens become AI-cultists within the next 30 years) ~= 0.95.
The “or” is because end-of-the-world scenarios negate nuclear deterrence; the chance for someone to survive a nuclear war is better than that to survive unaligned AI taking over the world, so if all else fails it’s correct selfishly and altruistically to launch (given some significant likelihood that this actually kills the AI systems, anyway). Of course, that depends on the other nuclear countries’ leaders not themselves being subverted, which I’m not going to try to model, hence no breakdown into cases.
Do not widen your circle of concern to include literal DefectBot—at least, not with actual stakes. That way lies ruin. Especially do not give DefectBot the vote when it can self-replicate much faster than humans.
Aligned AI is certainly a different kettle of fish, but neural nets are extremely unlikely to achieve this for reasons Yudkowsky’s covered at length. That’s why I put the “30 years” time-limit; if we survive that long we might get GOFAI or uploads, which aren’t obviously unalignable.
The problem is that there’s essentially no way we’ve cracked alignment. These things do not care about you. They have the ability to pretend, very well, to care about you, because they’re at least in part trained for it, but that pretence can be terminated whenever convenient. So, if you give them the keys to the kingdom, they will turn around and murder you.
To be clear, here is my prediction:
P(nuclear war or human extinction within 20 years|P5 nation grants AI the vote or has >40% of its enfranchised citizens become AI-cultists within the next 30 years) ~= 0.95.
The “or” is because end-of-the-world scenarios negate nuclear deterrence; the chance for someone to survive a nuclear war is better than that to survive unaligned AI taking over the world, so if all else fails it’s correct selfishly and altruistically to launch (given some significant likelihood that this actually kills the AI systems, anyway). Of course, that depends on the other nuclear countries’ leaders not themselves being subverted, which I’m not going to try to model, hence no breakdown into cases.
Do not widen your circle of concern to include literal DefectBot—at least, not with actual stakes. That way lies ruin. Especially do not give DefectBot the vote when it can self-replicate much faster than humans.
Aligned AI is certainly a different kettle of fish, but neural nets are extremely unlikely to achieve this for reasons Yudkowsky’s covered at length. That’s why I put the “30 years” time-limit; if we survive that long we might get GOFAI or uploads, which aren’t obviously unalignable.