Hmm, interesting. I think human cloning is an imperfect analogy because the only real reason to do it is to impress your friends, so if everyone coordinates on being scornful towards the first person to do human cloning (rather than being impressed), then there’s no more personal benefit to cheating. By contrast, with an AGI, there would be the hope that you’ll actually solve the safety problems, and then get tons of money and power and respect.
Biological weapons is maybe a better example, but not an especially encouraging one: as many as 8 countries may have secret bio-weapons programs, including North Korea. Maybe one could make an argument that there’s a taboo against using bio-weapons, as opposed to merely stockpiling them? Likewise, the taboo against using nuclear weapons was not successfully turned into a taboo against countries starting new nuclear weapons programs. Maybe it’s hard to get riled up against someone doing something that is not purposely aggressive? I don’t know. I can’t think of a great historical analogy.
There’s also the issue that there’s not too many actors who have any reason to start a bio-weapons programs, and the ability to do so without getting shut down. Really just secret military labs. Whereas in the worst case, many orders of magnitude more people would be willing and able to start doing illegal AGI experiments without the authorities realizing it.
I agree that AGI is more omni-use than bioweapons and thus will be harder to get people not to develop and use. I think our prospects look pretty bleak in this scenario, but it’s not completely hopeless.
For human cloning, what I had in mind was a nation cloning its smartest individuals for the purpose of having better science/tech. Think of what the US could have accomplished if they had 10,000 Von Neumanns instead of 1.
Hmm, interesting. I think human cloning is an imperfect analogy because the only real reason to do it is to impress your friends, so if everyone coordinates on being scornful towards the first person to do human cloning (rather than being impressed), then there’s no more personal benefit to cheating. By contrast, with an AGI, there would be the hope that you’ll actually solve the safety problems, and then get tons of money and power and respect.
Biological weapons is maybe a better example, but not an especially encouraging one: as many as 8 countries may have secret bio-weapons programs, including North Korea. Maybe one could make an argument that there’s a taboo against using bio-weapons, as opposed to merely stockpiling them? Likewise, the taboo against using nuclear weapons was not successfully turned into a taboo against countries starting new nuclear weapons programs. Maybe it’s hard to get riled up against someone doing something that is not purposely aggressive? I don’t know. I can’t think of a great historical analogy.
There’s also the issue that there’s not too many actors who have any reason to start a bio-weapons programs, and the ability to do so without getting shut down. Really just secret military labs. Whereas in the worst case, many orders of magnitude more people would be willing and able to start doing illegal AGI experiments without the authorities realizing it.
I agree that AGI is more omni-use than bioweapons and thus will be harder to get people not to develop and use. I think our prospects look pretty bleak in this scenario, but it’s not completely hopeless.
For human cloning, what I had in mind was a nation cloning its smartest individuals for the purpose of having better science/tech. Think of what the US could have accomplished if they had 10,000 Von Neumanns instead of 1.