Reasonable attempt, but two issues with this scenario as a current-techniques thing:
We don’t have techniques to create faithful copies of a benevolent human, especially ones which stay humanlike as you move off-distribution
A huge number of humanlike minds with an initially friendly template would be extremely far off distribution, it’s likely that memetic effects inside such a population would be pretty extreme. I’m not going to say 90% doom, but only because maybe novel techniques could be developed to stabilize the otherwise extremely chaotic system you actually get if you solve #1 with a new technique.
I certainly agree with your first point, but I don’t think it is relevant. I specifically say in footnote 3: “I’m aware that this doesn’t fall within ‘remotely like current techniques’, bear with me.” The part with the human ems is just to establish a a comparison point used in later arguments, not actually part of the proposed counter-example.
In your second point, do you argue that if we could create literal full ems of benevolent humans, you still expect their society to eventually kill everyone due to unpredictable memetic effects? If this is people’s opinion, I think it would be good to explicitly state it, because I think this would be an interesting disagreement between different people. I personally feel pretty confident that if you created an army of ems from me, we wouldn’t kill all humans, especially if we implement some reasonable precautionary measures discussed under my point (2).
Reasonable attempt, but two issues with this scenario as a current-techniques thing:
We don’t have techniques to create faithful copies of a benevolent human, especially ones which stay humanlike as you move off-distribution
A huge number of humanlike minds with an initially friendly template would be extremely far off distribution, it’s likely that memetic effects inside such a population would be pretty extreme. I’m not going to say 90% doom, but only because maybe novel techniques could be developed to stabilize the otherwise extremely chaotic system you actually get if you solve #1 with a new technique.
I certainly agree with your first point, but I don’t think it is relevant. I specifically say in footnote 3: “I’m aware that this doesn’t fall within ‘remotely like current techniques’, bear with me.” The part with the human ems is just to establish a a comparison point used in later arguments, not actually part of the proposed counter-example.
In your second point, do you argue that if we could create literal full ems of benevolent humans, you still expect their society to eventually kill everyone due to unpredictable memetic effects? If this is people’s opinion, I think it would be good to explicitly state it, because I think this would be an interesting disagreement between different people. I personally feel pretty confident that if you created an army of ems from me, we wouldn’t kill all humans, especially if we implement some reasonable precautionary measures discussed under my point (2).