The first concern seems like a much smaller risk that the one we currently face from unaligned AI. To be clear, I’m suggesting emulations of a relatively large number of people (more than 10, at least once the technology has been well tested, and eventually perhaps everyone). If some of them turn out be evil sociopaths, the others will just have to band together and enforce norms, exactly like we do now.
The second concern sounds like gradual disempowerment to me. However, I think there are a lot of ways for Corporation A to win. Perhaps Corporation B is regulated out of existence—reckless modifications should violate some sort of human alignment code. Perharps we learn how to recursively self improve as emulations, in such a way that the alignment tax is near 0, and then just ensure that initial conditions modestly favor Corporation A (most companies adopt reasonable standards, and over time control most of the resources). Or perhaps corporate power is drastically reduced and emulations are able to coordinate once their intelligence is sufficiently boosted. Or perhaps a small team of early emulations performs a pivotal act. Basically, I think this is something our emulations can figure out.
The first concern seems like a much smaller risk that the one we currently face from unaligned AI. To be clear, I’m suggesting emulations of a relatively large number of people (more than 10, at least once the technology has been well tested, and eventually perhaps everyone). If some of them turn out be evil sociopaths, the others will just have to band together and enforce norms, exactly like we do now.
The second concern sounds like gradual disempowerment to me. However, I think there are a lot of ways for Corporation A to win. Perhaps Corporation B is regulated out of existence—reckless modifications should violate some sort of human alignment code. Perharps we learn how to recursively self improve as emulations, in such a way that the alignment tax is near 0, and then just ensure that initial conditions modestly favor Corporation A (most companies adopt reasonable standards, and over time control most of the resources). Or perhaps corporate power is drastically reduced and emulations are able to coordinate once their intelligence is sufficiently boosted. Or perhaps a small team of early emulations performs a pivotal act. Basically, I think this is something our emulations can figure out.