It’s not so much the killing that’s an issue as the potential mistreatment. If you want to discover whether people like being burned, “Simulate EY, but on fire, and see how he responds” is just as bad of an option as “Duplicate EY, ignite him, and see how he responds”. This is a tool that should be used sparingly at best and that a successful AI shouldn’t need.
It’s not so much the killing that’s an issue as the potential mistreatment. If you want to discover whether people like being burned, “Simulate EY, but on fire, and see how he responds” is just as bad of an option as “Duplicate EY, ignite him, and see how he responds”. This is a tool that should be used sparingly at best and that a successful AI shouldn’t need.