Ian: the issue isn’t whether it could determine what humans want, but whether it would care. That’s what Eliezer was talking about with the “difference between chess pieces on white squares and chess pieces on black squares” analogy. There are infinitely many computable quantities that don’t affect your utility function at all. The important job in FAI is determining how to create an intelligence that will care about the things we care about.
Certainly it’s necessary for such an intelligence to be able to compute it, but it’s certainly not sufficient.
It’s not so much the killing that’s an issue as the potential mistreatment. If you want to discover whether people like being burned, “Simulate EY, but on fire, and see how he responds” is just as bad of an option as “Duplicate EY, ignite him, and see how he responds”. This is a tool that should be used sparingly at best and that a successful AI shouldn’t need.