If preferences cannot be “learned” from a full physical model of a brain, that isn’t it the truth that a human theory of mind is unlearnable as well?
I don’t see a good reason to privilege having the original copy of a brain here. If you’re willing to grant me that a brain can be copied, there is nothing the brain-holder should be unable to learn that the brain-user knows.
If preferences cannot be “learned” from a full physical model of a brain, that isn’t it the truth that a human theory of mind is unlearnable as well?
I don’t see a good reason to privilege having the original copy of a brain here. If you’re willing to grant me that a brain can be copied, there is nothing the brain-holder should be unable to learn that the brain-user knows.
See comment here: https://www.lesswrong.com/posts/kMJxwCZ4mc9w4ezbs/how-an-alien-theory-of-mind-might-be-unlearnable?commentId=iPitpgNxwJH2e98CK