One possible situation is when non-mental AGI creates a full model of a human mind, may be by scanning, and thus become partly “mental”. It is more or less inevitable as it is impossible to extract human values without creating some model of the human mind.
Or, saying it in other words, model of human values without a model of human mind is doomed to be wrong, because it will be based on many hidden assumptions implied by ideas like “humans have constant set of preferences, and act according to it”.
Interesting, didn’t think about that. Seems to suggest an alternative way to argue for AGI with minds even if we try to create them so that they don’t have minds!
One possible situation is when non-mental AGI creates a full model of a human mind, may be by scanning, and thus become partly “mental”. It is more or less inevitable as it is impossible to extract human values without creating some model of the human mind.
Or, saying it in other words, model of human values without a model of human mind is doomed to be wrong, because it will be based on many hidden assumptions implied by ideas like “humans have constant set of preferences, and act according to it”.
Interesting, didn’t think about that. Seems to suggest an alternative way to argue for AGI with minds even if we try to create them so that they don’t have minds!