Yes, I agree with that. I’m not claiming that knowing about it stops you from wanting ice cream.
I’m claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want “inclusive genetic fitness” rather than wanting ice cream.
i.e—we wouldn’t want ice cream at all but reason from first principles what we should eat based on fitness.
Knowing about “inclusive genetic fitness” does not stop you from wanting ice cream.
For superhuman AIs, knowing about human values won’t necessarily make them care.
Yes, I agree with that. I’m not claiming that knowing about it stops you from wanting ice cream.
I’m claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want “inclusive genetic fitness” rather than wanting ice cream.
i.e—we wouldn’t want ice cream at all but reason from first principles what we should eat based on fitness.