The Omohundro quote sounds like what humans do. If humans do it, machines might well do it too.
The Yudkowsky quote seems more speculative. It assumes that values are universal, and don’t need to adapt to local circumstances. This would be in contrast to what has happened in evolution so far—where there are many creatures with different niches and the organisms (and their values) adapt to the niches.
I pretty-much agree with the spirit of the Omohundro quote. It usually helps you meet your goals if you know what they are. That’s unlikely to be a feature specific to humans, and it is likely to apply to goal-directed agents above a certain threshold. (too-simple agents may not get much out of it). Of course, agents might start out with a clear representation of their goals—but if they don’t, they are likely to want one, as a basic component of the task of modelling themselves.
The Omohundro quote sounds like what humans do. If humans do it, machines might well do it too.
The Yudkowsky quote seems more speculative. It assumes that values are universal, and don’t need to adapt to local circumstances. This would be in contrast to what has happened in evolution so far—where there are many creatures with different niches and the organisms (and their values) adapt to the niches.
Yeah, that’s why I called it “anthropomorphizing” in the post. It’s always been a strikingly unsuccessful way to make predictions about computers.
I pretty-much agree with the spirit of the Omohundro quote. It usually helps you meet your goals if you know what they are. That’s unlikely to be a feature specific to humans, and it is likely to apply to goal-directed agents above a certain threshold. (too-simple agents may not get much out of it). Of course, agents might start out with a clear representation of their goals—but if they don’t, they are likely to want one, as a basic component of the task of modelling themselves.