Hm, I’m not sure about Mere Goodness, I read the sequences soon after they were finished, so I don’t much remember which concepts were where. There is a sequence post titled Terminal Values and Instrumental Values, though it mostly seems to be emphasizing that both things exist and are different, saving the rest of the content for other posts.
Gotcha that makes sense. I read them like 10 years ago and also don’t remember but am going to skim through Mere Goodness again.
You don’t remember Eliezer arguing against the idea that terminal goals are arbitrary though right? My memory is that he pushes you to introspect and makes arguments like “Are you really sure that is what your terminal value actually is?” but never goes as far as saying that they can be correct or incorrect.
I do remember a bunch of content around that, yeah. And I would agree that terminal goals are arbitrary in the sense that they could be anything. But, for any given agent/organism/”thing that wants stuff”, there will be a fact-of-the-matter of what terminals goals got instantiated inside that thing.
There are also a few separate but related and possibly confusing facts;
The process of evolution will tend to produce organisms that have certain kinds of terminal goals instantiated inside them.
Empirically, humans happen to have a huge overlap in their terminal goals (including the terminal goal that other beings have their terminal goals satisfied).
If there are a bunch of roughly equally-capable agents around, then it maximizes your own utility (= terminal goals) to do a lot of game-theoretic cooperation with them.
I do remember a bunch of content around that, yeah. And I would agree that terminal goals are arbitrary in the sense that they could be anything.
I see. Thanks for clarifying.
But, for any given agent/organism/”thing that wants stuff”, there will be a fact-of-the-matter of what terminals goals got instantiated inside that thing.
Yeah I agree and think that those are important points.
Hm, I’m not sure about Mere Goodness, I read the sequences soon after they were finished, so I don’t much remember which concepts were where. There is a sequence post titled Terminal Values and Instrumental Values, though it mostly seems to be emphasizing that both things exist and are different, saving the rest of the content for other posts.
Gotcha that makes sense. I read them like 10 years ago and also don’t remember but am going to skim through Mere Goodness again.
You don’t remember Eliezer arguing against the idea that terminal goals are arbitrary though right? My memory is that he pushes you to introspect and makes arguments like “Are you really sure that is what your terminal value actually is?” but never goes as far as saying that they can be correct or incorrect.
I do remember a bunch of content around that, yeah. And I would agree that terminal goals are arbitrary in the sense that they could be anything. But, for any given agent/organism/”thing that wants stuff”, there will be a fact-of-the-matter of what terminals goals got instantiated inside that thing.
There are also a few separate but related and possibly confusing facts;
The process of evolution will tend to produce organisms that have certain kinds of terminal goals instantiated inside them.
Empirically, humans happen to have a huge overlap in their terminal goals (including the terminal goal that other beings have their terminal goals satisfied).
If there are a bunch of roughly equally-capable agents around, then it maximizes your own utility (= terminal goals) to do a lot of game-theoretic cooperation with them.
I see. Thanks for clarifying.
Yeah I agree and think that those are important points.