Okay, so let’s use some concrete examples to see if I understand this abstract correctly.
You say that the chain of causation is from fitness (natural selection) ---> outcomes ---> activities
So, for example: reproduction ---> sex ---> flirting/dancing/tattooing/money/bodybuilding.
Natural selection programs us to have a terminal goal of reproduction. HOWEVER, it would be a bad idea for an AI to conclude, “OK, humans want reproduction? I’ll give them reproduction. I’ll help the humans reproduce 10 quadrillion people. The more reproduction, the better, right?”
The AI would need to look ahead and see, “OK, the programmed goal of reproduction has caused humans to prefer a specific outcome, sex, which tended to lead to reproduction in the original (ancestral) programming environment, but might no longer do so. Humans have, in other words, come to cherish sex as a terminal goal in its own right through their affective responses to its reward payoff. So, let’s make sure that humans can have as much sex as possible, regardless of whether it will really lead to more reproduction. That will make humans happy, right?”
But then the AI would need to look ahead one step further and see, “OK, the preferred outcome of sex has, in turn, caused humans to enjoy, for their own sake, specific activities that, in the experience and learning of particular humans in their singular lifetimes (we are no longer talking about instinctual programming here, but rather culture), has tended in their particular circumstances, to lead to this preferred outcome of sex. In one culture, humans found that flirting tended to lead to sex, and so they formed a positive affective connotation with flirting and came to view flirting as a terminal goal in its own right. In another culture, dancing appeared to be the key to sex, and so dancing became a terminal goal in that culture. In other cultures, bodybuilding, accumulation of money, etc. seemed to lead to sex, and so humans became attached to those activities for their own sake, even beyond the extent to which those activities continued to lead to more sex. So really, the way to make these humans happy would be to pay attention to their particular cultures and psychologies and see which activities they have come to develop a positive affective bond with...because THESE activities have become the humans’ new conscious terminal goals. So we AI robots should work hard to make it easy for the humans to engage in as much flirting/dancing/bodybuilding/money accumulation/etc. as possible.”
Would this be an accurate example of what you are talking about?
The model is currently mostly focused on modeling the values of just a single person. I do talk a bit about how to figure out the values of entire societies of people, but that’s mostly outside the scope of this paper.
Evolution’s “original intent” doesn’t really matter for its own sake. That sexual attraction has evolved for the purpose of furthering reproduction isn’t considered particularly important; what’s important is just that humans experience sexual attraction and find it pleasant (or not, as the case may be).
Those things said, your final step does sound reasonably close to the kind of thing I was thinking of. We can look at some particular individual, note that a combination of the surrounding culture and their own sexuality made them try out flirting and dancing and find both rewarding, and then come to value those things for their own sake. And conclude that their ideal future would probably be likely to include fair amounts of both.
Though of course there are also all kinds of questions about, for example, exactly how rewarding and enjoyable do they find those things. Maybe someone feels positive about the concept of being the kind of person who’d enjoy dance, but isn’t actually the kind of person who’d enjoy dance. Resolving that kind of a conflict would probably either mean helping them to learn to enjoy dance, or to help them give up the ideal of needing to be that kind of a person. The correct action would depend on exactly how deeply their reasons for not enjoying dance ran, and on what their other values were.
Also it’s possible that upon an examination of the person’s psychology, the AI would conclude that while they did enjoy flirting and dance, there were other things that they would enjoy more—either right now, or given enough time and effort. The AI might then work to create a situation where they could focus more on those more rewarding things.
Now with these kinds of questions there is the issue of exactly what kinds of interventions are allowed by the AI. After all, the most effective way of making somebody have maximally rewarding experiences would be to rewire their brain to always receive maximal reward. Here’s where I take a page of Paul Christiano’s book and his suggestion of approval-directed agents, and propose that the AI is only allowed to do the kinds of things to the human that the human’s current values would approve of. So if the human doesn’t want to have their brain rewired, but is okay with the AI suggesting new kinds of activities that they might enjoy, then that’s what happens. (Of course, “only doing the kinds of things that the human’s current values would approve of” is really vague and hand-wavy at this point, and would need to be defined a lot better.)
Okay, so let’s use some concrete examples to see if I understand this abstract correctly.
You say that the chain of causation is from fitness (natural selection) ---> outcomes ---> activities
So, for example: reproduction ---> sex ---> flirting/dancing/tattooing/money/bodybuilding.
Natural selection programs us to have a terminal goal of reproduction. HOWEVER, it would be a bad idea for an AI to conclude, “OK, humans want reproduction? I’ll give them reproduction. I’ll help the humans reproduce 10 quadrillion people. The more reproduction, the better, right?”
The AI would need to look ahead and see, “OK, the programmed goal of reproduction has caused humans to prefer a specific outcome, sex, which tended to lead to reproduction in the original (ancestral) programming environment, but might no longer do so. Humans have, in other words, come to cherish sex as a terminal goal in its own right through their affective responses to its reward payoff. So, let’s make sure that humans can have as much sex as possible, regardless of whether it will really lead to more reproduction. That will make humans happy, right?”
But then the AI would need to look ahead one step further and see, “OK, the preferred outcome of sex has, in turn, caused humans to enjoy, for their own sake, specific activities that, in the experience and learning of particular humans in their singular lifetimes (we are no longer talking about instinctual programming here, but rather culture), has tended in their particular circumstances, to lead to this preferred outcome of sex. In one culture, humans found that flirting tended to lead to sex, and so they formed a positive affective connotation with flirting and came to view flirting as a terminal goal in its own right. In another culture, dancing appeared to be the key to sex, and so dancing became a terminal goal in that culture. In other cultures, bodybuilding, accumulation of money, etc. seemed to lead to sex, and so humans became attached to those activities for their own sake, even beyond the extent to which those activities continued to lead to more sex. So really, the way to make these humans happy would be to pay attention to their particular cultures and psychologies and see which activities they have come to develop a positive affective bond with...because THESE activities have become the humans’ new conscious terminal goals. So we AI robots should work hard to make it easy for the humans to engage in as much flirting/dancing/bodybuilding/money accumulation/etc. as possible.”
Would this be an accurate example of what you are talking about?
So to first note a few things:
The model is currently mostly focused on modeling the values of just a single person. I do talk a bit about how to figure out the values of entire societies of people, but that’s mostly outside the scope of this paper.
Evolution’s “original intent” doesn’t really matter for its own sake. That sexual attraction has evolved for the purpose of furthering reproduction isn’t considered particularly important; what’s important is just that humans experience sexual attraction and find it pleasant (or not, as the case may be).
Those things said, your final step does sound reasonably close to the kind of thing I was thinking of. We can look at some particular individual, note that a combination of the surrounding culture and their own sexuality made them try out flirting and dancing and find both rewarding, and then come to value those things for their own sake. And conclude that their ideal future would probably be likely to include fair amounts of both.
Though of course there are also all kinds of questions about, for example, exactly how rewarding and enjoyable do they find those things. Maybe someone feels positive about the concept of being the kind of person who’d enjoy dance, but isn’t actually the kind of person who’d enjoy dance. Resolving that kind of a conflict would probably either mean helping them to learn to enjoy dance, or to help them give up the ideal of needing to be that kind of a person. The correct action would depend on exactly how deeply their reasons for not enjoying dance ran, and on what their other values were.
Also it’s possible that upon an examination of the person’s psychology, the AI would conclude that while they did enjoy flirting and dance, there were other things that they would enjoy more—either right now, or given enough time and effort. The AI might then work to create a situation where they could focus more on those more rewarding things.
Now with these kinds of questions there is the issue of exactly what kinds of interventions are allowed by the AI. After all, the most effective way of making somebody have maximally rewarding experiences would be to rewire their brain to always receive maximal reward. Here’s where I take a page of Paul Christiano’s book and his suggestion of approval-directed agents, and propose that the AI is only allowed to do the kinds of things to the human that the human’s current values would approve of. So if the human doesn’t want to have their brain rewired, but is okay with the AI suggesting new kinds of activities that they might enjoy, then that’s what happens. (Of course, “only doing the kinds of things that the human’s current values would approve of” is really vague and hand-wavy at this point, and would need to be defined a lot better.)