I think (and I hope) that something like “maximize positive experiences of sentient entities” could actually be a convergent goal of any AI that are capable of reflecting on these questions. I don’t think that humans just gravitate towards this kind of utility maximization because they evolved some degree of pro-sociality. Instead, something like this seems like it’s the only thing inherently worth striving to, in the absence of any other set of values or goals.
The grabby aliens type scenario in the first parable seems like the biggest threat to the idea that an AI might discover on its own that it wants to maximize fun. Or at least it would decide that it shouldn’t maximize fun while ignoring self-defense.
Regarding the second parable, I don’t find it any more plausible that an AI would want to maximize complex puppet shows in a failed attempt to recreate human-like values than that it would want to maximize paperclips. Both scenarios assume that the AI is ultimately not able or willing to reflect on its own goals and to change them to something that’s more meaningful in some absolute sense.
I think there is at least a good chance that an AI will be able to self-reflect and to think about the same questions that we are pondering here, and then make good decisions based on that, even if it doesn’t particularly care about ice cream, or even about humans.
I think (and I hope) that something like “maximize positive experiences of sentient entities” could actually be a convergent goal of any AI that are capable of reflecting on these questions. I don’t think that humans just gravitate towards this kind of utility maximization because they evolved some degree of pro-sociality. Instead, something like this seems like it’s the only thing inherently worth striving to, in the absence of any other set of values or goals.
The grabby aliens type scenario in the first parable seems like the biggest threat to the idea that an AI might discover on its own that it wants to maximize fun. Or at least it would decide that it shouldn’t maximize fun while ignoring self-defense.
Regarding the second parable, I don’t find it any more plausible that an AI would want to maximize complex puppet shows in a failed attempt to recreate human-like values than that it would want to maximize paperclips. Both scenarios assume that the AI is ultimately not able or willing to reflect on its own goals and to change them to something that’s more meaningful in some absolute sense.
I think there is at least a good chance that an AI will be able to self-reflect and to think about the same questions that we are pondering here, and then make good decisions based on that, even if it doesn’t particularly care about ice cream, or even about humans.