Edit: This comment is retracted. My comment is wrong, primarily because it misses the point of the post, which simply presents a usual game theory-style payoff matrix problem statement. Thanks to Tyrrell McAllister for pointing out the error, apologies to the readers. See this comment for details. (One more data point against going on a perceptual judgement at 4AM, and not double-checking own understanding before commenting on a perceived flaw in an argument. A bit of motivated procrastination also delayed reviewing Tyrrell’s response.)
Humanity has also worked out the optimal strategy S0...S9 for each utility function. But they just happen to score poorly on all of the others:
f_i(S_i) = 10 f_i(S_j) = 1 for i != j
Who is following these strategies? The only interpretation that seems to make sense is that it’s humanity in each case (is this correct?), that is S2 is the strategy that, if followed by humanity, would optimize aliens #2′s utility.
In this case, the question is what do the f_i(S_j) mean. These are expected utilities of a possible strategy, but how do you compute them? CDT, TDT and UDT would have it differently.
In any case, it’s conventional to mean by “expected utility of a possible decision” the value that you’ll be actually optimizing. With CDT, it’s computed in such a way that you two-box on Newcomb as a result, in TDT and UDT the bug is fixed and you one-box, but still by optimizing expected utility (computed differently) of the decision that you’d make as a result. Similarly for PD, where you one-box in UDT/ADT not because you take into account utilities of different agents, but because you take into account the effect on your own utility mediated by other agent’s hypothetical response to your hypothetical decision, that is you just compute your own expected utility more accurately, and still just maximize only your own utility.
Cooperation in PD of the kind TDT and UDT enable is not about helping the other, it’s about being able to take into account other’s hypothetical cooperation arising in response to your hypothetical cooperation. Altruistic agents already have their altruism as part of their own utility function, it’s a property of their values that’s abstracted away at the level where you talk about utilities and should no longer be considered at that level.
So the answer to “What should you maximize?” is, by convention, “Your own expected utility, period.” This is just what “expected utility” means (that is, where you can factor utility as expected value of a utility function over some probability distribution; otherwise you use “utility” in this role). The right question should be, “How should you compute your expected utility?”, and it can’t be answered given the setup in this post, since f_i are given as black boxes. (Alternatively, you could give a way of estimating utility as a black box, and then consider various ways of constructing an estimate of expected utility out of it.)
In this case, the question is what do the f_i(S_j) mean. These are expected utilities of a possible strategy, but how do you compute them? CDT, TDT and UDT would have it differently.
[...]
(Alternatively, you could give a way of estimating utility as a black box, and then consider various ways of constructing an estimate of expected utility out of it.)
The post calls the functions f_i “utility functions”, not “expected utility functions”. So, I take Giles to be pursuing your “alternative” approach. However, I don’t think that f_i(S_j) denotes the total utility of a state of the universe. It is just one of the terms used to compute such a total utility.
From the comments about additivity, I take f_i(S_j) to be the amount by which the utility of a universe to species i would increase if a planet following strategy j were added to it (while the strategies of all other planets remained unchanged), regardless of how or by whom that planet is added. Giles’s question, as I understand it, is, how should these “utility” terms be incorporated into an expected utility calculation? For example, what should the probability weights say is the probability that species i will produce a planet following the compromise strategy, given that we do?
The post calls the functions f_i “utility functions”, not “expected utility functions”.
(As an aside, some terminological confusion can result from there being a “utility relation” that compares lotteries, that can be represented by a “utility function” that takes lotteries as inputs, and separately expected utility representation of utility relation (or of “utility function”) that breaks it down into a probability distribution and a “utility function” in a different sense, that takes pure outcomes as inputs.)
However, I don’t think that f_i(S_j) denotes the total utility of a state of the universe. It is just one of the terms used to compute such a total utility.
Right.
From the comments about additivity, I take f_i(S_j) to be the amount by which the utility of a universe to species i would increase if a planet following strategy j were added to it (while the strategies of all other planets remained unchanged), regardless of how or by whom that planet is added.
Or, more usefully (since we can’t actually add planets), the utility function of aliens #k that takes a collection S of strategies for each of the planets under consideration (i.e. a state of the world) is
F_k (S) = sum_p f_k(S_p)
Then, the decision problem is to maximize expected value of F_0(S) by controlling S_0, a standard game theory setting. It’s underdetermined only to the extent PD is underdetermined, in that you should still defect against CooperationBots or DefectBots, etc.
I believe the issue discussed in the post doesn’t exist, and only appears to be present because of the confusion described in my comment. [Edit: I believe this no longer, see the edit to the original comment.]
(I’m actually not sure what you refer to by “that philosophical issue”, “one issue discussed here” and what you are less confident about.)
I believe the issue discussed here doesn’t exist, and only appears to be present because of the confusion I discussed in my comment.
It is not absolutely determined that finding that multiplying the probability of universe-state by the value of it is what must be done period. Another relationship between probabilities, values for states of the universe and behavior could actually be legitimate. I noted that this is an obscure philosophical question that is not intended to detract from your point.
Right; since probabilities (and expected utility axioms) break in some circumstances (for decision-theoretic purposes), expected utility of the usual kind isn’t fundamental, but its role seems to be.
(I did anticipate this objection/clarification, see the parenthetical about utility failing to factor as expectation of a utility function...)
Edit: This comment is retracted. My comment is wrong, primarily because it misses the point of the post, which simply presents a usual game theory-style payoff matrix problem statement. Thanks to Tyrrell McAllister for pointing out the error, apologies to the readers. See this comment for details. (One more data point against going on a perceptual judgement at 4AM, and not double-checking own understanding before commenting on a perceived flaw in an argument. A bit of motivated procrastination also delayed reviewing Tyrrell’s response.)
Who is following these strategies? The only interpretation that seems to make sense is that it’s humanity in each case (is this correct?), that is S2 is the strategy that, if followed by humanity, would optimize aliens #2′s utility.
In this case, the question is what do the f_i(S_j) mean. These are expected utilities of a possible strategy, but how do you compute them? CDT, TDT and UDT would have it differently.
In any case, it’s conventional to mean by “expected utility of a possible decision” the value that you’ll be actually optimizing. With CDT, it’s computed in such a way that you two-box on Newcomb as a result, in TDT and UDT the bug is fixed and you one-box, but still by optimizing expected utility (computed differently) of the decision that you’d make as a result. Similarly for PD, where you one-box in UDT/ADT not because you take into account utilities of different agents, but because you take into account the effect on your own utility mediated by other agent’s hypothetical response to your hypothetical decision, that is you just compute your own expected utility more accurately, and still just maximize only your own utility.
Cooperation in PD of the kind TDT and UDT enable is not about helping the other, it’s about being able to take into account other’s hypothetical cooperation arising in response to your hypothetical cooperation. Altruistic agents already have their altruism as part of their own utility function, it’s a property of their values that’s abstracted away at the level where you talk about utilities and should no longer be considered at that level.
So the answer to “What should you maximize?” is, by convention, “Your own expected utility, period.” This is just what “expected utility” means (that is, where you can factor utility as expected value of a utility function over some probability distribution; otherwise you use “utility” in this role). The right question should be, “How should you compute your expected utility?”, and it can’t be answered given the setup in this post, since f_i are given as black boxes. (Alternatively, you could give a way of estimating utility as a black box, and then consider various ways of constructing an estimate of expected utility out of it.)
[...]
The post calls the functions f_i “utility functions”, not “expected utility functions”. So, I take Giles to be pursuing your “alternative” approach. However, I don’t think that f_i(S_j) denotes the total utility of a state of the universe. It is just one of the terms used to compute such a total utility.
From the comments about additivity, I take f_i(S_j) to be the amount by which the utility of a universe to species i would increase if a planet following strategy j were added to it (while the strategies of all other planets remained unchanged), regardless of how or by whom that planet is added. Giles’s question, as I understand it, is, how should these “utility” terms be incorporated into an expected utility calculation? For example, what should the probability weights say is the probability that species i will produce a planet following the compromise strategy, given that we do?
You are right, I retract my comment.
(As an aside, some terminological confusion can result from there being a “utility relation” that compares lotteries, that can be represented by a “utility function” that takes lotteries as inputs, and separately expected utility representation of utility relation (or of “utility function”) that breaks it down into a probability distribution and a “utility function” in a different sense, that takes pure outcomes as inputs.)
Right.
Or, more usefully (since we can’t actually add planets), the utility function of aliens #k that takes a collection S of strategies for each of the planets under consideration (i.e. a state of the world) is
F_k (S) = sum_p f_k(S_p)
Then, the decision problem is to maximize expected value of F_0(S) by controlling S_0, a standard game theory setting. It’s underdetermined only to the extent PD is underdetermined, in that you should still defect against CooperationBots or DefectBots, etc.
I’m a little less confident with the period for ‘expected’ but that is a whole different philosophical issue to the one important here!
I believe the issue discussed in the post doesn’t exist, and only appears to be present because of the confusion described in my comment. [Edit: I believe this no longer, see the edit to the original comment.]
(I’m actually not sure what you refer to by “that philosophical issue”, “one issue discussed here” and what you are less confident about.)
It is not absolutely determined that finding that multiplying the probability of universe-state by the value of it is what must be done period. Another relationship between probabilities, values for states of the universe and behavior could actually be legitimate. I noted that this is an obscure philosophical question that is not intended to detract from your point.
Right; since probabilities (and expected utility axioms) break in some circumstances (for decision-theoretic purposes), expected utility of the usual kind isn’t fundamental, but its role seems to be.
(I did anticipate this objection/clarification, see the parenthetical about utility failing to factor as expectation of a utility function...)