I’m confused since as a buyer if I believed the seller could predict with probability .75 I would flip a fair coin to decide which box to take meaning that the seller couldn’t predict with probability .75. If I can’t randomize to pick a box I’m not sure how to fit in what you are doing to standard game theory (which I teach).
If standard game theory has nothing to say about what to do in situations where you don’t have access to an unpredictable randomization mechanism, so much the worse for standard game theory, I say!
I thought the ability to deploy mixed strategies was a pretty standard part of CDT. Is this not the case, or are you considering a non-standard formulation of CDT?
I think some people may have their pet theories which they call CDT and which require randomization. But CDT as it is usually/traditionally described doesn’t ever insist on randomizing (unless randomizing has a positive causal effect). In this particular case, even if a randomization device were made available, CDT would either uniquely favor one of the boxes or be indifferent between all distributions over {buy box B1,buy box B2}. Compare Section IV.1 of the paper.
What you’re referring to are probably so-called ratificationist variants of CDT. These would indeed require randomizing 50-50 between the two boxes. But one can easily construct scenarios which trip these theories up. For example, the seller could put no money in any box if she predicts that the buyer will randomize. Then no distribution is ratifiable. See Section IV.4 for a discussion of Ratificationism.
For example, the seller could put no money in any box if she predicts that the buyer will randomize.
This is a bit unsatisfying, because in my view of decision theory you don’t get to predict things like “the agent will randomize” or “the agent will take one box but feel a little wistful about it” and so on. This is unfair, in the same way as predicting that “the agent will use UDT” and punishing for it is unfair. No, you just predict the agent’s output. Or if the agent can randomize, you can sample (as many times as you like, but finitely many) from the distribution of the agent’s output. A bit more on this here, though the post got little attention.
in my view of decision theory you don’t get to predict things like “the agent will randomize”
Why not? You surely agree that sometimes people can in fact predict such things. So your objection must be that it’s unfair when they do and that it’s not a strike against a decision theory if it causes you to get money-pumped in those situations. Well… why? Seems pretty bad to me, especially since some extremely high-stakes real-world situations our AIs might face will be of this type.
Sure, and sometimes people can predict things like “the agent will use UDT” and use that to punish the agent. But this kind of prediction is “unfair” because it doesn’t lead to an interesting decision theory—you can punish any decision theory that way. So to me the boundaries of “fair” and “unfair” are also partly about mathematical taste and promising-ness, not just what will lead to a better tank and such.
Right, that kind of prediction is unfair because it doesn’t lead to an interesting decision theory… but I asked why you don’t get to predict things like “the agent will randomize.” All sorts of interesting decision theory comes out of considering situations where you do get to predict such things. (Besides, such situations are important in real life.)
I might suggest “not interesting” rather than “not fair” as the complaint. One can image an Omega that leaves the box empty if the player is unpredictable, or if the player doesn’t rigorously follow CDT, or just always leaves it empty regardless. But there’s no intuition pump that it drives, and no analysis of why a formalization would or wouldn’t get the right answer.
When I’m in challenge-the-hypothetical mode, I defend CDT by making the agent believe Omega cheats. It’s a trick box that changes contents AFTER the agent chooses, BEFORE the contents are revealed. This is much higher probability to any rational agent than mind-reading or extreme predictability.
On the more philosophical points. My position is perhaps similar to Daniel K’s. But anyway...
Of course, I agree that problems that punish the agent for using a particular theory (or using float multiplication or feeling a little wistful or stuff like that) are “unfair”/”don’t lead to interesting theory”. (Perhaps more precisely, I don’t think our theory needs to give algorithms that perform optimally in such problems in the way I want my theory to “perform optimally” Newcomb’s problem. Maybe we should still expect our theory to say something about them, in the way that causal decision theorists feel like CDT has interesting/important/correct things to say about Newcomb’s problem, despite Newcomb’s problem being designed to (unfairly, as they allege) reward non-CDT agents.)
But I don’t think these are particularly similar to problems with predictions of the agent’s distribution over actions. The distribution over actions is behavioral, whereas performing floating point operations or whatever is not. When randomization is allowed, the subject of your choice is which distribution over actions you play. So to me, which distribution over actions you choose in a problem with randomization allowed, is just like the question of which action you take when randomization is not allowed. (Of course, if you randomize to determine which action’s expected utility to calculate first, but this doesn’t affect what you do in the end, then I’m fine with not allowing this to affect your utility, because it isn’t behavioral.)
I also don’t think this leads to uninteresting decision theory. But I don’t know how to argue for this here, other than by saying that CDT, EDT, UDT, etc. don’t really care whether they choose from/rank a set of distributions or a set of three discrete actions. I think ratificationism-type concepts are the only ones that break when allowing discontinuous dependence on the chosen distribution and I don’t find these very plausible anyway.
To be honest, I don’t understand the arguments against predicting distributions and predicting actions that you give in that post. I’ll write a comment on this to that post.
No, I don’t think so. Take the class of problems. The agent can pick any distribution over actions. The final payoff is determined only as a function of the implemented action and some finite number of samples generated by Omega from that distribution. Note that the expectation is continuous in the distribution chosen. It can therefore be shown (using e.g. Kakutani’s fixed-point theorem) that there is always at least one ratifiable distribution. See Theorem 3 at https://users.cs.duke.edu/~ocaspar/NDPRL.pdf .
(Note that the above is assuming the agent maximizes expected vNM utility. If, e.g., the agent maximizes some lexical utility function, then the predictor can just take, say, two samples and if they differ use a punishment that is of a higher lexicality than the other rewards in the problem.)
I agree with both of Daniel Kokotajlo’s points (both of which we also make in the paper in Sections IV.1 and IV.2): Certainly for humans it’s normal to not be able to randomize; and even if it was a primarily hypothetical situation without any obvious practical application, I’d still be interested in knowing how to deal with the absence of the ability to randomize.
Besides, as noted in my other comment insisting on the ability to randomize doesn’t get you that far (cf. Sections IV.1 and IV.4 on Ratificationism): even if you always have access to some nuclear decay noise channel, your choice of whether to consult that channel (or of whether to factor the noise into your decision) is still deterministic. So you can set up scenarios where if you are punished for randomizing. In the particular case of the Adversarial Offer, the seller might remove all money from both boxes if she predicts the buyer to randomize.
The reason why our main scenario just assumes that randomization isn’t possible is that our target of attack in this paper is primarily CDT, which is fine with not being allowed to randomize.
Care to elaborate on the every day thing? Aside from literal coins, your cell phone is perfectly capable of generating pseudorandom numbers, and I’m almost never without mine.
I guess whether your point stands depends on whether we are more concerned with abstract theory or practical decision making.
Here are some circumstances where you don’t have access to an unpredictable random number generator:
--You need to make a decision very quickly and so don’t have time to flip a coin
--Someone is watching you and will behave differently towards you if they see you make the decision via randomness, so consulting a coin isn’t a random choice between options but rather an additional option with its own set of payoffs
--Someone is logically entangled with you and if you randomize they will no longer be.
--You happen to be up against someone who is way smarter than you and can predict your coin / RNG / etc.
Admittedly, while in some sense these things happen literally every day to all of us, they typically don’t happen for important decisions.
But there are important decisions having to do with acausal trade that fit into this category, that either we our our AI successors will face one day.
And even if that wasn’t true, decision theory is decision THEORY. If one theory outperforms another in some class of cases, that’s a point in its favor, even if the class of cases is unusual.
EDIT: See Paul Christiano’s example below, it’s an excellent example because it takes Caspar’s paper and condenses it into a very down-to-earth, probably-has-actually-happened-to-someone-already example.
I’ve picked up my game theory entirely informally. But in real world terms, perhaps we’re imagining a situation where a randomization approach isn’t feasible for some other reason than a random number generator being unavailable.
This connects slightly with the debate over whether or not to administer untested COVID vaccine en masse. To pick randomly “feels scary” compared to picking “for a reason,” but to pick “for a reason” when there isn’t an actual evidence basis yet undermines the authority of regulators, so regulators don’t pick anything until they have a “good reason” to do so. Their political calculus, in short, makes them unable to use a randomization scheme.
So in terms of real world applicability, the constraint on a non-randomizing strategy seems potentially relevant, although the other aspects of this puzzle don’t map onto COVID vaccine selection specifically.
Yeah, basically standard game theory doesn’t really have anything to say about the scenarios of the paper, because they don’t fit the usual game-theoretical models.
By the way, the paper has some discussion of what happens if you insist on having access to an unpredictable randomization device, see Sections IV.1 and the discussion of Ratificationism in Section IV.4. (The latter may be of particular interest because Ratificationism is somewhat similar to Nash equilibrium. Unfortunately, the section doesn’t explain Ratificationism in detail.)
I’m confused since as a buyer if I believed the seller could predict with probability .75 I would flip a fair coin to decide which box to take meaning that the seller couldn’t predict with probability .75. If I can’t randomize to pick a box I’m not sure how to fit in what you are doing to standard game theory (which I teach).
If standard game theory has nothing to say about what to do in situations where you don’t have access to an unpredictable randomization mechanism, so much the worse for standard game theory, I say!
I thought the ability to deploy mixed strategies was a pretty standard part of CDT. Is this not the case, or are you considering a non-standard formulation of CDT?
I think some people may have their pet theories which they call CDT and which require randomization. But CDT as it is usually/traditionally described doesn’t ever insist on randomizing (unless randomizing has a positive causal effect). In this particular case, even if a randomization device were made available, CDT would either uniquely favor one of the boxes or be indifferent between all distributions over {buy box B1,buy box B2}. Compare Section IV.1 of the paper.
What you’re referring to are probably so-called ratificationist variants of CDT. These would indeed require randomizing 50-50 between the two boxes. But one can easily construct scenarios which trip these theories up. For example, the seller could put no money in any box if she predicts that the buyer will randomize. Then no distribution is ratifiable. See Section IV.4 for a discussion of Ratificationism.
This is a bit unsatisfying, because in my view of decision theory you don’t get to predict things like “the agent will randomize” or “the agent will take one box but feel a little wistful about it” and so on. This is unfair, in the same way as predicting that “the agent will use UDT” and punishing for it is unfair. No, you just predict the agent’s output. Or if the agent can randomize, you can sample (as many times as you like, but finitely many) from the distribution of the agent’s output. A bit more on this here, though the post got little attention.
Can your argument be extended to this case?
Why not? You surely agree that sometimes people can in fact predict such things. So your objection must be that it’s unfair when they do and that it’s not a strike against a decision theory if it causes you to get money-pumped in those situations. Well… why? Seems pretty bad to me, especially since some extremely high-stakes real-world situations our AIs might face will be of this type.
Sure, and sometimes people can predict things like “the agent will use UDT” and use that to punish the agent. But this kind of prediction is “unfair” because it doesn’t lead to an interesting decision theory—you can punish any decision theory that way. So to me the boundaries of “fair” and “unfair” are also partly about mathematical taste and promising-ness, not just what will lead to a better tank and such.
Right, that kind of prediction is unfair because it doesn’t lead to an interesting decision theory… but I asked why you don’t get to predict things like “the agent will randomize.” All sorts of interesting decision theory comes out of considering situations where you do get to predict such things. (Besides, such situations are important in real life.)
I might suggest “not interesting” rather than “not fair” as the complaint. One can image an Omega that leaves the box empty if the player is unpredictable, or if the player doesn’t rigorously follow CDT, or just always leaves it empty regardless. But there’s no intuition pump that it drives, and no analysis of why a formalization would or wouldn’t get the right answer.
When I’m in challenge-the-hypothetical mode, I defend CDT by making the agent believe Omega cheats. It’s a trick box that changes contents AFTER the agent chooses, BEFORE the contents are revealed. This is much higher probability to any rational agent than mind-reading or extreme predictability.
On the more philosophical points. My position is perhaps similar to Daniel K’s. But anyway...
Of course, I agree that problems that punish the agent for using a particular theory (or using float multiplication or feeling a little wistful or stuff like that) are “unfair”/”don’t lead to interesting theory”. (Perhaps more precisely, I don’t think our theory needs to give algorithms that perform optimally in such problems in the way I want my theory to “perform optimally” Newcomb’s problem. Maybe we should still expect our theory to say something about them, in the way that causal decision theorists feel like CDT has interesting/important/correct things to say about Newcomb’s problem, despite Newcomb’s problem being designed to (unfairly, as they allege) reward non-CDT agents.)
But I don’t think these are particularly similar to problems with predictions of the agent’s distribution over actions. The distribution over actions is behavioral, whereas performing floating point operations or whatever is not. When randomization is allowed, the subject of your choice is which distribution over actions you play. So to me, which distribution over actions you choose in a problem with randomization allowed, is just like the question of which action you take when randomization is not allowed. (Of course, if you randomize to determine which action’s expected utility to calculate first, but this doesn’t affect what you do in the end, then I’m fine with not allowing this to affect your utility, because it isn’t behavioral.)
I also don’t think this leads to uninteresting decision theory. But I don’t know how to argue for this here, other than by saying that CDT, EDT, UDT, etc. don’t really care whether they choose from/rank a set of distributions or a set of three discrete actions. I think ratificationism-type concepts are the only ones that break when allowing discontinuous dependence on the chosen distribution and I don’t find these very plausible anyway.
To be honest, I don’t understand the arguments against predicting distributions and predicting actions that you give in that post. I’ll write a comment on this to that post.
Let’s start with the technical question:
>Can your argument be extended to this case?
No, I don’t think so. Take the class of problems. The agent can pick any distribution over actions. The final payoff is determined only as a function of the implemented action and some finite number of samples generated by Omega from that distribution. Note that the expectation is continuous in the distribution chosen. It can therefore be shown (using e.g. Kakutani’s fixed-point theorem) that there is always at least one ratifiable distribution. See Theorem 3 at https://users.cs.duke.edu/~ocaspar/NDPRL.pdf .
(Note that the above is assuming the agent maximizes expected vNM utility. If, e.g., the agent maximizes some lexical utility function, then the predictor can just take, say, two samples and if they differ use a punishment that is of a higher lexicality than the other rewards in the problem.)
Thanks! That’s what I wanted to know. Will reply to the philosophical stuff in the comments to the other post.
How often do you encounter a situation where an unpredictable randomization mechanism is unavailable?
I agree with both of Daniel Kokotajlo’s points (both of which we also make in the paper in Sections IV.1 and IV.2): Certainly for humans it’s normal to not be able to randomize; and even if it was a primarily hypothetical situation without any obvious practical application, I’d still be interested in knowing how to deal with the absence of the ability to randomize.
Besides, as noted in my other comment insisting on the ability to randomize doesn’t get you that far (cf. Sections IV.1 and IV.4 on Ratificationism): even if you always have access to some nuclear decay noise channel, your choice of whether to consult that channel (or of whether to factor the noise into your decision) is still deterministic. So you can set up scenarios where if you are punished for randomizing. In the particular case of the Adversarial Offer, the seller might remove all money from both boxes if she predicts the buyer to randomize.
The reason why our main scenario just assumes that randomization isn’t possible is that our target of attack in this paper is primarily CDT, which is fine with not being allowed to randomize.
Every day. But even if it was only something that happened in weird hypotheticals, my point would still stand.
Care to elaborate on the every day thing? Aside from literal coins, your cell phone is perfectly capable of generating pseudorandom numbers, and I’m almost never without mine.
I guess whether your point stands depends on whether we are more concerned with abstract theory or practical decision making.
Here are some circumstances where you don’t have access to an unpredictable random number generator:
--You need to make a decision very quickly and so don’t have time to flip a coin
--Someone is watching you and will behave differently towards you if they see you make the decision via randomness, so consulting a coin isn’t a random choice between options but rather an additional option with its own set of payoffs
--Someone is logically entangled with you and if you randomize they will no longer be.
--You happen to be up against someone who is way smarter than you and can predict your coin / RNG / etc.
Admittedly, while in some sense these things happen literally every day to all of us, they typically don’t happen for important decisions.
But there are important decisions having to do with acausal trade that fit into this category, that either we our our AI successors will face one day.
And even if that wasn’t true, decision theory is decision THEORY. If one theory outperforms another in some class of cases, that’s a point in its favor, even if the class of cases is unusual.
EDIT: See Paul Christiano’s example below, it’s an excellent example because it takes Caspar’s paper and condenses it into a very down-to-earth, probably-has-actually-happened-to-someone-already example.
I’ve picked up my game theory entirely informally. But in real world terms, perhaps we’re imagining a situation where a randomization approach isn’t feasible for some other reason than a random number generator being unavailable.
This connects slightly with the debate over whether or not to administer untested COVID vaccine en masse. To pick randomly “feels scary” compared to picking “for a reason,” but to pick “for a reason” when there isn’t an actual evidence basis yet undermines the authority of regulators, so regulators don’t pick anything until they have a “good reason” to do so. Their political calculus, in short, makes them unable to use a randomization scheme.
So in terms of real world applicability, the constraint on a non-randomizing strategy seems potentially relevant, although the other aspects of this puzzle don’t map onto COVID vaccine selection specifically.
Yeah, basically standard game theory doesn’t really have anything to say about the scenarios of the paper, because they don’t fit the usual game-theoretical models.
By the way, the paper has some discussion of what happens if you insist on having access to an unpredictable randomization device, see Sections IV.1 and the discussion of Ratificationism in Section IV.4. (The latter may be of particular interest because Ratificationism is somewhat similar to Nash equilibrium. Unfortunately, the section doesn’t explain Ratificationism in detail.)