So, the Dutch Book argument for the axioms of probability theory has an adversarial form as well. The same can be said of the money-pump argument which justifies expected utility theory. Bayesians are not so averse to adversarial assumptions as they may seem; lurking behind the very notion of “doing well in expectation” is a “for all” requirement! Bayesians turn up their noses at decision procedures which try to do well in any other than an average-case sense because they know such a procedure is money-pumpable; an adversary could swoop in and take advantage of it!
This funny mix of average-case and worst-case reasoning is at the very foundation of the Bayesian edifice.
This is incorrect. The Bayesian edifice involves a complete rejection of worst-case reasoning. The possibility of an adversary tricking you into paying money to go through a sequence of trades that leave you back where you started isn’t a good justification for the transitivity axiom; it’s just a story that sacrifices being a correct argument when taken literally in favor of dramatic appeal, and is intended to hint in the direction of the actual argument. It’s the “a paperclip factory will build an AI that will turn you into paperclips” of justifying transitive preferences. Concluding from this story that advocates of expected utility theory are worried primarily about being exploited by adversaries is missing the point in the same way that concluding that advocates of taking AI risk seriously are worried primarily about being turned into paperclips is.
The actual justification for transitive preferences is not a worst-case argument, but an EVERY-case argument. There doesn’t exist any way for an agent to demonstrate that it has intransitive preferences without wasting resources by fighting itself. If there are any lotteries A, B, and C such that an agent prefers A>B, B>C, and C>A, and a nonzero probability for each pair from those three that the agent will get the opportunity to pay a small cost (“small” meaning small enough not to change any of those preferences) to choose between those two, then the agent pays for a probability distribution over outcomes that it could have gotten without paying. So the best case for an agent with intransitive preferences is that it never gets the chance to act on those preferences, and thus acts the same was that an agent with transitive preferences would. Anything else results in the agent spending unnecessary costs.
Why is the scenario you describe the “real” argument for transitivity, rather than the sequential scenario? Or are you pointing to a class of scenarios that includes the sequential one?
A few reasons. First, the VNM framework isn’t about sequential decisions; it’s about one-shot decisions. This doesn’t matter too much in practice because sequential decision problems can be turned into one-shot decision problems either by having the agent pick a policy, or by using what the agent expects it will do in the future in each case to figure out what future outcomes are currently the available options. So if the agent is being supposedly being offered a choice between A and B, but if it picks B, then it will taken an option to switch to C in the future, then it isn’t actually being offered a choice between A and B. The sequential argument doesn’t really make sense in the static VNM context.
But also, the argument from the sequential scenario is much less robust, since as Abram pointed out, it is only one scenario that could happen with intransitive preferences. The fact that every scenario in which an agent gets to act on its intransitive preference also involves unnecessary costs to the agent seems more important. Another way in which the sequential scenario is less robust is that it can be defeated by having a policy of stopping before you get back where you started if offered the opportunity to pay to repeatedly switch outcomes. But of course this tactic does not change the fact that if you pay to go even one step no matter what your starting position was, then you’re paying unnecessary costs.
This is incorrect. The Bayesian edifice involves a complete rejection of worst-case reasoning. The possibility of an adversary tricking you into paying money to go through a sequence of trades that leave you back where you started isn’t a good justification for the transitivity axiom; it’s just a story that sacrifices being a correct argument when taken literally in favor of dramatic appeal, and is intended to hint in the direction of the actual argument. It’s the “a paperclip factory will build an AI that will turn you into paperclips” of justifying transitive preferences. Concluding from this story that advocates of expected utility theory are worried primarily about being exploited by adversaries is missing the point in the same way that concluding that advocates of taking AI risk seriously are worried primarily about being turned into paperclips is.
The actual justification for transitive preferences is not a worst-case argument, but an EVERY-case argument. There doesn’t exist any way for an agent to demonstrate that it has intransitive preferences without wasting resources by fighting itself. If there are any lotteries A, B, and C such that an agent prefers A>B, B>C, and C>A, and a nonzero probability for each pair from those three that the agent will get the opportunity to pay a small cost (“small” meaning small enough not to change any of those preferences) to choose between those two, then the agent pays for a probability distribution over outcomes that it could have gotten without paying. So the best case for an agent with intransitive preferences is that it never gets the chance to act on those preferences, and thus acts the same was that an agent with transitive preferences would. Anything else results in the agent spending unnecessary costs.
Why is the scenario you describe the “real” argument for transitivity, rather than the sequential scenario? Or are you pointing to a class of scenarios that includes the sequential one?
A few reasons. First, the VNM framework isn’t about sequential decisions; it’s about one-shot decisions. This doesn’t matter too much in practice because sequential decision problems can be turned into one-shot decision problems either by having the agent pick a policy, or by using what the agent expects it will do in the future in each case to figure out what future outcomes are currently the available options. So if the agent is being supposedly being offered a choice between A and B, but if it picks B, then it will taken an option to switch to C in the future, then it isn’t actually being offered a choice between A and B. The sequential argument doesn’t really make sense in the static VNM context.
But also, the argument from the sequential scenario is much less robust, since as Abram pointed out, it is only one scenario that could happen with intransitive preferences. The fact that every scenario in which an agent gets to act on its intransitive preference also involves unnecessary costs to the agent seems more important. Another way in which the sequential scenario is less robust is that it can be defeated by having a policy of stopping before you get back where you started if offered the opportunity to pay to repeatedly switch outcomes. But of course this tactic does not change the fact that if you pay to go even one step no matter what your starting position was, then you’re paying unnecessary costs.