I wonder if some of the conflation between belief-as-prediction and belief-as-investment is actually a functional social technology for solving coordination problems. To avoid multi-polar traps, people need to trust eachother to act against individual incentives- to rationally pre-commit to acting irrationally in the future. Just telling people “I’m planning act against my incentives, even though I know that doing so will be irrational at the time” might not be very convincing, but instead claiming to have irrationally certain beliefs that would change your incentives were that certainty warranted can be more convincing. Even if people strongly suspect that you’re exaggerating, they know that the social pressure to avoid a loss of status by admitting that you were wrong will make you less likely to defect.
For example, say you’re planning to start a band with some friends. You all think the effort and investment will be worth it so long as there’s a 50% chance of the band succeeding, and you all privately think there’s about about a 70% chance of the band succeeding if everyone stays committed, and a near 0% chance if anybody drops out. Say there’s enough random epistemic noise that you think it’s pretty likely someone in the band will eventually drop their odds below that 50% threshold, even when you personally still give success conditional on commitment much better odds. So, unless you can trust everyone to stay committed even if they come to believe it’s not worth the effort, you might as well give up on the band before starting it. Classic multi-polar trap. If, however, everyone at the start is willing to say “I’m certain we’ll succeed”, putting more of their reputation on the line, that might build enough trust to overcome the coordination problem.
Of course, this can create all sorts of epistemic problems. Maybe everyone in the band comes to believe that it’s not worth the effort, but incorrectly think that saying so will be a defection. Maybe their exaggerated certainty misleads other people in ways that cause them to make bad investments or to dangerously misunderstand the music industry.
Maybe there’s a sense in which this solution to individual coordination problems is part of a larger coordination problem- everyone incentivized to reap the value of greater trust, but causing a greater loss of value to people more broadly by damaging the epistemic commons.
There might be some motivated reasoning on that last point, however, since I definitely find it emotionally uncomfortable when people say inaccurate things for social reasons.
So you form the band and try to figure out how to keep everyone working together so that no one’s confidence drops below 50%. If you’re not sure you can do that, consider the value of trying anyway and seeing if you can do it. If the expected values still don’t work out, don’t start the band.
(If there’s something you should precommit to, it’s not centraly “irrational” to do that thing. Or if it is indeed centrally “irrational” to do it, maybe you shouldn’t precommit to it. In this case, it’s only “irrational” according to a myopic problem statement that is itself not the right thing to follow. And in the above narrow sense of “rationality” as preference towards better methods rather than merely correctness of individual beliefs and decisions according to given methods, none of these things are either “rational” or “irrational”.)
I agree, though if we’re defining rationality as a preference for better methods, I think we ought to further disambiguate between “a decision theory that will dissolve apparent conflicts between what we currently want our future selves to do and what those future selves actually want to do” and “practical strategies for aligning our future incentives with our current ones”
Suppose someone tells you that they’ll offer you $100 tomorrow and $10,000 today if you make a good-faith effort to prevent yourself from accepting the $100 tomorrow. The best outcome would be to make a genuine attempt to disincentivize yourself from accepting the money tomorrow, but fail and accept the money anyway- however, you can’t actually try and make that happen without violating the terms of the deal.
if your effort to constrain your future self on day one does fail, I don’t think there’s a reasonable decision theory that would argue you should reject the money anyway. On day one, you’re being paid to temporarily adopt preferences misaligned with your preferences on day two. You can try to make that change in preferences permanent, or to build an incentive structure to enforce that preference, or maybe even strike an acausal bargain with your day two self, but if all of that fails, you ought to go ahead and accept the $100.
I think coordination problems are a lot like that. They reward you for adopting preferences genuinely at odds with those you may have later on. And what’s rational according to one set of preferences will be irrational according to another.
if your effort to constrain your future self on day one does fail, I don’t think there’s a reasonable decision theory that would argue you should reject the money anyway
That’s one of the things motivating UDT. On day two, you still ask what global policy you should follow (that in particular encompasses your actions in the past, and in the counterfactuals relative to what you actually observe in the current situation). Then you see where/when you actually are, what you actually observe, and enact what the best policy says you do in the current situation. You don’t constrain yourself on day one, but still enact the global policy on day two.
I think coordination problems are a lot like that. They reward you for adopting preferences genuinely at odds with those you may have later on.
Adopting preferences is a lot like enacting a policy, but when enacting a policy you don’t need to adopt preferences, a policy is something external, an algorithmic action (instead of choosing Cooperate, you choose to follow some algorithm that decides what to do, even if that algorithm gets no further input). Contracts in the usual sense act like that, assurance contracts is an example where you are explicitly establishing coordination. You can judge an algorithmic action like you judge an explicit action, but there are more algorithmic actions than there are explicit actions, and algorithmic actions taken by you and your opponents can themselves reason about each other, which enablescoordination.
I wonder if some of the conflation between belief-as-prediction and belief-as-investment is actually a functional social technology for solving coordination problems. To avoid multi-polar traps, people need to trust eachother to act against individual incentives- to rationally pre-commit to acting irrationally in the future. Just telling people “I’m planning act against my incentives, even though I know that doing so will be irrational at the time” might not be very convincing, but instead claiming to have irrationally certain beliefs that would change your incentives were that certainty warranted can be more convincing. Even if people strongly suspect that you’re exaggerating, they know that the social pressure to avoid a loss of status by admitting that you were wrong will make you less likely to defect.
For example, say you’re planning to start a band with some friends. You all think the effort and investment will be worth it so long as there’s a 50% chance of the band succeeding, and you all privately think there’s about about a 70% chance of the band succeeding if everyone stays committed, and a near 0% chance if anybody drops out. Say there’s enough random epistemic noise that you think it’s pretty likely someone in the band will eventually drop their odds below that 50% threshold, even when you personally still give success conditional on commitment much better odds. So, unless you can trust everyone to stay committed even if they come to believe it’s not worth the effort, you might as well give up on the band before starting it. Classic multi-polar trap. If, however, everyone at the start is willing to say “I’m certain we’ll succeed”, putting more of their reputation on the line, that might build enough trust to overcome the coordination problem.
Of course, this can create all sorts of epistemic problems. Maybe everyone in the band comes to believe that it’s not worth the effort, but incorrectly think that saying so will be a defection. Maybe their exaggerated certainty misleads other people in ways that cause them to make bad investments or to dangerously misunderstand the music industry.
Maybe there’s a sense in which this solution to individual coordination problems is part of a larger coordination problem- everyone incentivized to reap the value of greater trust, but causing a greater loss of value to people more broadly by damaging the epistemic commons.
There might be some motivated reasoning on that last point, however, since I definitely find it emotionally uncomfortable when people say inaccurate things for social reasons.
So you form the band and try to figure out how to keep everyone working together so that no one’s confidence drops below 50%. If you’re not sure you can do that, consider the value of trying anyway and seeing if you can do it. If the expected values still don’t work out, don’t start the band.
Like conflation around “belief”, it’s better to have a particular meaning in mind when calling something “rational”, such as methods that help more with finding truth, or with making effective plans.
(If there’s something you should precommit to, it’s not centraly “irrational” to do that thing. Or if it is indeed centrally “irrational” to do it, maybe you shouldn’t precommit to it. In this case, it’s only “irrational” according to a myopic problem statement that is itself not the right thing to follow. And in the above narrow sense of “rationality” as preference towards better methods rather than merely correctness of individual beliefs and decisions according to given methods, none of these things are either “rational” or “irrational”.)
I agree, though if we’re defining rationality as a preference for better methods, I think we ought to further disambiguate between “a decision theory that will dissolve apparent conflicts between what we currently want our future selves to do and what those future selves actually want to do” and “practical strategies for aligning our future incentives with our current ones”
Suppose someone tells you that they’ll offer you $100 tomorrow and $10,000 today if you make a good-faith effort to prevent yourself from accepting the $100 tomorrow. The best outcome would be to make a genuine attempt to disincentivize yourself from accepting the money tomorrow, but fail and accept the money anyway- however, you can’t actually try and make that happen without violating the terms of the deal.
if your effort to constrain your future self on day one does fail, I don’t think there’s a reasonable decision theory that would argue you should reject the money anyway. On day one, you’re being paid to temporarily adopt preferences misaligned with your preferences on day two. You can try to make that change in preferences permanent, or to build an incentive structure to enforce that preference, or maybe even strike an acausal bargain with your day two self, but if all of that fails, you ought to go ahead and accept the $100.
I think coordination problems are a lot like that. They reward you for adopting preferences genuinely at odds with those you may have later on. And what’s rational according to one set of preferences will be irrational according to another.
That’s one of the things motivating UDT. On day two, you still ask what global policy you should follow (that in particular encompasses your actions in the past, and in the counterfactuals relative to what you actually observe in the current situation). Then you see where/when you actually are, what you actually observe, and enact what the best policy says you do in the current situation. You don’t constrain yourself on day one, but still enact the global policy on day two.
Adopting preferences is a lot like enacting a policy, but when enacting a policy you don’t need to adopt preferences, a policy is something external, an algorithmic action (instead of choosing Cooperate, you choose to follow some algorithm that decides what to do, even if that algorithm gets no further input). Contracts in the usual sense act like that, assurance contracts is an example where you are explicitly establishing coordination. You can judge an algorithmic action like you judge an explicit action, but there are more algorithmic actions than there are explicit actions, and algorithmic actions taken by you and your opponents can themselves reason about each other, which enables coordination.