Something seems pretty weird about all this reasoning though. For one thing, there’s a sense that you sort of “travel backwards in logical time” as you think longer in normal time. Like, first you don’t know about TDT, and then you invent TDT, and UDT, and then you can do UDT better. So you start making decisions in accordance with policies you’d’ve wanted to pick “a priori” (earlier in some kind of “time”). But like what’s going on? We could say that UDT is convergent, as the only thing that’s reflectively stable, or as the only kind of thing that can be pareto optimal in conflicts, or something like that. But how do we make sense of our actual reasoning before having invented UDT? Is the job of that reasoning not to invent UDT, but just to avoid avoiding adopting UDT?
I don’t know how to formalize the reasoning process that goes into how we choose decision theories. And I doubt anyone does. Because if you could formalize the reasoning we use, then you could (indirectly) formalize decision theory itself as being, “whatever decision theory we would use given unlimited reflection”.
I don’t really think UDT is necessarily reflectively stable, or the only decision theory that is. I’ve argued previously that I, in certain situations, would act essential as an evidential decision theorist. I’m not sure what others think of this, though, since no one actually ever replied to me.
I don’t think UDT is pareto optimal in conflicts. If the agent is in a conflict with an irrational agent, then the resulting interaction between the two agents could easily be non-pareto optimal. For example, imagine a UDT agent is in a conflict with the same payoff to the prisoner’s dilemma. And suppose the agent it’s in conflict with is a causal decision theorist. Then the causal decision theorist would defect no matter what the UDT agent would do, so the UDT agent would also defect, and then everyone would do poorly.
Yeah I don’t know of a clear case for those supposed properties of UDT.
By pareto optimal I mean just, two UDT agents will pick a Pareto optimal policy. Whereas, say, two CDT agents may defect on each other in a PD.
This isn’t a proof, or even really a general argument, but one reason to suspect UDT is convergent, is that CDT would modify to be a sort of UDT-starting-now. At least, say you have a CDT agent, and further assume that it’s capable of computing the causal consequences of all possible complete action-policies it could follow. This agent would replace itself with P-bot, the bot that follows policy P, where P is the one with the best causal consequences at the time of replacement. This is different from CDT: if Omega scans P-bot the next day, P-bot will win the Transparent Newcomb’s problem, whereas if CDT hadn’t self-modified to be P-bot and Omega had scanned CDT tomorrow, CDT would fail the TNP for the usual reason. So CDT is in conflict with its future self.
Two UDT agents actually can potentially defect in prisoner’s dilemma. See the agent simulates predictor problem if you’re interested.
But I think you’re right that agents would generally modify themselves to more closely resemble UDT. Note, though, that the decision theory a CDT agent would modify itself to use wouldn’t exactly be UDT. For example, suppose the causal decision theory agent had its output predicted by Omega for Newcomb’s problem before the agent even came into existence. Then by the time the CDT agent comes to existence, modifying itself to use UDT would have no causal impact on the content of the boxes. So it wouldn’t adopt UDT in this situation and would still two-box.
Well, the way the agent loses in ASP is by failing to be updateless about certain logical facts (what the predictor predicts). So from this perspective, it’s a SemiUDT that does update whenever it learns logical facts, and this explains why it defects.
> So it wouldn’t adopt UDT in this situation and would still two-box.
True, it’s always [updateless, on everything after now].
Something seems pretty weird about all this reasoning though. For one thing, there’s a sense that you sort of “travel backwards in logical time” as you think longer in normal time. Like, first you don’t know about TDT, and then you invent TDT, and UDT, and then you can do UDT better. So you start making decisions in accordance with policies you’d’ve wanted to pick “a priori” (earlier in some kind of “time”). But like what’s going on? We could say that UDT is convergent, as the only thing that’s reflectively stable, or as the only kind of thing that can be pareto optimal in conflicts, or something like that. But how do we make sense of our actual reasoning before having invented UDT? Is the job of that reasoning not to invent UDT, but just to avoid avoiding adopting UDT?
I don’t know how to formalize the reasoning process that goes into how we choose decision theories. And I doubt anyone does. Because if you could formalize the reasoning we use, then you could (indirectly) formalize decision theory itself as being, “whatever decision theory we would use given unlimited reflection”.
I don’t really think UDT is necessarily reflectively stable, or the only decision theory that is. I’ve argued previously that I, in certain situations, would act essential as an evidential decision theorist. I’m not sure what others think of this, though, since no one actually ever replied to me.
I don’t think UDT is pareto optimal in conflicts. If the agent is in a conflict with an irrational agent, then the resulting interaction between the two agents could easily be non-pareto optimal. For example, imagine a UDT agent is in a conflict with the same payoff to the prisoner’s dilemma. And suppose the agent it’s in conflict with is a causal decision theorist. Then the causal decision theorist would defect no matter what the UDT agent would do, so the UDT agent would also defect, and then everyone would do poorly.
Yeah I don’t know of a clear case for those supposed properties of UDT.
By pareto optimal I mean just, two UDT agents will pick a Pareto optimal policy. Whereas, say, two CDT agents may defect on each other in a PD.
This isn’t a proof, or even really a general argument, but one reason to suspect UDT is convergent, is that CDT would modify to be a sort of UDT-starting-now. At least, say you have a CDT agent, and further assume that it’s capable of computing the causal consequences of all possible complete action-policies it could follow. This agent would replace itself with P-bot, the bot that follows policy P, where P is the one with the best causal consequences at the time of replacement. This is different from CDT: if Omega scans P-bot the next day, P-bot will win the Transparent Newcomb’s problem, whereas if CDT hadn’t self-modified to be P-bot and Omega had scanned CDT tomorrow, CDT would fail the TNP for the usual reason. So CDT is in conflict with its future self.
Two UDT agents actually can potentially defect in prisoner’s dilemma. See the agent simulates predictor problem if you’re interested.
But I think you’re right that agents would generally modify themselves to more closely resemble UDT. Note, though, that the decision theory a CDT agent would modify itself to use wouldn’t exactly be UDT. For example, suppose the causal decision theory agent had its output predicted by Omega for Newcomb’s problem before the agent even came into existence. Then by the time the CDT agent comes to existence, modifying itself to use UDT would have no causal impact on the content of the boxes. So it wouldn’t adopt UDT in this situation and would still two-box.
Well, the way the agent loses in ASP is by failing to be updateless about certain logical facts (what the predictor predicts). So from this perspective, it’s a SemiUDT that does update whenever it learns logical facts, and this explains why it defects.
> So it wouldn’t adopt UDT in this situation and would still two-box.
True, it’s always [updateless, on everything after now].