So if I understand you right, even with the short-sighted utility function, there’s an echo of Parfit’s Hitchhiker here: what TDT decides on these problems actually controls which situations the agent finds itself in, and thus its possible payoff matrices. Since TDT is supposed to get Parfit’s Hitchhiker right, therefore, it should give the long-term-winning answer even in this case.
Well, there are some more caveats (it’s not clear that agents in the first round would do this, since TDT doesn’t win the Counterfactual Mugging, and if agents in round N don’t think that way, then what about round N+1...), but you’re right that the simple calculation doesn’t actually suffice. Drat, and thanks.
Since TDT is supposed to get Parfit’s Hitchhiker right, therefore, it should give the long-term-winning answer even in this case.
Its goal is still different though (if we restore some missing pieces): it wants to game the probabilities of encountering certain opponents so that a single round that contains TDT delivers the most reward. It just so happens that getting rid of DefectBots serves this purpose, but if the opponents were CooperateBots, it looks like TDTs would drive themselves to extinction (or farm the opponents) to maximize the number of expected cooperating opponents that they can defect against (for each instance where there’s a TDT agent in the round). (I didn’t check this example carefully, so could be wrong, but the principle it exemplifies seems to hold.)
So if I understand you right, even with the short-sighted utility function, there’s an echo of Parfit’s Hitchhiker here: what TDT decides on these problems actually controls which situations the agent finds itself in, and thus its possible payoff matrices. Since TDT is supposed to get Parfit’s Hitchhiker right, therefore, it should give the long-term-winning answer even in this case.
Well, there are some more caveats (it’s not clear that agents in the first round would do this, since TDT doesn’t win the Counterfactual Mugging, and if agents in round N don’t think that way, then what about round N+1...), but you’re right that the simple calculation doesn’t actually suffice. Drat, and thanks.
Its goal is still different though (if we restore some missing pieces): it wants to game the probabilities of encountering certain opponents so that a single round that contains TDT delivers the most reward. It just so happens that getting rid of DefectBots serves this purpose, but if the opponents were CooperateBots, it looks like TDTs would drive themselves to extinction (or farm the opponents) to maximize the number of expected cooperating opponents that they can defect against (for each instance where there’s a TDT agent in the round). (I didn’t check this example carefully, so could be wrong, but the principle it exemplifies seems to hold.)
That’s a seriously sick idea, but there doesn’t seem to be a way to both set up such a favorable matchup and exploit it- is there?