If you can alter your source code so that you will definitely pass up a somewhat desirable contract if your ideal terms aren’t met, you’ll be offered better contracts than agents that can’t alter their source code.
Why do you think this is true as a general rule?
It obviously won’t be true in a population where everyone modifies their source code in the same way. Everyone will reject every contract there, and (if their modified source code allows them any sapience apart from the rule about accepting contracts) everyone will know that all contracts will be rejected and not bother to offer any.
It might be effective in a population of mostly CDT theorists who look only at only the direct causal outcomes of their actions in whatever situation they happen to have stumbled into. However, even a CDT user might determine that accepting a bad deal now raises the probability of future bots modifying their source code and offering bad deals in the future that would otherwise have been good. This might outweigh the immediate slight utility gain of accepting a single bad deal rather than no deal. Given that “there are lots of other people who decide in much the same way that I do” is outside CDT’s scope, the effect will be weak and only decisive if there are likely to be many future deals with bots that use the outcome of this particular deal to decide whether to self-modify.
It certainly wouldn’t be effective in a population of largely FDT users who know that most others use FDT, since predictably accepting bad deals yields a world in which you only get offered bad deals. Altering their own source code is just one of the many ways that they can credibly offer bad deals to you, and is nothing special. Missing out occasionally on a little utility if you happen to encounter an irrational self-modified bot is more than made up by the gains from being in a world where self-modifying is irrational.
It’s also no use in a world where most of the population has a “fairness” heuristic and don’t care whether it’s rational or not. The bot may get a few 90% deals, and in that sense “gets offered better contracts”, but misses out on a lot more 50% deals that would have been of greater total benefit.
Why do you think this is true as a general rule?
It obviously won’t be true in a population where everyone modifies their source code in the same way. Everyone will reject every contract there, and (if their modified source code allows them any sapience apart from the rule about accepting contracts) everyone will know that all contracts will be rejected and not bother to offer any.
It might be effective in a population of mostly CDT theorists who look only at only the direct causal outcomes of their actions in whatever situation they happen to have stumbled into. However, even a CDT user might determine that accepting a bad deal now raises the probability of future bots modifying their source code and offering bad deals in the future that would otherwise have been good. This might outweigh the immediate slight utility gain of accepting a single bad deal rather than no deal. Given that “there are lots of other people who decide in much the same way that I do” is outside CDT’s scope, the effect will be weak and only decisive if there are likely to be many future deals with bots that use the outcome of this particular deal to decide whether to self-modify.
It certainly wouldn’t be effective in a population of largely FDT users who know that most others use FDT, since predictably accepting bad deals yields a world in which you only get offered bad deals. Altering their own source code is just one of the many ways that they can credibly offer bad deals to you, and is nothing special. Missing out occasionally on a little utility if you happen to encounter an irrational self-modified bot is more than made up by the gains from being in a world where self-modifying is irrational.
It’s also no use in a world where most of the population has a “fairness” heuristic and don’t care whether it’s rational or not. The bot may get a few 90% deals, and in that sense “gets offered better contracts”, but misses out on a lot more 50% deals that would have been of greater total benefit.