The solution of precommitting on a meta level by adopting a decision theory that’s not susceptible to this kind of race is pretty strong, in a universe where you encounter such things, and where you matter to the bot. Bot’s counter-counter is to self-modify to offer small amounts, and train the humans to take the deal rather than undertaking clever precommittments. In a universe that’s more rich and has a mix of agents with different capabilities and mechanisms for decision-making, it’ll depend on how much knowledge you have of your opponent.
The real lesson in thought experiments of this nature is “whoever can model their opponent better than their opponent can model them has a huge edge”. Also, real games are far more complicated, and whoever can model the rest of life in relation to the game at hand better has an edge as well.
The solution of precommitting on a meta level by adopting a decision theory that’s not susceptible to this kind of race is pretty strong, in a universe where you encounter such things, and where you matter to the bot. Bot’s counter-counter is to self-modify to offer small amounts, and train the humans to take the deal rather than undertaking clever precommittments. In a universe that’s more rich and has a mix of agents with different capabilities and mechanisms for decision-making, it’ll depend on how much knowledge you have of your opponent.
The real lesson in thought experiments of this nature is “whoever can model their opponent better than their opponent can model them has a huge edge”. Also, real games are far more complicated, and whoever can model the rest of life in relation to the game at hand better has an edge as well.
Yeah, the crucial point is whoever out-models the opponent wins.