This is an important theorem. There is no perfect decision theory, especially against equal-or-better opponents. I tend to frame it as “the better predictor wins”. Almost all such adversarial/fixed-sum cases are about power, not fairness or static strategy/mechanism.
We (humans, including very smart theorists) REALLY want to frame it as clever ways to get outcomes that fit our intuitions. But it’s still all about “who goes first (in the logical/credible-committment sense)”.
MIRI has also done work on decision problems outside LDT’s fair problem class, like Open-Source Prisoner’s Dilemma. FairBot cooperates if it can prove you cooperate, defects otherwise. In this case, being too hard to predict gets you defected against.
Sure—this goes to my “equal-or-better opponent” description. Any interesting real-world agent is not provably cooperative (if it’s equal or more complex than you), or if it is, it’s exploitable by other agents that can prove it’s cooperation.
This is an important theorem. There is no perfect decision theory, especially against equal-or-better opponents. I tend to frame it as “the better predictor wins”. Almost all such adversarial/fixed-sum cases are about power, not fairness or static strategy/mechanism.
We (humans, including very smart theorists) REALLY want to frame it as clever ways to get outcomes that fit our intuitions. But it’s still all about “who goes first (in the logical/credible-committment sense)”.
MIRI has also done work on decision problems outside LDT’s fair problem class, like Open-Source Prisoner’s Dilemma.
FairBot cooperates if it can prove you cooperate, defects otherwise. In this case, being too hard to predict gets you defected against.
Sure—this goes to my “equal-or-better opponent” description. Any interesting real-world agent is not provably cooperative (if it’s equal or more complex than you), or if it is, it’s exploitable by other agents that can prove it’s cooperation.