I don’t understand either of your two corrections. For #1, is it really usually a requirement that the person making you the offer be a perfect predictor? What about a human-level predictor in an iterated game? Yudkowsky writes that on dath ilan, people use The Algorithm, which is to reject unfair offers with high enough probability to make it a bad gamble for the offer-maker. Dath ilan is populated by humans, not perfect predictors, so I think it’s fair to say standard theory says that this applies to humans.
Or, well, okay, maybe I should say “the zeitgeist of Lesswrong” instead of “standard theory”. In which case, my post is challenging our zeitgeist’s understanding of these two principles, not some theorem with much stricter requirements for who it applies to.
For #2, are you sure nobody’s advocating that? I think Planecrash has a plot element that demonstrates that people do advocate this (spoilers for Planecrash, obviously):
Keltham wants to destroy the entire universe because he’d rather do that than let Hell continue to exist. He realizes that the gods may bargain him down from that by offering to just fix Hell, but ~ignores this consideration because if he fixates on that and does anything differently in pursuit of such bargaining, then he’d be making a Threat instead of just maximizing his utility function, and the gods would predictably ignore him. Carissa is around too, but he rejects several possible ways she could influence his project on the grounds that, since she doesn’t prefer to destroy the universe and only wants Hell to be fixed, then if he lets her influence him in the wrong way, the gods might consider Keltham’s universe-destruction game to ultimately be a Threat made by Carissa, and predictably ignore it.
This seems very analogous to the mugging case. If it’s different, what’s the difference? Mind you that the person making the threat is a (very smart) human, not a perfect predictor.
Alternatively, what does the zeitgiest around here say about the human-governance principle “don’t negotiate with terrorists”? That one is unambiguously applied to humans. Do we generally disagree with it for that reason?
I’m just not convinced that my post frames either of these two things incorrectly. I think people around here do generally say that human beings should both reject unfair Ultimatum Game offers and not give in to threats.
I don’t understand either of your two corrections. For #1, is it really usually a requirement that the person making you the offer be a perfect predictor? What about a human-level predictor in an iterated game? Yudkowsky writes that on dath ilan, people use The Algorithm, which is to reject unfair offers with high enough probability to make it a bad gamble for the offer-maker. Dath ilan is populated by humans, not perfect predictors, so I think it’s fair to say standard theory says that this applies to humans.
Or, well, okay, maybe I should say “the zeitgeist of Lesswrong” instead of “standard theory”. In which case, my post is challenging our zeitgeist’s understanding of these two principles, not some theorem with much stricter requirements for who it applies to.
For #2, are you sure nobody’s advocating that? I think Planecrash has a plot element that demonstrates that people do advocate this (spoilers for Planecrash, obviously):
Keltham wants to destroy the entire universe because he’d rather do that than let Hell continue to exist. He realizes that the gods may bargain him down from that by offering to just fix Hell, but ~ignores this consideration because if he fixates on that and does anything differently in pursuit of such bargaining, then he’d be making a Threat instead of just maximizing his utility function, and the gods would predictably ignore him. Carissa is around too, but he rejects several possible ways she could influence his project on the grounds that, since she doesn’t prefer to destroy the universe and only wants Hell to be fixed, then if he lets her influence him in the wrong way, the gods might consider Keltham’s universe-destruction game to ultimately be a Threat made by Carissa, and predictably ignore it.
This seems very analogous to the mugging case. If it’s different, what’s the difference? Mind you that the person making the threat is a (very smart) human, not a perfect predictor.
Alternatively, what does the zeitgiest around here say about the human-governance principle “don’t negotiate with terrorists”? That one is unambiguously applied to humans. Do we generally disagree with it for that reason?
I’m just not convinced that my post frames either of these two things incorrectly. I think people around here do generally say that human beings should both reject unfair Ultimatum Game offers and not give in to threats.