I believe it’s the algorithm from https://www.lesswrong.com/posts/z2YwmzuT7nWx62Kfh/cooperating-with-agents-with-different-ideas-of-fairness. Basically, if you’re offered an unfair deal (and the other trader isn’t willing to renegotiate), you should accept the trade with a probability just low enough that the other trader does worse in expectation than if they offered a fair trade. For example, if you think that a fair deal would provide $10 to both players over not trading and the other trader offers a deal where they get $15 and you get $4, then you should accept with probability , so that in expectation they get less than if they offered a fair trade.
Any Pareto bargaining method is vulnerable to lying about utility functions, and so to have a chance at bargaining fairly, it’s necessary to have some idea of what your partner’s utility function is. I don’t think that using this method for dealing with unfair trades is especially vulnerable to deception, though possibly there’s some better way to deal with uncertainty over your partner’s utility function.
This is exactly why Eliezer (and I) would turn down a rock with an unfair offer. Sure, there’s some tiny chance that it was indeed created ex nihilo, but it’s far more likely that it was produced by some process that deliberately tried to hide the process that produced the offer.
This depends on the basis of that premonition. At every point, Bot is considering the effect of its commitment on some space of possible agents, which gets narrowed down whenever Bot learns more about Eliezer. If Bot knows everything about Eliezer when it makes the commitment, then of course Eliezer should not give in. If Bot knows some evidence, then actual-Eliezer is essentially in a cooperation-dilemma with the possible agents that Bot thinks Eliezer could be. Then, Eliezer should not give in if he thinks that he can logically cooperate with enough of the possible agents to make Bot’s commitment unwise.
This isn’t true all of the time; I expect that a North Korean version of Eliezer would give into threats, since he would be unable to logically cooperate with enough of the population to make the government see threat-making as pointless. Still, I do expect that in the situations which Eliezer (and I, for that matter) encounters in the future to not be like this.