Bets and updating
Suppose the US presidential election is tomorrow. You currently assign a probability of 50% to each outcome. (We are ignoring the small possibility that neither of the main party candidates will win).
A man approaches you, and offers you a bet of $10, at 2-1 odds. In other words, if candidate one wins, he pays you $20, if candidate two wins, you pay him $10.
Should you accept this bet? What if the bet was for $10000 instead? Assume that your utility is linear in dollars (or assume that the bet is for utilons instead, whatever). If not, why not? Try to think about this before reading on.
The answer is that it depends on your priors—in particular, it depends on how you interpret the evidence of being offered the bet. In general, if someone offers you a large bet on some outcome, it’s probably safe to assume they have access to a reasonable amount of information about the outcome. Depending on how much information your own probability estimate is based on, you should update towards the odds they offered you.
If you update too little, and accept too many bets, you will lose a lot of money to people with better information than you. On the other hand, you can also go too far in the other direction. If your response to being offered a five-cent bet is to immediately update to accept their probabilities (and refuse the bet), you will be very easy to fool (although hard to exploit by betting).
Now suppose there are two superintelligences, Omega and Omicron. They are both excellent at modelling both you and the presidential election. Omega has a strong preference for money, and a weak preference for having you believe false things about the presidential election. Omicron has this swapped—it wants you to believe that actual outcome of the election (which it has predicted) is extremely unlikely, and has a weak preference for money.
Omega executes the following plan: It looks through a large number of possible bets, looking for ones that will give it a lot of money (according to its predicted outcome of the election). For each of them, it predicts whether you will, if offered, take the bet, or simply update your belief to be more accurate (if the bet is such that Omega wins money, it must be “in the correct direction”). It finds the best bet you will take (if any), and offers you this bet.
Omicron does a similar thing, but instead looks for bets that you won’t take—bets which will instead cause you to update strongly in the wrong direction, and not take the bet (since it doesn’t want to give you money). Again, it finds the best bet you won’t take (if any) and approaches you.
You are approached by someone and offered a bet, although you don’t know if it’s Omega or Omicron. What should your policy be?
If you accept a bet, the bet is likely to have come from Omega, and thus be extremely costly for you. So you shouldn’t take the bet.
On the other hand, if you update strongly in the direction of the bet, it most likely came from Omicron, and this means it’s probably an update in the wrong direction. So you shouldn’t update.
This leaves you in the perplexing situation of believing that the bet is probably extremely good, but not wanting to take it.