Bets and updating

Sup­pose the US pres­i­den­tial elec­tion is to­mor­row. You cur­rently as­sign a prob­a­bil­ity of 50% to each out­come. (We are ig­nor­ing the small pos­si­bil­ity that nei­ther of the main party can­di­dates will win).

A man ap­proaches you, and offers you a bet of $10, at 2-1 odds. In other words, if can­di­date one wins, he pays you $20, if can­di­date two wins, you pay him $10.

Should you ac­cept this bet? What if the bet was for $10000 in­stead? As­sume that your util­ity is lin­ear in dol­lars (or as­sume that the bet is for utilons in­stead, what­ever). If not, why not? Try to think about this be­fore read­ing on.

The an­swer is that it de­pends on your pri­ors—in par­tic­u­lar, it de­pends on how you in­ter­pret the ev­i­dence of be­ing offered the bet. In gen­eral, if some­one offers you a large bet on some out­come, it’s prob­a­bly safe to as­sume they have ac­cess to a rea­son­able amount of in­for­ma­tion about the out­come. Depend­ing on how much in­for­ma­tion your own prob­a­bil­ity es­ti­mate is based on, you should up­date to­wards the odds they offered you.

If you up­date too lit­tle, and ac­cept too many bets, you will lose a lot of money to peo­ple with bet­ter in­for­ma­tion than you. On the other hand, you can also go too far in the other di­rec­tion. If your re­sponse to be­ing offered a five-cent bet is to im­me­di­ately up­date to ac­cept their prob­a­bil­ities (and re­fuse the bet), you will be very easy to fool (al­though hard to ex­ploit by bet­ting).

Now sup­pose there are two su­per­in­tel­li­gences, Omega and Omicron. They are both ex­cel­lent at mod­el­ling both you and the pres­i­den­tial elec­tion. Omega has a strong prefer­ence for money, and a weak prefer­ence for hav­ing you be­lieve false things about the pres­i­den­tial elec­tion. Omicron has this swapped—it wants you to be­lieve that ac­tual out­come of the elec­tion (which it has pre­dicted) is ex­tremely un­likely, and has a weak prefer­ence for money.

Omega ex­e­cutes the fol­low­ing plan: It looks through a large num­ber of pos­si­ble bets, look­ing for ones that will give it a lot of money (ac­cord­ing to its pre­dicted out­come of the elec­tion). For each of them, it pre­dicts whether you will, if offered, take the bet, or sim­ply up­date your be­lief to be more ac­cu­rate (if the bet is such that Omega wins money, it must be “in the cor­rect di­rec­tion”). It finds the best bet you will take (if any), and offers you this bet.

Omicron does a similar thing, but in­stead looks for bets that you won’t take—bets which will in­stead cause you to up­date strongly in the wrong di­rec­tion, and not take the bet (since it doesn’t want to give you money). Again, it finds the best bet you won’t take (if any) and ap­proaches you.

You are ap­proached by some­one and offered a bet, al­though you don’t know if it’s Omega or Omicron. What should your policy be?

If you ac­cept a bet, the bet is likely to have come from Omega, and thus be ex­tremely costly for you. So you shouldn’t take the bet.

On the other hand, if you up­date strongly in the di­rec­tion of the bet, it most likely came from Omicron, and this means it’s prob­a­bly an up­date in the wrong di­rec­tion. So you shouldn’t up­date.

This leaves you in the per­plex­ing situ­a­tion of be­liev­ing that the bet is prob­a­bly ex­tremely good, but not want­ing to take it.