Probability and radical uncertainty

In the pre­vi­ous ar­ti­cle in this se­quence, I con­ducted a thought ex­per­i­ment in which sim­ple prob­a­bil­ity was not suffi­cient to choose how to act. Ra­tion­al­ity re­quired rea­son­ing about meta-prob­a­bil­ities, the prob­a­bil­ities of prob­a­bil­ities.

Re­lat­edly, luke­prog has a brief post that ex­plains how this mat­ters; a long ar­ti­cle by Hold­enKarnofsky makes meta-prob­a­bil­ity cen­tral to util­i­tar­ian es­ti­mates of the effec­tive­ness of char­i­ta­ble giv­ing; and Jonathan_Lee, in a re­ply to that, has used the same frame­work I pre­sented.

In my pre­vi­ous ar­ti­cle, I ran thought ex­per­i­ments that pre­sented you with var­i­ous col­ored boxes you could put coins in, gam­bling with un­cer­tain odds.

The last box I showed you was blue. I ex­plained that it had a fixed but un­known prob­a­bil­ity of a twofold pay­out, uniformly dis­tributed be­tween 0 and 0.9. The over­all prob­a­bil­ity of a pay­out was 0.45, so the ex­pec­ta­tion value for gam­bling was 0.9—a bad bet. Yet your op­ti­mal strat­egy was to gam­ble a bit to figure out whether the odds were good or bad.

Let’s con­tinue the ex­per­i­ment. I hand you a black box, shaped rather differ­ently from the oth­ers. Its sealed fa­ce­plate is carved with runic in­scrip­tions and el­dritch figures. “I find this one par­tic­u­larly in­ter­est­ing,” I say.

What is the pay­out prob­a­bil­ity? What is your op­ti­mal strat­egy?

In the frame­work of the pre­vi­ous ar­ti­cle, you have no knowl­edge about the in­sides of the box. So, as with the “sports­ball” case I an­a­lyzed there, your meta-prob­a­bil­ity curve is flat from 0 to 1.

The blue box also has a flat meta-prob­a­bil­ity curve; but these two cases are very differ­ent. For the blue box, you know that the curve re­ally is flat. For the black box, you have no clue what the shape of even the meta-prob­a­bil­ity curve is.

The re­la­tion­ship be­tween the blue and black boxes is the same as that be­tween the coin flip and sports­ball—ex­cept at the meta level!

So if we’re go­ing on in this style, we need to look at the dis­tri­bu­tion of prob­a­bil­ities of prob­a­bil­ities of prob­a­bil­ities. The blue box has a sharp peak in its meta-meta-prob­a­bil­ity (around flat­ness), whereas the black box has a flat meta-meta-prob­a­bil­ity.

You ought now to be a lit­tle un­easy. We are putting epicy­cles on epicy­cles. An in­finite regress threat­ens.

Maybe at this point you sud­denly re­con­sider the blue box… I told you that its meta-prob­a­bil­ity was uniform. But per­haps I was ly­ing! How re­li­able do you think I am?

Let’s say you think there’s a 0.8 prob­a­bil­ity that I told the truth. That’s the meta-meta-prob­a­bil­ity of a flat meta-prob­a­bil­ity. In the worst case, the ac­tual pay­out prob­a­bil­ity is 0, so the av­er­age just plain prob­a­bil­ity is 0.8 x 0.45 = 0.36. You can feed that worst case into your de­ci­sion anal­y­sis. It won’t dras­ti­cally change the op­ti­mal policy; you’ll just quit a bit ear­lier than if you were en­tirely con­fi­dent that the meta-prob­a­bil­ity dis­tri­bu­tion was uniform.

To get this re­ally right, you ought to make a best guess at the meta-meta-prob­a­bil­ity curve. It’s not just 0.8 of a uniform prob­a­bil­ity dis­tri­bu­tion, and 0.2 of zero pay­out. That’s the worst case. Even if I’m ly­ing, I might give you bet­ter than zero odds. How much bet­ter? What’s your con­fi­dence in your meta-meta-prob­a­bil­ity curve? Ought you to draw a meta-meta-meta-prob­a­bil­ity curve? Yikes!

Mean­while… that black box is rather sinister. See­ing it makes you won­der. What if I rigged the blue box so there is a small prob­a­bil­ity that when you put a coin in, it jabs you with a poi­son dart, and you die hor­ribly?

Ap­par­ently a zero pay­out is not the worst case, af­ter all! On the other hand, this seems para­noid. I’m odd, but prob­a­bly not that evil.

Still, what about the black box? You re­al­ize now that it could do any­thing.

  • It might spring open to re­veal a col­lec­tion of fos­sil trilo­bites.

  • It might play Corvus Co­rax’s Vitium in Opere at ear-split­ting vol­ume.

  • It might an­a­lyze the trace DNA you left on the coin and use it to write you a per­son­al­ized love poem.

  • It might emit a strip of pa­per with a recipe for dun­dun noo­dles writ­ten in Chi­nese.

  • It might sprout six me­chan­i­cal legs and jump into your lap.

What is the prob­a­bil­ity of its giv­ing you $2?

That no longer seems quite so rele­vant. In fact… it might be ut­terly mean­ingless! This is now a situ­a­tion of rad­i­cal un­cer­tainty.

What is your op­ti­mal strat­egy?

I’ll an­swer that later in this se­quence. You might like to figure it out for your­self now, though.

Fur­ther reading

The black box is an in­stance of Knigh­tian un­cer­tainty. That’s a catch-all cat­e­gory for any type of un­cer­tainty that can’t use­fully be mod­eled in terms of prob­a­bil­ity (or meta-prob­a­bil­ity!), be­cause you can’t make mean­ingful prob­a­bil­ity es­ti­mates. Cal­ling it “Knigh­tian” doesn’t help solve the prob­lem, be­cause there’s lots of sources of non-prob­a­bil­is­tic un­cer­tainty. How­ever, it’s use­ful to know that there’s a liter­a­ture on this.

The blue box is closely re­lated to Ells­berg’s para­dox, which com­bines prob­a­bil­ity with Knigh­tian un­cer­tainty. In­ter­est­ingly, it was in­vented by the same Daniel Ells­berg who re­leased the Pen­tagon Papers in 1971. I won­der how his work in de­ci­sion the­ory might have af­fected his de­ci­sion to leak the Papers?