Willing gamblers, spherical cows, and AIs

Note: post­ing this in Main rather than Dis­cus­sion in light of re­cent dis­cus­sion that peo­ple don’t post in Main enough and their rea­sons for not do­ing so aren’t nec­es­sar­ily good ones. But I sus­pect I may be rein­vent­ing the wheel here, and some­one else has in fact got­ten farther on this prob­lem than I have. If so, I’d be very happy if some­one could point me to ex­ist­ing dis­cus­sion of the is­sue in the com­ments.

tldr; Gam­bling-based ar­gu­ments in the philos­o­phy of prob­a­bil­ity can be seen as de­pend­ing on a con­ve­nient sim­plifi­ca­tion of as­sum­ing peo­ple are far more will­ing to gam­ble than they are in real life. Some jus­tifi­ca­tions for this sim­plifi­ca­tion can be given, but it’s un­clear to me how far they can go and where the jus­tifi­ca­tion starts to be­come prob­le­matic.

In “In­tel­li­gence Ex­plo­sion: Ev­i­dence and Im­port,” Luke and Anna men­tion the fact that, “Ex­cept for weather fore­cast­ers (Mur­phy and Win­kler 1984), and suc­cess­ful pro­fes­sional gam­blers, nearly all of us give in­ac­cu­rate prob­a­bil­ity es­ti­mates...” When I read this, it struck me as an odd thing to say in a pa­per on ar­tifi­cial in­tel­li­gence. I mean, those of us who are not pro­fes­sional ac­coun­tants tend to make book­keep­ing er­rors, and those of us who are not math, physics, en­g­ineer­ing, or eco­nomics ma­jors make mis­takes on GRE quant ques­tions that we were sup­posed to have learned how to do in our first two years of high school. Why fo­cus on this par­tic­u­lar hu­man failing?

A re­lated point can be made about Dutch Book Ar­gu­ments in the philos­o­phy of prob­a­bil­ity. Dutch Book Ar­gu­ments claim, in a nut­shell, that you should rea­son in ac­cor­dance with the ax­ioms of prob­a­bil­ity be­cause if you don’t, a clever bookie will be able to take all your money. But an­other way to pre­vent a clever bookie from tak­ing all your money is to not gam­ble. Which many peo­ple don’t, or at least do rarely.

Dutch Book Ar­gu­ments seem to im­plic­itly make what we might call the “will­ing gam­bler as­sump­tion”: ev­ery­one always has a pre­cise prob­a­bil­ity as­sign­ment for ev­ery propo­si­tion, and they’re will­ing to take any bet which has a non-nega­tive ex­pected value given their prob­a­bil­ity as­sign­ments. (Or per­haps: ev­ery­one is always will­ing to take at least one side of any pro­posed bet.) Need­less to say, even peo­ple who gam­ble a lot gen­er­ally aren’t that ea­ger to gam­ble.

So how does any­one get away with us­ing Dutch Book ar­gu­ments for any­thing? A plau­si­ble an­swer comes from a joke Luke re­cently told in his ar­ti­cle on Fermi es­ti­mates:

Milk pro­duc­tion at a dairy farm was low, so the farmer asked a lo­cal uni­ver­sity for help. A mul­ti­dis­ci­plinary team of pro­fes­sors was as­sem­bled, headed by a the­o­ret­i­cal physi­cist. After two weeks of ob­ser­va­tion and anal­y­sis, the physi­cist told the farmer, “I have the solu­tion, but it only works in the case of spher­i­cal cows in a vac­uum.”

If you’ve stud­ied physics, you know that physi­cists don’t just use those kinds of ap­prox­i­ma­tions when do­ing Fermi es­ti­mates; of­ten they can be counted on to yield re­sults that are in fact very close to re­al­ity. So maybe the will­ing gam­bler as­sump­tion works as a sort of spher­i­cal cow, that al­lows philoso­phers work­ing on is­sues re­lated to prob­a­bil­ity to gen­er­ate im­por­tant re­sults in spite of the un­re­al­is­tic na­ture of the as­sump­tion.

Some parts of how this would work are fairly clear. In real life, bets have trans­ac­tion costs; they take time and effort to set up and col­lect. But it doesn’t seem too bad to ig­nore that fact in thought ex­per­i­ments. Similarly, in real life money has de­clin­ing marginal util­ity; the util­ity of dou­bling your money is less than the di­su­til­ity of los­ing your last dol­lar. In prin­ci­ple, if you know some­one’s util­ity func­tion over money, you can take a bet with zero ex­pected value in dol­lar terms and re­place it with a bet that has zero ex­pected value in util­ity terms. But ig­nor­ing that and just us­ing dol­lars for your thought ex­per­i­ments seems like an ac­cept­able sim­plifi­ca­tion for con­ve­nience’s sake.

Even mak­ing those as­sump­tions so that it isn’t definitely harm­ful to ac­cept bets with zero ex­pected (dol­lar) value, we might still won­der why our spher­i­cal cow gam­bler should ac­cept them. An­swer: be­cause if nec­es­sary you could just add one penny to the side of the bet you want the gam­bler to take, but always hav­ing to men­tion the ex­tra penny is an­noy­ing, so you may as well as­sume the gam­bler takes any bet with non-nega­tive ex­pected value rather than re­quire pos­i­tive ex­pected value.

Another thing that keeps peo­ple from gam­bling more in real life is the prin­ci­ple that if you can’t spot the sucker in the room, it’s prob­a­bly you. If you’re un­sure whether an offered bet is fa­vor­able to you, the mere fact that some­one is offer­ing it to you is pretty strong ev­i­dence that it’s in their fa­vor. One way to avoid this prob­lem is to stipu­late that in Dutch Book Ar­gu­ments, we just as­sume the bookie doesn’t know any­thing more about what­ever the bets are about than the per­son be­ing offered the bet, and the per­son be­ing offered the bet knows this. The bookie has to con­struct her book pri­mar­ily based on know­ing the propen­si­ties of the other per­son to bet. Nick Bostrom ex­plic­itly makes such an as­sump­tion in a pa­per on the sleep­ing beauty prob­lem. Maybe other peo­ple ex­plic­itly make this as­sump­tion, I don’t know.

In this last case, though, it’s not to­tally clear whether limit­ing the bookie’s knowl­edge is all you need to bridge the gap be­tween the will­ing gam­bler as­sump­tion and how peo­ple be­have in real life. In real life, peo­ple don’t of­ten make very ex­act prob­a­bil­ity as­sign­ments, and may be aware of their con­fu­sion about how to make ex­act prob­a­bil­ity as­sign­ments. Given that, it seems rea­son­able to hes­i­tate in mak­ing bets (even if you ig­nore trans­ac­tion costs and de­clin­ing marginal util­ity and know that the bookie doesn’t know any more about the sub­ject of the bet than you do), be­cause you’d still know the bookie might be try­ing to ex­ploit your con­fu­sion over how to make ex­act prob­a­bil­ity as­sign­ments.

At an even sim­pler level, you might adopt a rule, “be­fore mak­ing mul­ti­ple bets on re­lated ques­tions, check to make sure you aren’t guaran­tee­ing you’ll lose money.” After all, real book­ies offer odds such that if any­one was stupid enough to bet on each side of a ques­tion with the same bookie, they’d be guaran­teed to lose money. In a sense, book­ies could be in­ter­preted as “money pump­ing” the pub­lic as a whole. But some­how, it turns out that any sin­gle in­di­vi­d­ual will rarely be stupid enough to take both sides of the same bet from the same bookie, in spite of the fact that they’re ap­par­ently ir­ra­tional enough to be gam­bling in the first place.

In the end, I’m con­fused about how use­ful the will­ing gam­bler as­sump­tion re­ally is when do­ing philos­o­phy of prob­a­bil­ity. It cer­tainly seems like worth­while work gets done based on it, but just how ap­pli­ca­ble are those re­sults to real life? How do we tell when we should re­ject a re­sult be­cause the will­ing gam­bler as­sump­tion causes prob­lems in that par­tic­u­lar case? I don’t know.

One pos­si­ble jus­tifi­ca­tion for the will­ing gam­bler as­sump­tion is that even those of us who don’t liter­ally gam­ble, ever, still must make de­ci­sions where the out­come is not cer­tain, and we there­fore we need to do a de­cent job of mak­ing prob­a­bil­ity as­sign­ments for those situ­a­tions. But there are lots of peo­ple who are suc­cess­ful at their cho­sen field (in­clud­ing in fields that re­quire de­ci­sions with un­cer­tain out­comes) who aren’t weather fore­cast­ers or pro­fes­sional gam­blers, and there­fore can be ex­pected to make in­ac­cu­rate prob­a­bil­ity es­ti­mates. Con­versely, it doesn’t seem that the skills ac­quired by suc­cess­ful pro­fes­sional gam­blers give them much of an edge in other fields. There­fore, it seems that the re­la­tion­ship be­tween be­ing able to make ac­cu­rate prob­a­bil­ity es­ti­mates and suc­cess in fields that don’t speci­fi­cally re­quire them is weak.

Another jus­tifi­ca­tion for purs­ing lines of in­quiry based on the will­ing gam­bler as­sump­tion, a jus­tifi­ca­tion that will be par­tic­u­larly salient for peo­ple on LessWrong, is that if we want to build an AI based on an ideal­iza­tion of how ra­tio­nal agents think (Bayesi­anism or what­ever), we need tools like the will­ing gam­bler as­sump­tion to figure out how to get the ideal­iza­tion right. That sounds like a plau­si­ble thought at first. But if we flawed hu­mans have any hope of build­ing a good AI, it seems like an AI that’s as flawed as (but no more flawed than) hu­mans should also have a hope of self-im­prov­ing into some­thing bet­ter. An AI might be pro­grammed in a way that makes it a bad gam­bler, but aware of this limi­ta­tion, and left to de­cide for it­self whether, when it self-im­proves, it wants to fo­cus on im­prov­ing its gam­bling abil­ity or im­prov­ing other as­pects of it­self.

As some­one who cares a lot about AI, this is­sue of just how use­ful var­i­ous ideal­iza­tions are for think­ing about AI and pos­si­bly pro­gram­ming an AI one day are es­pe­cially im­por­tant to me. Un­for­tu­nately, I’m not sure what to say about them, so at this point I’ll turn the ques­tion over to the com­ments.