Double Illusion of Transparency

Fol­lowup to: Ex­plain­ers Shoot High, Illu­sion of Trans­parency

My first true foray into Bayes For Every­one was writ­ing An In­tu­itive Ex­pla­na­tion of Bayesian Rea­son­ing, still one of my most pop­u­lar works. This is the In­tu­itive Ex­pla­na­tion’s ori­gin story.

In De­cem­ber of 2002, I’d been ser­mo­niz­ing in a ha­bit­ual IRC chan­nels about what seemed to me like a very straight­for­ward idea: How words, like all other use­ful forms of thought, are se­cretly a dis­guised form of Bayesian in­fer­ence. I thought I was ex­plain­ing clearly, and yet there was one fel­low, it seemed, who didn’t get it. This wor­ried me, be­cause this was some­one who’d been very en­thu­si­as­tic about my Bayesian ser­mons up to that point. He’d gone around tel­ling peo­ple that Bayes was “the se­cret of the uni­verse”, a phrase I’d been known to use.

So I went into a pri­vate IRC con­ver­sa­tion to clear up the stick­ing point.

And he still didn’t get it.

I took a step back and ex­plained the im­me­di­ate pre­req­ui­sites, which I had thought would be ob­vi­ous -

He didn’t un­der­stand my ex­pla­na­tion of the pre­req­ui­sites.

In des­per­a­tion, I re­cursed all the way back to Bayes’s The­o­rem, the ul­ti­mate foun­da­tion stone of -

He didn’t know how to ap­ply Bayes’s The­o­rem to up­date the prob­a­bil­ity that a fruit is a ba­nana, af­ter it is ob­served to be yel­low. He kept mix­ing up p(b|y) and p(y|b).

It seems like a small thing, I know. It’s strange how small things can trig­ger ma­jor life-re­al­iza­tions. Any former TAs among my read­ers are prob­a­bly laugh­ing: I hadn’t re­al­ized, un­til then, that in­struc­tors got mis­lead­ing feed­back. Robin com­mented yes­ter­day that the best way to aim your ex­pla­na­tions is feed­back from the in­tended au­di­ence, “an ad­van­tage teach­ers of­ten have”. But what if self-an­chor­ing also causes you to over­es­ti­mate how much un­der­stand­ing ap­pears in your feed­back?

I fell prey to a dou­ble illu­sion of trans­parency. First, I as­sumed that my words meant what I in­tended them to mean—that my listen­ers heard my in­ten­tions as though they were trans­par­ent. Se­cond, when some­one re­peated back my sen­tences us­ing slightly differ­ent word or­der­ings, I as­sumed that what I heard was what they had in­tended to say. As if all words were trans­par­ent win­dows into thought, in both di­rec­tions.

I thought that if I said, “Hey, guess what I no­ticed to­day! Bayes’s The­o­rem is the se­cret of the uni­verse!”, and some­one else said, “Yes! Bayes’s The­o­rem is the se­cret of the uni­verse!”, then this was what a suc­cess­ful teacher-stu­dent in­ter­ac­tion looked like: knowl­edge con­veyed and ver­ified. I’d read Pir­sig and I knew, in the­ory, about how stu­dents learn to re­peat back what the teacher says in slightly differ­ent words. But I thought of that as a de­liber­ate tac­tic to get good grades, and I wasn’t grad­ing any­one.

This may sound odd, but un­til that very day, I hadn’t re­al­ized why there were such things as uni­ver­si­ties. I’d thought it was just rent-seek­ers who’d got­ten a lock on the cre­den­tial­ing sys­tem. Why would you need teach­ers to learn? That was what books were for.

But now a great and ter­rible light was dawn­ing upon me. Gen­uinely ex­plain­ing com­pli­cated things took months or years, and an en­tire uni­ver­sity in­fras­truc­ture with painstak­ingly crafted text­books and pro­fes­sional in­struc­tors. You couldn’t just tell peo­ple.

You’re laugh­ing at me right now, aca­demic read­ers; but think back and you’ll re­al­ize that aca­demics are gen­er­ally very care­ful not to tell the gen­eral pop­u­la­tion how difficult it is to ex­plain things, be­cause it would come across as con­de­scend­ing. Physi­cists can’t just say, “What we do is be­yond your com­pre­hen­sion, fool­ish mor­tal” when Congress is con­sid­er­ing their fund­ing. Richard Feyn­man once said that if you re­ally un­der­stand some­thing in physics you should be able to ex­plain it to your grand­mother. I be­lieved him. I was shocked to dis­cover it wasn’t true.

But once I re­al­ized, it be­came hor­ribly clear why no one had picked up and run with any of the won­der­ful ideas I’d been tel­ling about Ar­tifi­cial In­tel­li­gence.

If I wanted to ex­plain all these mar­velous ideas I had, I’d have to go back, and back, and back. I’d have to start with the things I’d figured out be­fore I was even think­ing about Ar­tifi­cial In­tel­li­gence, the foun­da­tions with­out which noth­ing else would make sense.

Like all that stuff I’d worked out about hu­man ra­tio­nal­ity, back at the dawn of time.

Which I’d con­sid­er­ably re­worked af­ter re­ceiv­ing my Bayesian En­light­en­ment. But ei­ther way, I had to start with the foun­da­tions. Noth­ing I said about AI was go­ing to make sense un­less I started at the be­gin­ning. My listen­ers would just de­cide that emer­gence was a bet­ter ex­pla­na­tion.

And the be­gin­ning of all things in the re­worked ver­sion was Bayes, to which there didn’t seem to be any de­cent on­line in­tro­duc­tion for new­bies. Most sources just stated Bayes’s The­o­rem and defined the terms. This, I now re­al­ized, was not go­ing to be suffi­cient. The on­line sources I saw didn’t even say why Bayes’s The­o­rem was im­por­tant. E. T. Jaynes seemed to get it, but Jaynes spoke only in calcu­lus—no hope for novices there.

So I men­tally con­signed ev­ery­thing I’d writ­ten be­fore 2003 to the trash heap—it was mostly ob­so­lete in the wake of my Bayesian En­light­en­ment, any­way—and started over at what I fondly con­ceived to be the be­gin­ning.

(It wasn’t.)

And I would ex­plain it so clearly that even grade school stu­dents would get it.

(They didn’t.)

I had, and have, much left to learn about ex­plain­ing. But that’s how it all be­gan.