# Estimate Stability

I’ve been try­ing to get clear on some­thing you might call “es­ti­mate sta­bil­ity.” Steven Kaas re­cently posted my ques­tion to Stack­Ex­change, but we might as well post it here as well:

I’m try­ing to rea­son about some­thing I call “es­ti­mate sta­bil­ity,” and I’m hop­ing you can tell me whether there’s some rele­vant tech­ni­cal lan­guage...
What do I mean by “es­ti­mate sta­bil­ity?” Con­sider these three differ­ent propo­si­tions:
1. We’re 50% sure that a coin (known to be fair) will land on heads.

2. We’re 50% sure that Matt will show up at the party.

3. We’re 50% sure that Strong AI will be in­vented by 2080.

Th­ese es­ti­mates feel differ­ent. One rea­son they feel differ­ent is that the es­ti­mates have differ­ent de­grees of “sta­bil­ity.” In case (1) we don’t ex­pect to gain in­for­ma­tion that will change our prob­a­bil­ity es­ti­mate. But for cases (2) and (3), we may well come upon some in­for­ma­tion that causes us to ad­just the es­ti­mate ei­ther up or down.
So es­ti­mate (1) is more “sta­ble,” but I’m not sure how this should be quan­tified. Should I think of it in terms of run­ning a Monte Carlo simu­la­tion of what fu­ture ev­i­dence might be, and look­ing at some­thing like the var­i­ance of the dis­tri­bu­tion of the re­sult­ing es­ti­mates? What hap­pens when it’s a whole prob­a­bil­ity dis­tri­bu­tion for e.g. the time Strong AI is in­vented? (Do you do calcu­late the sta­bil­ity of the prob­a­bil­ity den­sity for ev­ery year, then av­er­age the re­sult?)
Here are some other con­sid­er­a­tions that would be use­ful to re­late more for­mally to con­sid­er­a­tions of es­ti­mate sta­bil­ity:
• If we’re es­ti­mat­ing some vari­able, hav­ing a nar­row prob­a­bil­ity dis­tri­bu­tion (prior to fu­ture ev­i­dence with re­spect to which we’re try­ing to as­sess the sta­bil­ity) cor­re­sponds to hav­ing a lot of data. New data, in that case, would make less of a con­tri­bu­tion in terms of chang­ing the mean and re­duc­ing the var­i­ance.

• There are differ­ences in model un­cer­tainty be­tween the three cases. I know what model to use when pre­dict­ing a coin flip. My method of pre­dict­ing whether Matt will show up at a party is shak­ier, but I have some idea of what I’m do­ing. With the Strong AI case, I don’t re­ally have any good idea of what I’m do­ing. Pre­sum­ably model un­cer­tainty is re­lated to es­ti­mate sta­bil­ity, be­cause the more model un­cer­tainty we have, the more we can change our es­ti­mate by re­duc­ing our model un­cer­tainty.

• Another differ­ence be­tween the three cases is the de­gree to which our ac­tions al­low us to im­prove our es­ti­mates, in­creas­ing their sta­bil­ity. For ex­am­ple, we can re­duce the un­cer­tainty and in­crease the sta­bil­ity of our es­ti­mate about Matt by call­ing him, but we don’t re­ally have any good ways to get bet­ter es­ti­mates of Strong AI timelines (other than by wait­ing).

• Value-of-in­for­ma­tion af­fects how we should deal with de­lay. Es­ti­mates that are un­sta­ble in the face of ev­i­dence we ex­pect to get in the fu­ture seem to im­ply higher VoI. This cre­ates a rea­son to ac­cept de­lays in our ac­tions. Or if we can eas­ily gather in­for­ma­tion that will make our es­ti­mates more ac­cu­rate and sta­ble, that means we have more rea­son to pay the cost of gath­er­ing that in­for­ma­tion. If we ex­pect to for­get in­for­ma­tion, or ex­pect our fu­ture selves not to take in­for­ma­tion into ac­count, dy­namic in­con­sis­tency be­comes im­por­tant. This is an­other rea­son why es­ti­mates might be un­sta­ble. One pos­si­ble strat­egy here is to pre­com­mit to have our es­ti­mates regress to the mean.

Thanks for any thoughts!
• There’s a chap­ter of Jaynes called “The A_p dis­tri­bu­tion and Rule of Suc­ces­sion” that an­swers this ques­tion.

• Se­conded.

• Given $X$ a ran­dom vari­able with prob­a­bil­ity dis­tri­bu­tion $\\rho$, the prob­a­bil­ity dis­tri­bu­tion $\\rho\_t$ you will have about $X$ af­ter time $t$ is a ran­dom vari­able in it­self. It must satisfy $E$\\rho\_t$=\\rho$. The (un)sta­bil­ity pa­ram­e­ter you are look­ing for sounds like $E\[D\_\{KL\}\(\\rho\_t||\\rho\$]), where $D\_\{KL\}$ stands for Kul­lback–Leibler di­ver­gence. The mean­ing of this pa­ram­e­ter is the ex­pected num­ber of bits of in­for­ma­tion about $X$ you will re­ceive over pe­riod $t$.

• The first two sen­tences of this com­ment are a very good for­mal (re-) state­ment of the idea of Con­ser­va­tion of Ex­pected Ev­i­dence.

• That’s an in­ter­est­ing ap­proach, but I’m not re­ally sure it’s what Luke’s af­ter. He seems to be talk­ing about closer to Knigh­tian un­cer­tainty and out-of-sam­ple er­ror; given a spe­cific model of AI risk over time, I sup­pose you could figure out how many bits you re­ceive per time pe­riod and calcu­late such a num­ber, but I think Luke is ask­ing a ques­tion more like ‘how much re­li­a­bil­ity do I have that this model is cap­tur­ing any­thing mean­ingful about the real dy­nam­ics? are the re­sults be­ing driven by one par­tic­u­lar as­sump­tion or some small un­re­li­able set of dat­a­points? Is this set of pre­dic­tions just overfit­ting?’ One of his points:

There are differ­ences in model un­cer­tainty be­tween the three cases. I know what model to use when pre­dict­ing a coin flip. My method of pre­dict­ing whether Matt will show up at a party is shak­ier, but I have some idea of what I’m do­ing. With the Strong AI case, I don’t re­ally have any good idea of what I’m do­ing. Pre­sum­ably model un­cer­tainty is re­lated to es­ti­mate sta­bil­ity, be­cause the more model un­cer­tainty we have, the more we can change our es­ti­mate by re­duc­ing our model un­cer­tainty.

• It’s always a con­cern in Bayesian rea­son­ing whether you’re us­ing a sen­si­ble prior. The­o­ret­i­cally you should always start with the Solomonoff prior and up­date from there but im­ple­ment­ing it in prac­tice is difficult, to say the least. How­ever, if we wish to stay in the realm of math­e­mat­i­cal for­mal­ism (I think Knigh­tian un­cer­tainty lies out­side of it by defi­ni­tion?) then the pa­ram­e­ter I sug­gested is sen­si­ble. In par­tic­u­lar the re­la­tion­ship be­tween model un­cer­tainty and es­ti­mate sta­bil­ity is well-cap­tured by this pa­ram­e­ter. For ex­am­ple sup­pose you have three pos­si­ble mod­els of strong AI de­vel­op­ment M1, M2 and M3 and you have a meta-model which as­signs them prob­a­bil­ities p1, p2 and p3. Then your prob­a­bil­ity dis­tri­bu­tion is the con­vex lin­ear com­bi­na­tion of the prob­a­bil­ity dis­tri­bu­tions as­signed by M1, M2 and M3 with co­effi­cients p1, p2 and p3. Now, if dur­ing time pe­riod t you ex­pect to learn which of these mod­els is the right one then my pa­ram­e­ter will show the re­sult­ing “un­sta­bil­ity”.

• The­o­ret­i­cally you should always start with the Solomonoff prior and up­date from there but im­ple­ment­ing it in prac­tice is difficult, to say the least.

Yes, but you can check your mod­els in a va­ri­ety of ways. You can test your in­ferred re­sults from your dataset by do­ing boot­strap­ping or cross-val­i­da­tion, and see how of­ten your re­sult changed (co­effi­cients or es­ti­ma­tion ac­cu­racy etc). To step up a level, you can set pa­ram­e­ters in your model to differ­ing val­ues based on hy­per­pa­ram­e­ters, and see how each of the var­i­ants on the model performs on the data (and then you can boot­strap/​cross-val­i­date each of the pos­si­ble mod­els as well), and then see how sen­si­tive your re­sults are to spe­cific pa­ram­e­ters like, yes, what­ever pri­ors you were feed­ing in. You can have fam­i­lies of mod­els, like pit­ting lo­gis­tic re­gres­sion mod­els against ran­dom forests, and you can see how of­ten they differ as an­other form of sen­si­tivity (and then you can vary the hy­per­pa­ram­e­ters in each model and then boot­strap/​cross-val­i­date with each pos­si­ble model). You can have en­sem­bles of mod­els from var­i­ous fam­i­lies and ob­vi­ously vary which mod­els are picked and what weights are put on them… and there my knowl­edge pe­ters out.

But while you still would not have come close to what a Solomonoff ap­proach might do, you have still learned a great deal about your model’s re­li­a­bil­ity in a way that I can’t see as hav­ing any con­nec­tion with your time and KL-re­lated ap­proach.

• But while you still would not have come close to what a Solomonoff ap­proach might do, you have still learned a great deal about your model’s re­li­a­bil­ity in a way that I can’t see as hav­ing any con­nec­tion with your time and KL-re­lated ap­proach.

I think there is a con­nec­tion. Namely, the meth­ods you men­tioned are pos­si­ble mechanisms of a learn­ing pro­cess but $E\[D\_\{KL\}\(\\rho\_t||\\rho\$]) is a quan­tifi­ca­tion of the ex­pected im­pact of this learn­ing pro­cess.

• Yes, I see what you mean—the mean/​ex­pec­ta­tion of how big the di­ver­gence be­tween our cur­rent prob­a­bil­ity dis­tri­bu­tion and the fu­ture prob­a­bil­ity dis­tri­bu­tion—but this seems like a post hoc or purely de­scrip­tive ap­proach: how do we es­ti­mate how much di­ver­gence there may be?

Hav­ing got­ten es­ti­mates of fu­ture di­ver­gence, quan­tify­ing the di­ver­gence may then be use­ful, but it seems like putting the horse be­fore the cart to start with your mea­sure.

• I guess sub­jec­tive logic is also try­ing to han­dle this kind of thing. From Jøsang’s book draft:

Sub­jec­tive logic is a type of prob­a­bil­is­tic logic that al­lows prob­a­bil­ity val­ues to be ex­pressed with de­grees of un­cer­tainty. The idea of prob­a­bil­is­tic logic is to com­bine the strengths of logic and prob­a­bil­ity calcu­lus, mean­ing that it has bi­nary logic’s ca­pac­ity to ex­press struc­tured ar­gu­ment mod­els, and it has the power of prob­a­bil­ities to ex­press de­grees of truth of those ar­gu­ments. The idea of sub­jec­tive logic is to ex­tend prob­a­bil­is­tic logic by also ex­press­ing un­cer­tainty about the prob­a­bil­ity val­ues them­selves, mean­ing that it is pos­si­ble to rea­son with ar­gu­ment mod­els in pres­ence of un­cer­tain or in­com­plete ev­i­dence.

Though maybe this par­tic­u­lar for­mal sys­tem has re­ally un­de­sir­able prop­er­ties, I don’t know.

• In case of the coin (I am as­sum­ing a coin that is mak­ing a fairly large num­ber of bounces when it lands, so that it can’t be eas­ily bi­ased with trick toss­ing), you have high con­fi­dence in a pro­cess (coin bounc­ing around) which maps ini­tial coin po­si­tion and ori­en­ta­tion into fi­nal ori­en­ta­tion in such a way that the fi­nal dis­tri­bu­tion is al­most in­de­pen­dent on your prior for coin ori­en­ta­tion and po­si­tion. I.e. you have a highly ob­jec­tive prob­a­bil­ity of 0.5 , in the sense that it is not de­pen­dent on sub­jec­tive, ar­bi­trary quan­tifi­ca­tion of un­known.

If you were to as­sign a prob­a­bil­ity of 0.7 in­stead, you’d have to en­tirely change the way physics works for the coin, or adopt a very ridicu­lous prior over coin’s ini­tial ori­en­ta­tion, which in­volves coin’s own mo­tion as part of the prior.

Mean­while, in the case of “We’re 50% sure that Strong AI will be in­vented by 2080.” , it’s a num­ber en­tirely pul­led out of your ass. You could of pul­led 10%, or 90%, de­pend­ing to what suits you best, with­out chang­ing any­thing even about your knowl­edge of the world. Num­bers pul­led out of your ass have an in­ter­est­ing, em­piri­cally ver­ifi­able prop­erty that max­i­miza­tion of their prod­ucts does not tend to ac­tu­ally re­sult in win, ir­re­spec­tive of the la­bels that you give those num­bers.

• Thanks for work­ing this up. I’ve thought for a long time that there was some­thing fishy about the lack of er­ror bars, and if I re­mem­ber cor­rectly, I was told that the er­ror bars were some­how sub­sumed in the per­centage. Maybe there should be more than one kind of er­ror bar.

• I was told that the er­ror bars were some­how sub­sumed in the per­centage.

They sort of are. In the long run, if your per­centages aren’t cal­ibrated, you’re do­ing some­thing wrong; is­sues of sen­si­tivity and ro­bust­ness are one form of er­ror among oth­ers and sub­sumed un­der the grand ul­ti­mate rubric—if you did your ar­ith­metic wrong, you can ex­pect to be un­cal­ibrated; if you over­es­ti­mated the qual­ity of your data, you can ex­pect to be un­cal­ibrated; if you used rigid mod­els which re­quired pow­er­ful as­sump­tions which of­ten are vi­o­lated in prac­tice, you can ex­pect to be un­cal­ibrated. And so on.

Maybe there should be more than one kind of er­ror bar.

Every sum­mary statis­tic is go­ing to cause prob­lems some­how com­pared to some­thing like a pos­te­rior dis­tri­bu­tion. One er­ror bar won’t cover all the ques­tions one might want to ask, and it’s not clear what er­ror bars you want in ad­vance. (Statis­tics and ma­chine learn­ing seem to be mov­ing to­wards en­sem­bles of mod­els and hi­er­ar­chi­cal ap­proaches like mod­els over mod­els and so forth, where one can vary all the knobs in gen­eral and see how the fi­nal an­swers perform, but ‘perform’ is go­ing to be defined differ­ently in differ­ent places.)

• There’s also Skyrms’ 1980 pa­per “Higher or­der de­grees of be­lief” and Gaif­man’s 1988 pa­per “A the­ory of higher or­der prob­a­bil­ities.”