Is your uncertainty resolvable?

I was chat­ting with An­drew Critch about the idea of Re­acts on LessWrong.

Speci­fi­cally, the part where I thought there are par­tic­u­lar epistemic states that don’t have words yet, but should. And that a func­tion of LessWrong might be to make var­i­ous pos­si­ble epistemic states more salient as op­tions. You might have re­acts for “ap­prove/​dis­ap­prove” and “agree/​dis­agree”… but you might also want re­ac­tions that let you quickly and effortless ex­press “this isn’t ex­actly false or bad but it’s sub­tly mak­ing this dis­cus­sion worse.”

Fic­tion­al­ized, Para­phrased Critch said “hmm, this re­minds me of some par­tic­u­lar epistemic states I re­cently no­ticed that don’t have names.”

“Go on”, said I.

“So, you know the feel­ing of be­ing un­cer­tain? And how it feels differ­ent to be 60% sure of some­thing, vs 90%?”


“Okay. So here’s two other states you might be in:

  • 75% sure that you’ll even­tu­ally be 99% sure,

  • 80% sure that you’ll even­tu­ally be 90% sure.

He let me pro­cess those num­bers for a mo­ment.


Then he con­tinued: “Okay, now imag­ine you’re think­ing about a par­tic­u­lar AI sys­tem you’re de­sign­ing, which might or might not be al­ignable.

“If you’re feel­ing 75% sure that you’ll even­tu­ally be 99% sure that that AI is safe, this means you think that even­tu­ally you’ll have a clear un­der­stand­ing of the AI, such that you feel con­fi­dent turn­ing it on with­out de­stroy­ing hu­man­ity. More­over you ex­pect to be able to con­vince other peo­ple that it’s safe to turn it on with­out de­stroy­ing hu­man­ity.

“Whereas if you’re 80% sure that even­tu­ally you’ll be 90% sure that it’ll be safe, even in the fu­ture state where you’re bet­ter in­formed and more op­ti­mistic, you might still not ac­tu­ally be con­fi­dent enough to turn it on. And even if for some rea­son you are, other peo­ple might dis­agree about whether you should turn it on.

“I’ve no­ticed peo­ple track­ing how cer­tain they are of some­thing, with­out pay­ing at­ten­tion to whether their un­cer­tainty is pos­si­ble to re­solve. And this has im­por­tant ram­ifi­ca­tions for what kind of plans they can make. Some plans re­quire near-cer­tainty. Espe­cially many plans that re­quire group co­or­di­na­tion.

“Makes sense”, said I. “Can I write this up as a blog­post?”

I’m not quite sure about the best name here, but this seems like a use­ful con­cept to have a han­dle for. Some­thing like “un­re­solv­able un­cer­tainty?”