I may be misunderstanding but does this miss the category of unknown unknowns that are captured by Knightian uncertainty? For extremely out of distribution scenarios you may not be able to accurately anticipate even the general category of ways things could go wrong.
I can talk about what is in-distribution in terms of a bunch of finite components, and thereby name the cases that are out of distribution: those in which my components break.
(This seems like an advantage inside views have, they come with limits attached, because they build a distribution out of pieces that you can tell are broken when they don’t match reality.)
My example doesn’t talk about the probability I assign on “crazy thing I can’t model”, but such a thing would break something like my model of “who is doing what with the AI code by default”.
Maybe it would have been better of me to include a case for “and reality might invalidate my attempt at meta-reasoning too”?
I may be misunderstanding but does this miss the category of unknown unknowns that are captured by Knightian uncertainty? For extremely out of distribution scenarios you may not be able to accurately anticipate even the general category of ways things could go wrong.
Idk if it’s actually missing that?
I can talk about what is in-distribution in terms of a bunch of finite components, and thereby name the cases that are out of distribution: those in which my components break.
(This seems like an advantage inside views have, they come with limits attached, because they build a distribution out of pieces that you can tell are broken when they don’t match reality.)
My example doesn’t talk about the probability I assign on “crazy thing I can’t model”, but such a thing would break something like my model of “who is doing what with the AI code by default”.
Maybe it would have been better of me to include a case for “and reality might invalidate my attempt at meta-reasoning too”?
Your comment has made me think I’ve misunderstood something fundamental.
Hm, sorry! I don’t think a good reply on my part should do that :P
I think I’m rejecting a certain mental stance toward unknown-unknowns, and I don’t think I’m clearly pointing at it yet.