I think the intuition for why the AI is wrong rests on the humans having extra structure that tangles everything together. They’re not actually following Bayesian uncertainty about some platonic “right thing,” instead they want to follow some “good process” (the process they believe will disambiguate puppies and rainbows), and if they’d built the AI correctly it wouldn’t reason using Bayesian uncertainty either, it would just follow the good process.
In the hypothetical where the humans don’t have this extra structure, updateless reasoning seems great.
I think the intuition for why the AI is wrong rests on the humans having extra structure that tangles everything together. They’re not actually following Bayesian uncertainty about some platonic “right thing,” instead they want to follow some “good process” (the process they believe will disambiguate puppies and rainbows), and if they’d built the AI correctly it wouldn’t reason using Bayesian uncertainty either, it would just follow the good process.
In the hypothetical where the humans don’t have this extra structure, updateless reasoning seems great.