The basic idea of a trapped prior is purely epistemic. It can happen (in theory) even in someone who doesn’t feel emotions at all. If you gather sufficient evidence that there are no polar bears near you, and your algorithm for combining prior with new experience is just a little off, then you can end up rejecting all apparent evidence of polar bears as fake, and trapping your anti-polar-bear prior. This happens without any emotional component.
I either don’t follow your more general / “purely epistemic” point, or disagree. If a person’s algorithm is doing correct Bayesian epistemology, a low prior of polar bears won’t obscure the accumulating likelihood ratios in favor of polar bears; a given observation will just be classified as “an update in favor of polar bears maybe being a thing, though they’re still very very unlikely even after this datapoint”; priors don’t mess with the direction of the update.
I guess you’re trying to describe some other situation than correct Bayesian updating, when you talk about a person/alien/AI’s algorithm being “just a little off”. But I can’t figure out what kind of “a little off” you are imagining, that would yield this.
My (very non-confident) guess at what is going on with self-reinforcing prejudices/phobias/etc. in humans, is that it involves actively insulating some part of the mind from the data (as you suggest), and that this is not the sort of phenomenon that would happen with an algorithm that didn’t go out of its way to do something like compartmentalization.
If there’s a mechanism you’re proposing that doesn’t require compartmentalization, might you clarify it?
One interesting datum about willpower (which I’ve observed repeatedly in many people and contexts; not sure if it’s officially documented anywhere) is that it’s much easier to take a fully scripted action than to take an action that requires creatively filling in details.
For example, suppose several people are out trying to do “rejection therapy” (a perhaps-dubious game in which folks make requests of strangers that are likely to be rejected, e.g. “Can I listen to the walkman you’re listening to for a moment?” or “Can we trade socks?”). Many many people who set out to ask a stranger such a question will… “decide” not to, once the stranger is actually near them. However, people who have written down exactly which words they plan to stay in what order to exactly which stranger, with no room for ambiguity, are (anecdotally but repeatedly) more likely to actually say the words. (Or to do many other difficult habits, I think.)
(I originally noticed this pattern in undergrad, when there was a study group I wanted to leave, but whose keeper I felt flinchy about disappointing by leaving. I planned to leave the group and then … didn’t. And the next week, planned again to leave the group and then didn’t. And the third week, came in with a fully written exact sentence, said my scripted sentence, and left.)
Further examples:
It’s often easier to do (the dishes, or exercise, or other ‘difficult’ tasks) if there’s a set time for it.
Creativity-requiring tasks are often harder to attempt than more structured/directions-following-y tasks (e.g. writing poetry, especially if you’re “actually trying” at it; or attempting alignment research in a more “what actually makes sense here?” way and a less “let me make deductions from this framework other people are using” way; or just writing a blog post vs critiquing one).
— I’ve previously taken the above observations as evidence for the “subparts of my mind with differing predictions” view — if there are different bits of my (mind/brain) that are involved in e.g. assembling the sentence, vs saying the sentence, then if I need to figure out what words to say I’ll need whatever’s involved in calling the “assemble sentences” bit to also be on board, which is to say that more of me will need to be on board.
I guess you could also try this from the “Bayesian evidence” standpoint. But I’m curious how you’d do it in detail. Like, would you say the prior against “moving muscles” extends also for “assembling sentences”?