My (slightly vague) answer is that, somewhere in the cortex, you’ll find some signals that systematically distinguish viable-plan-thoughts from fantasizing-thoughts. (No comment on the exact nature of these signals, but we clearly have introspective access to this information, so it has to be in there somewhere.)
And there’s a learning algorithm that continuously updates the “valence guess” thought assessor, and very early in life, this learning algorithm will pick up on the fact that these fantasy-vs-plan indicator signals are importantly useful for predicting good outcomes.
Possible objection: By that logic, wouldn’t it learn that only viable-plan-thoughts are worthwhile, and fantasizing-thoughts are a waste of time? And yet, we continue to feel motivated to fantasize, all the way into adulthood! What’s the deal? My response: No, it would not learn that fantasizing-thoughts are a complete waste of time, because fantasizing-thoughts DO in fact (sometimes) lead directly to viable-plan-thoughts. So the algorithm would not learn that fantasizing is always a complete waste of time, although it might learn from experience that particular types of fantasizing are. Instead it would learn in general that fantasizing about good things has nonzero goodness, but viable plans towards those same things are better.
Good thought-provoking question!
My (slightly vague) answer is that, somewhere in the cortex, you’ll find some signals that systematically distinguish viable-plan-thoughts from fantasizing-thoughts. (No comment on the exact nature of these signals, but we clearly have introspective access to this information, so it has to be in there somewhere.)
And there’s a learning algorithm that continuously updates the “valence guess” thought assessor, and very early in life, this learning algorithm will pick up on the fact that these fantasy-vs-plan indicator signals are importantly useful for predicting good outcomes.
Possible objection: By that logic, wouldn’t it learn that only viable-plan-thoughts are worthwhile, and fantasizing-thoughts are a waste of time? And yet, we continue to feel motivated to fantasize, all the way into adulthood! What’s the deal? My response: No, it would not learn that fantasizing-thoughts are a complete waste of time, because fantasizing-thoughts DO in fact (sometimes) lead directly to viable-plan-thoughts. So the algorithm would not learn that fantasizing is always a complete waste of time, although it might learn from experience that particular types of fantasizing are. Instead it would learn in general that fantasizing about good things has nonzero goodness, but viable plans towards those same things are better.
Another part of the story is that, when we’re fantasizing, it’s often the case that the fantasy itself provides immediate ground-truth reward signals. Recall that we can trigger reward signals merely by thinking.
UPDATE 2026-12-19: I rewrote §7.5 a bit, thanks.