I get that this post is skipping over a lot of the mechanistic details, but I’ll observe that that there are important behavioral differences between the two hypotheses...
The thought assessors award valence to a thought in proportion to how strongly it activates a high-yumminess concept.
The thought assessors award valence to a thought in proportion to their estimate (or the cortex’s own estimate?) of how likely the thought is to lead to actually experiencing the high-yumminess experience.
...because those two things come apart. There are some thoughts that will strongly activate a concept of an experience, but have low probability of leading to that experience, and vis versa.
As an example, I would bet that vividly imagining eating Prinsesstårta cake activates the concept “eating Prinsesstårta cake” much more going to google maps, so that you can find an ATM, so that you can get cash, so that you can go to the store and buy eggs and milk, so that you can bake a Prinsesstårta cake, so you can eat it. But the second thought seems more likely to result in you actually eating the cake!
Though, in practice, it seems like both kinds of behavior (vividly fantasizing and making multi-step plans) both occur, some of the time, so it might be both, or something similar enough to both.
My (slightly vague) answer is that, somewhere in the cortex, you’ll find some signals that systematically distinguish viable-plan-thoughts from fantasizing-thoughts. (No comment on the exact nature of these signals, but we clearly have introspective access to this information, so it has to be in there somewhere.)
And there’s a learning algorithm that continuously updates the “valence guess” thought assessor, and very early in life, this learning algorithm will pick up on the fact that these fantasy-vs-plan indicator signals are importantly useful for predicting good outcomes.
Possible objection: By that logic, wouldn’t it learn that only viable-plan-thoughts are worthwhile, and fantasizing-thoughts are a waste of time? And yet, we continue to feel motivated to fantasize, all the way into adulthood! What’s the deal? My response: No, it would not learn that fantasizing-thoughts are a complete waste of time, because fantasizing-thoughts DO in fact (sometimes) lead directly to viable-plan-thoughts. So the algorithm would not learn that fantasizing is always a complete waste of time, although it might learn from experience that particular types of fantasizing are. Instead it would learn in general that fantasizing about good things has nonzero goodness, but viable plans towards those same things are better.
I get that this post is skipping over a lot of the mechanistic details, but I’ll observe that that there are important behavioral differences between the two hypotheses...
The thought assessors award valence to a thought in proportion to how strongly it activates a high-yumminess concept.
The thought assessors award valence to a thought in proportion to their estimate (or the cortex’s own estimate?) of how likely the thought is to lead to actually experiencing the high-yumminess experience.
...because those two things come apart. There are some thoughts that will strongly activate a concept of an experience, but have low probability of leading to that experience, and vis versa.
As an example, I would bet that vividly imagining eating Prinsesstårta cake activates the concept “eating Prinsesstårta cake” much more going to google maps, so that you can find an ATM, so that you can get cash, so that you can go to the store and buy eggs and milk, so that you can bake a Prinsesstårta cake, so you can eat it. But the second thought seems more likely to result in you actually eating the cake!
Though, in practice, it seems like both kinds of behavior (vividly fantasizing and making multi-step plans) both occur, some of the time, so it might be both, or something similar enough to both.
Good thought-provoking question!
My (slightly vague) answer is that, somewhere in the cortex, you’ll find some signals that systematically distinguish viable-plan-thoughts from fantasizing-thoughts. (No comment on the exact nature of these signals, but we clearly have introspective access to this information, so it has to be in there somewhere.)
And there’s a learning algorithm that continuously updates the “valence guess” thought assessor, and very early in life, this learning algorithm will pick up on the fact that these fantasy-vs-plan indicator signals are importantly useful for predicting good outcomes.
Possible objection: By that logic, wouldn’t it learn that only viable-plan-thoughts are worthwhile, and fantasizing-thoughts are a waste of time? And yet, we continue to feel motivated to fantasize, all the way into adulthood! What’s the deal? My response: No, it would not learn that fantasizing-thoughts are a complete waste of time, because fantasizing-thoughts DO in fact (sometimes) lead directly to viable-plan-thoughts. So the algorithm would not learn that fantasizing is always a complete waste of time, although it might learn from experience that particular types of fantasizing are. Instead it would learn in general that fantasizing about good things has nonzero goodness, but viable plans towards those same things are better.
Another part of the story is that, when we’re fantasizing, it’s often the case that the fantasy itself provides immediate ground-truth reward signals. Recall that we can trigger reward signals merely by thinking.
UPDATE 2026-12-19: I rewrote §7.5 a bit, thanks.