You can still use the same positively-oriented brainstorming process for figuring out how to avoid bad outcomes. As soon as there’s even a vague idea of avoiding a very bad outcome, that becomes a very good reward prediction after taking the differential. The dopamine system does calculate such differentials, and it seems like the valance system, while probably different from direct reward prediction and more conceptual, should and could also take differentials in useful ways. Valance needs to at least somewhat dependent on context. I don’t think this requires unique mechanisms (although it might have them); it’s sufficient to learn variants of the concepts like “avoiding a really bad event” and then attaching valance to that concept variant.
Btw, there’s another simpler possible mechanism, though I don’t know the neuroscience and perhaps Steve’s hypothesis with separate valence assessors and involuntary attention control fits the neuroscience evidence much better and it may also fit observed motivated reasoning better.
But the obvious way to design a mind would be to make it just focus on whatever is most important, aka where most expected utility per necessary resources could be gained.
So we still have a learned value function which assigns how good/bad something would be, but we also have an estimator of how much the value would increase if we continue thinking (which might e.g. happen because one makes plans for making a somewhat bad situation better), and what gets attended on depends on this estimator, not the value function directly.
Interesting! I think that works.
You can still use the same positively-oriented brainstorming process for figuring out how to avoid bad outcomes. As soon as there’s even a vague idea of avoiding a very bad outcome, that becomes a very good reward prediction after taking the differential. The dopamine system does calculate such differentials, and it seems like the valance system, while probably different from direct reward prediction and more conceptual, should and could also take differentials in useful ways. Valance needs to at least somewhat dependent on context. I don’t think this requires unique mechanisms (although it might have them); it’s sufficient to learn variants of the concepts like “avoiding a really bad event” and then attaching valance to that concept variant.
Btw, there’s another simpler possible mechanism, though I don’t know the neuroscience and perhaps Steve’s hypothesis with separate valence assessors and involuntary attention control fits the neuroscience evidence much better and it may also fit observed motivated reasoning better.
But the obvious way to design a mind would be to make it just focus on whatever is most important, aka where most expected utility per necessary resources could be gained.
So we still have a learned value function which assigns how good/bad something would be, but we also have an estimator of how much the value would increase if we continue thinking (which might e.g. happen because one makes plans for making a somewhat bad situation better), and what gets attended on depends on this estimator, not the value function directly.