Our subconscious is “rational” if rational is defined as “winning”. True, it’s winning at something you don’t identify as “your” utility function. But the claim I’m making is that subconscious mechanisms have some winning strategies open to them, that conscious strategies don’t. The question whether it’s implementing “your” utility function or not is a different question.
Our subconscious may be somewhat optimized for the task of increasing inclusive genetic fitness but I doubt it’s optimal, evolution is stupid and gets stuck in local maxima all the time. There are probably points in the space of all possible subconsciouses that would do much better, especially since we are in quite a different environment to the one it was optimized for.
I disagree. I don’t think the subconscious computes any expected utility at all. I think it’s in large part a dumb set of heuristics and reflexes that isn’t particularly good at ‘winning’ in its current environment. Sure, it’s true that “subconscious mechanisms have some winning strategies open to them that conscious strategies don’t”, but that’s a much weaker claim.
That it’s using a different utility function is important to remember and nonobvious in the great-grandparent (we wouldn’t say it’s good to be taken over by a brainslug simply because the brainslug is better at achieving its goals than we are at achieving ours).
Our subconscious is “rational” if rational is defined as “winning”. True, it’s winning at something you don’t identify as “your” utility function. But the claim I’m making is that subconscious mechanisms have some winning strategies open to them, that conscious strategies don’t. The question whether it’s implementing “your” utility function or not is a different question.
Our subconscious may be somewhat optimized for the task of increasing inclusive genetic fitness but I doubt it’s optimal, evolution is stupid and gets stuck in local maxima all the time. There are probably points in the space of all possible subconsciouses that would do much better, especially since we are in quite a different environment to the one it was optimized for.
I disagree. I don’t think the subconscious computes any expected utility at all. I think it’s in large part a dumb set of heuristics and reflexes that isn’t particularly good at ‘winning’ in its current environment. Sure, it’s true that “subconscious mechanisms have some winning strategies open to them that conscious strategies don’t”, but that’s a much weaker claim.
That it’s using a different utility function is important to remember and nonobvious in the great-grandparent (we wouldn’t say it’s good to be taken over by a brainslug simply because the brainslug is better at achieving its goals than we are at achieving ours).