I agree there is an important distinction, but I think the social memetic aspect of “Goodness” is not central. The central distinction is that we have access to yumminess directly, it is the only thing we “truly care about” in some sense, but as bounded and not even perfectly coherent agents, we’re unable to roll our predictions forward over all possible action paths and maximize yumminess.
Instead we need to form a compact /abstracted representation of our values/yuminess to 1) make them legible to ourselves and 2) make plans to attain them 3) communicate them 4) make them more coherent
I agree there is an important distinction, but I think the social memetic aspect of “Goodness” is not central. The central distinction is that we have access to yumminess directly, it is the only thing we “truly care about” in some sense, but as bounded and not even perfectly coherent agents, we’re unable to roll our predictions forward over all possible action paths and maximize yumminess.
Instead we need to form a compact /abstracted representation of our values/yuminess to 1) make them legible to ourselves and 2) make plans to attain them 3) communicate them 4) make them more coherent