No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.
Less wrong model in less confused terminology: top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.
Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”. - This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”). - Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”
The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.
One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.
Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.
--
No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)
Stepping back for a moment, just want to clarify goal of this comment exchange. In drafting a reply, I realize I was mixing between:
1) determining whether the decision to curate was good or not 2) determining what is true (according to my own beliefs) 3) determining whether the post is “good” or not.
Of course 1) impacts 2) impacts 3).
I think I came in with LessWrong model you describe and the piece didn’t update me so much as seemed like a straightforward explainer of a simple point (“what people say is Good isn’t the same as your Values). I think you have a point that the post does something like set up one side of the dichotomy as S1 boxes, though it’s salient to me that it also has:
We don’t really know what human values are, or what shape they are, or even whether they’re A Thing at all. We don’t have trivial introspective access to our own values; sometimes we think we value a thing a lot, but realize in hindsight that we value it only a little.
That feels appropriately non-committal.
I agree there’s complexity around egregores/memeplexes and how it gets carved up.
It’s definitely not the bar for curation that everything in the post seems correct to the curator. I do think it should leave people better off than if they’d not read it. After this discussion, I’m less sure about this post. “Values are just the S1 boxes” seems so ridiculous to me that I wouldn’t expect anyone to think it, I don’t know. The egregore stuff feels much higher resolution than what this post is going for, though I think there’s interesting stuff to figure out there. I kind of like this post for having sparked that conversation, though perhaps it is a rehash that is tiresome to others.
No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.
Less wrong model in less confused terminology:
top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.
Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”.
- This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”).
- Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”
The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.
One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.
Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.
--
No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)
Stepping back for a moment, just want to clarify goal of this comment exchange. In drafting a reply, I realize I was mixing between:
1) determining whether the decision to curate was good or not
2) determining what is true (according to my own beliefs)
3) determining whether the post is “good” or not.
Of course 1) impacts 2) impacts 3).
I think I came in with LessWrong model you describe and the piece didn’t update me so much as seemed like a straightforward explainer of a simple point (“what people say is Good isn’t the same as your Values). I think you have a point that the post does something like set up one side of the dichotomy as S1 boxes, though it’s salient to me that it also has:
That feels appropriately non-committal.
I agree there’s complexity around egregores/memeplexes and how it gets carved up.
It’s definitely not the bar for curation that everything in the post seems correct to the curator. I do think it should leave people better off than if they’d not read it. After this discussion, I’m less sure about this post. “Values are just the S1 boxes” seems so ridiculous to me that I wouldn’t expect anyone to think it, I don’t know. The egregore stuff feels much higher resolution than what this post is going for, though I think there’s interesting stuff to figure out there. I kind of like this post for having sparked that conversation, though perhaps it is a rehash that is tiresome to others.