Curated. While in my personal language, I would have treated Goodness as a synonym for Human Values[1], the distinction John is making here is correct, plus his advice on how to approach it. A very important point I have noticed is that when people ask (or anguish), “am I a good person?” this is asking according to the social egregore sense of good – am I good in the way that will be approved by others? Social, despite seeming like a morality thing. By extension, I wonder how much scrupolisity, as an anxiety disorder, is a social anxiety disorder.
I’d guess that the social egregore of Goodness also gets muddled in how it mixes “here are things you do to be a good member of society” and “here are things that are good because they’re personally prudent and or make you attractive to affiliate with for others”, e.g. it’s good to exercise and save money.
In my view this is the worst currated post decision in 2025 so far: - Please, Don’t Roll Your Own Metaethics - Not sure if you noticed, but the ethical stance suggested in the post is approximately the same as what many newage gurus will tell you “Stop being in your head! Listen to your heart! Follow the sense of yumminess! Free yourself from the expectations of your parents, school, friends and society!”. Tbh this is actually directionally sensible for some types of confused rationalists or people with exteme ammounts of scrupulosity, but is not generally good advice. - The only reason why this somewhat viable in some contexts is because every adult around internalized a lot of “Memetic values” (by which point John suggests to follow them). It’s a bit like commune of people living ‘free from modern civilization’ which means living 20m from nearby town and growing their own vegetables, relying on the modern civilization only in security, healthacre, tools, industrial production, education, culture, etc etc - If you read the comments it seems also John agrees that the target audience is specific (in my read people who never thought about human values much, and err on the side of following S2 culturally spread values)
the ethical stance suggested in the post is approximately the same as what many newage gurus will tell you “Stop being in your head! Listen to your heart! Follow the sense of yumminess! Free yourself from the expectations of your parents, school, friends and society!”.
The post writes:
I’d like to say here “screw memetic egregores, follow the actual values of actual humans”, but then many people will be complete fucking idiots about it.
....
And so Albert throws out all that Goodness crap, and just queries his own feelings of yumminess in-the-moment when making decisions.
This goes badly in a few different ways...
Yes, I can see a crude resemblance to that kind of advice but there’s a whole big section about not interpreting it in a dumb way. I’m also confused what the complaint is...there could be a hypothetical audience, different from the actual audience, who would take this the wrong way and do dumb things and therefore it’s a bad post even if it makes a correct point?. Granted, seems you think the point is correct.
I am more interested in the question of whether the post’s model is correct, seems like we maybe disagree there based on your comment. I’m not convinced. (Among other things I might say that egregores can be composed of sub-egregores and that’s fine, doesn’t mean there isn’t one here). A bit it feels like details, and the core point of something like your actual values (that are quite hard to determine!) are not the say thing as societal sense of “Good”. This doesn’t preclude interaction between the two and them shaping each other, that feels like it undermines the picture here.
No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.
Less wrong model in less confused terminology: top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.
Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”. - This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”). - Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”
The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.
One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.
Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.
--
No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)
Stepping back for a moment, just want to clarify goal of this comment exchange. In drafting a reply, I realize I was mixing between:
1) determining whether the decision to curate was good or not 2) determining what is true (according to my own beliefs) 3) determining whether the post is “good” or not.
Of course 1) impacts 2) impacts 3).
I think I came in with LessWrong model you describe and the piece didn’t update me so much as seemed like a straightforward explainer of a simple point (“what people say is Good isn’t the same as your Values). I think you have a point that the post does something like set up one side of the dichotomy as S1 boxes, though it’s salient to me that it also has:
We don’t really know what human values are, or what shape they are, or even whether they’re A Thing at all. We don’t have trivial introspective access to our own values; sometimes we think we value a thing a lot, but realize in hindsight that we value it only a little.
That feels appropriately non-committal.
I agree there’s complexity around egregores/memeplexes and how it gets carved up.
It’s definitely not the bar for curation that everything in the post seems correct to the curator. I do think it should leave people better off than if they’d not read it. After this discussion, I’m less sure about this post. “Values are just the S1 boxes” seems so ridiculous to me that I wouldn’t expect anyone to think it, I don’t know. The egregore stuff feels much higher resolution than what this post is going for, though I think there’s interesting stuff to figure out there. I kind of like this post for having sparked that conversation, though perhaps it is a rehash that is tiresome to others.
I didn’t see your comment and the thread there, but yes. There is refinement and precision that could be added, whether the feelings vs the generator, etc, etc., but still that there’s something more inherent to you vs something lives outside of you and is more social, that point is correct.
Regarding the sadists, yes, I think the values of the sadist might well be torture and from their perspective, they should be optimizing for that. If my values are anti-sadism (and I think they are), then we are at odds and maybe we fight. I don’t think the structure of values prohibits people from having values different from my own. Strongly feel John’s “people object to this for dumb reasons” stance.
One way you could apply it is by not endorsing so completely/confidently the kind of “rolling your own metaethics” that I argued against (that I see John as doing here), i.e., by saying “the distinction John is making here is correct, plus his advice on how to approach it.” (Of course you wrote that before I posted, but I’m hoping this is one of the takeaways people get from my post.)
Ok, there’s argument I can see of “unlike other domains, ethics/meta-ethics lacks any empirical feedback loop on beliefs [at least that we’ve found] and this means all such claims should be made more lightly than anything more empirical/factual”. Given that, perhaps more hedging is warranted than “is correct”.
Now even before any of this discussion, I’d have been extremely hesitant to lock in my meta-ethical views to ASI, but day to day though, I feel like I need some kind of ethical framework to operate on. That’s where I’m not sure about what to do other than figure out what makes sense to me, in the same way I do for other things.
I’d need to think longer/be convinced to switch to a more modest epistemology specifically for this domain, if that’s kind of the suggestion of “not rolling your own”. That feels like a big topic though.
But yeah, I can take away “be less confident” here.
Curated. While in my personal language, I would have treated Goodness as a synonym for Human Values[1], the distinction John is making here is correct, plus his advice on how to approach it. A very important point I have noticed is that when people ask (or anguish), “am I a good person?” this is asking according to the social egregore sense of good – am I good in the way that will be approved by others? Social, despite seeming like a morality thing. By extension, I wonder how much scrupolisity, as an anxiety disorder, is a social anxiety disorder.
I’d guess that the social egregore of Goodness also gets muddled in how it mixes “here are things you do to be a good member of society” and “here are things that are good because they’re personally prudent and or make you attractive to affiliate with for others”, e.g. it’s good to exercise and save money.
And specifically my values because it’s an open question to me how broad my values are shared, cf. The Psychological Unity of Humankind
Object level disagreement with the post explained here.
In my view this is the worst currated post decision in 2025 so far:
- Please, Don’t Roll Your Own Metaethics
- Not sure if you noticed, but the ethical stance suggested in the post is approximately the same as what many newage gurus will tell you “Stop being in your head! Listen to your heart! Follow the sense of yumminess! Free yourself from the expectations of your parents, school, friends and society!”. Tbh this is actually directionally sensible for some types of confused rationalists or people with exteme ammounts of scrupulosity, but is not generally good advice.
- The only reason why this somewhat viable in some contexts is because every adult around internalized a lot of “Memetic values” (by which point John suggests to follow them). It’s a bit like commune of people living ‘free from modern civilization’ which means living 20m from nearby town and growing their own vegetables, relying on the modern civilization only in security, healthacre, tools, industrial production, education, culture, etc etc
- If you read the comments it seems also John agrees that the target audience is specific (in my read people who never thought about human values much, and err on the side of following S2 culturally spread values)
You write:
The post writes:
Yes, I can see a crude resemblance to that kind of advice but there’s a whole big section about not interpreting it in a dumb way. I’m also confused what the complaint is...there could be a hypothetical audience, different from the actual audience, who would take this the wrong way and do dumb things and therefore it’s a bad post even if it makes a correct point?. Granted, seems you think the point is correct.
I am more interested in the question of whether the post’s model is correct, seems like we maybe disagree there based on your comment. I’m not convinced. (Among other things I might say that egregores can be composed of sub-egregores and that’s fine, doesn’t mean there isn’t one here). A bit it feels like details, and the core point of something like your actual values (that are quite hard to determine!) are not the say thing as societal sense of “Good”. This doesn’t preclude interaction between the two and them shaping each other, that feels like it undermines the picture here.
No, the model in the post is mostly not correct. I’m discussing object level disagreements with the post elsewhere, but the ontology of the model is bad, and recommendations are also problematic.
Less wrong model in less confused terminology:
top-level category is human values; these can have many different origins, including body regulatory systems, memes, philosophical reflection, incentives,… ; these can be represented in different ways, including bodily sensations, not really legible S1 boxes producing ‘feelings’, S2 verbal beleifs. There is some correlation between the type of representation and orgin of the value, but its not too strong. Many values of memetic origin are internalized and manifest as S1 feelings, yumminess, etc.
Main thing the post is doing is posting a dichotomy between “not really legible S1 boxes representing values” and “memetic values”.
- This is not a natural way how to carve up the space, because one category is based on type of representation, and other on origin. (It’s a bit like if you divided computers into “Unix-based” and “laptops”).
- Second weird move is to claim that the natural name for the top level categoriy should apply just to the “not really legible S1 boxes representing values”
The “memetic values” box is treated quite weirdly. It is identified with just one example of value—“Goodness”, at is claimed that this value is an egregore. Egregore is the phenotype of a memeplex—the relation to memeplex is similar to the relation of the animal to its genome. Not all memeplexes build egregores, but some develop sufficient coordination technology that it becomes useful to model them through the intentional stance—as having goals, beliefs, and some form of agency. An egregore is usually a distributed agent running across multiple minds. Think of how an ideology can seem to “want” things and “act” through its adherents. In my view goodness is mostly verbal handle people use to point to values. It can point to almost any kind of value, including the S1 values. What egregores often try to do is to hijack the pointer and make it point to some verbal model spread by the memeples. For example: Social Justice is an egregore (while justice is not). What SJ egregore often does is rewrite the content of concepts like justice and fairness and point them to some specific verbal models, often in tension with S1 boxes, often serving the egregore. More useful model of goodness is it as particularly valuable pointer, due to extreme generality. As a result many egregores fight over what should it point to—eg rationalism would want ‘updating on evidence’ to be/feel good, and ‘making up fake evidence to win a debate’ to be bad. But it is a small minority of pathways by which cultural evolution changes your values.
One true claim about memetic values is they are subject to complex selection pressures, sometimes serve egregores, sometimes the collective,… If you meet claims like “the best thing you can do is sacrifice your life to spread this idea” its clearly suspicious.
Overall, the not-carving-reality-at-its-joints means the model in the post is not straightforwardly applicable. The first order read “kick out memetic values, S1 boxes good” is clearly bad advice (and also large part of your S1 boxes is memetic values). Hence a whole section on “don’t actually try to follow this and instead … reflect”. My impression is there is some unacknowledged other type of values guiding the reflection in the direction of “don’t be an asshole”.
--
No, I don’t mean hypotethical audience. I mean, for example, you. If—after reading the post—you believe there is this basic dichotomy between Human Values, and Goodness. Goodness is a memetic egregore, while Human Values are authentically yours and you should follow them, but in non-dumb ways… My claim is this is not carving reality at its joints and if you believe this you are confused. Probably confused in a different way than before (“Goodness as a synonym for Human Values”)
Stepping back for a moment, just want to clarify goal of this comment exchange. In drafting a reply, I realize I was mixing between:
1) determining whether the decision to curate was good or not
2) determining what is true (according to my own beliefs)
3) determining whether the post is “good” or not.
Of course 1) impacts 2) impacts 3).
I think I came in with LessWrong model you describe and the piece didn’t update me so much as seemed like a straightforward explainer of a simple point (“what people say is Good isn’t the same as your Values). I think you have a point that the post does something like set up one side of the dichotomy as S1 boxes, though it’s salient to me that it also has:
That feels appropriately non-committal.
I agree there’s complexity around egregores/memeplexes and how it gets carved up.
It’s definitely not the bar for curation that everything in the post seems correct to the curator. I do think it should leave people better off than if they’d not read it. After this discussion, I’m less sure about this post. “Values are just the S1 boxes” seems so ridiculous to me that I wouldn’t expect anyone to think it, I don’t know. The egregore stuff feels much higher resolution than what this post is going for, though I think there’s interesting stuff to figure out there. I kind of like this post for having sparked that conversation, though perhaps it is a rehash that is tiresome to others.
Really? Did you see this comment of mine? Do you endorse John’s reply to it (specifically the part about the sadist)?
I didn’t see your comment and the thread there, but yes. There is refinement and precision that could be added, whether the feelings vs the generator, etc, etc., but still that there’s something more inherent to you vs something lives outside of you and is more social, that point is correct.
Regarding the sadists, yes, I think the values of the sadist might well be torture and from their perspective, they should be optimizing for that. If my values are anti-sadism (and I think they are), then we are at odds and maybe we fight. I don’t think the structure of values prohibits people from having values different from my own. Strongly feel John’s “people object to this for dumb reasons” stance.
Have you also seen https://www.lesswrong.com/posts/KCSmZsQzwvBxYNNaT/please-don-t-roll-your-own-metaethics which was also partly in response to that thread? BTW why is my post still in “personal blog”?
Yes, though I am unsure how to apply it. Your thread with Raemom was a little helpful.
Posts are manually frontpaged and are typically done as a batch once a day. When I’m assigned, I typically process them around 10-11 PT.
One way you could apply it is by not endorsing so completely/confidently the kind of “rolling your own metaethics” that I argued against (that I see John as doing here), i.e., by saying “the distinction John is making here is correct, plus his advice on how to approach it.” (Of course you wrote that before I posted, but I’m hoping this is one of the takeaways people get from my post.)
Ok, there’s argument I can see of “unlike other domains, ethics/meta-ethics lacks any empirical feedback loop on beliefs [at least that we’ve found] and this means all such claims should be made more lightly than anything more empirical/factual”. Given that, perhaps more hedging is warranted than “is correct”.
Now even before any of this discussion, I’d have been extremely hesitant to lock in my meta-ethical views to ASI, but day to day though, I feel like I need some kind of ethical framework to operate on. That’s where I’m not sure about what to do other than figure out what makes sense to me, in the same way I do for other things.
I’d need to think longer/be convinced to switch to a more modest epistemology specifically for this domain, if that’s kind of the suggestion of “not rolling your own”. That feels like a big topic though.
But yeah, I can take away “be less confident” here.