Epistemic Status: Tries to get away with making nuanced points about social reality by using cute graphics of geometric objects. All models are wrong. Some models are useful.
Traditionally, when nerds try to understand social systems and fix the obvious problems in them, they end up looking something like this:
Social dynamics is hard to understand with your system 2 (i.e. deliberative/logical) brain. There’s a lot of subtle nuances going on, and typically, nerds tend to see the obvious stuff, maybe go one or two levels deeper than the obvious stuff, and miss that it’s in fact 4+ levels deep and it’s happening in realtime faster than you can deliberate. Human brains are pretty good (most of the time) at responding to the nuances intuitively. But in the rationality community, we’ve self-selected for a lot of people who:
Don’t really trust things that they can’t understand fully with their system 2 brain.
Tend not to be as naturally skilled at intuitive mainstream social styles.
Are trying to accomplish things that mainstream social interactions aren’t designed to accomplish (i.e. thinking deeply and clearly on a regular basis).
This post is an overview of essays that rationalist-types have written over the past several years, that I think add up to a “secret sequence” exploring why social dynamics are hard, and why they are important to get right. This may useful both to understand some previous attempts by the rationality community to change social dynamics on purpose, as well as to current endeavors to improve things.
(Note: I occasionally have words in [brackets], where I think original jargon was pointing in a misleading direction and I think it’s worth changing)
To start with, a word of caution:
Armchair sociolology can be harmful [link]- Ozy’s post is pertinent—most essays below fall into the category of “armchair sociology”, and attempts by nerds to understand and articulate social-dynamics that they aren’t actually that good at. Several times when an outsider has looked in at rationalist attempts to understood human interaction they’ve said “Oh my god, this is the blind leading the blind”, and often that seemed to me like a fair assessment.
I think all the essays that follow are useful, and are pointing at something real. But taken individually, they’re kinda like the blind men groping at the elephant, each coming away with the distinct impression an elephant is like a snake, tree, a boulder, depending on which aspect they’re looking at.
[Fake Edit: Ozy informs me that they were specifically warning against amateur sociology and not psychology. I think the idea still roughly applies]
Different people have different ways of articulating their needs and asking for help. Different ways of asking require different assumptions of trust. If people are bringing different expectations of trust into an interaction, they may feel that that trust is being violated, which can seem rude, passive aggressive or oppressive.
I’m listing this article, instead of numerous others about Ask/Guess/Tell, because I think: a) Malcolm does a good job of explaining how all the cultures work, and b) I think his presentation of Reveal culture is a good, clearer upgrade for Brienne’s Tell culture, and I’m a bit sad it didn’t seem to make it into the zeitgeist yet.
I also like the suggestion to call Guess Culture “Infer Culture” (implying a bit more about what skills the culture actually emphasizes).
Rationality folk (and more generally, nerds), tend to prefer explicit communication over implicit, and generally see Guess culture as strictly inferior to Ask culture once you’ve learned to assert yourself.
But there is something Guess culture does which Ask culture doesn’t, which is give you evidence of how much people understand you and are trying to cooperate. Guess cultures filters for people who have either invested effort into understanding your culture overall, or people who are good at inferring your own wants.
[WARNING: It turned out lots of people thought this meant something different than what I thought it meant. Some people thought it meant soft culture didn’t involve giving people feedback or criticism at all. I don’t think Soft/Sharp are totally-naturally clusters in the first place, and the distinction I’m interested in (as applies to rationality-culture), is how you give feedback.
(i.e. “Dude, your art sucks. It has no perspective.” vs “oh, cool. Nice colors. For the next drawing, you might try incorporating perspective”, as a simplified example)
Somewhat orthogonal to Infer/Ask/Reveal culture is “Soft” vs “Sharp” culture. Sharp culture tends to have more biting humor, ribbing each other, and criticism. Soft culture tends to value kindness and social harmony more. Sam says that Sharp culture “values honesty more.” Robby Bensinger counters in the comments: “My own experience is that sharp culture makes it more OK to be open about certain things (e.g., anger, disgust, power disparities, disagreements), but less OK to be open about other things (e.g., weakness, pain, fear, loneliness, things that are true but not funny or provocative or badass).”
Small talk often sounds nonsensical to literally-minded people, but it serves a fairly important function: giving people a structured path to figure out how much time/sympathy/interest they want to give each other. And even when the answer is “not much”, it still is, significantly, nonzero—you regard each other as persons, not faceless strangers.
This essays gets a lot of mixed reactions, much of which I think has to do with its use of the word “Person.” The essay is aimed at explaining how people end up treating each other as persons or nonpersons, without making any kind of judgement about it. This includes noting some things human tend to do that you might consider horrible.
Like many grand theories, I think it overstates it’s case and ignores some places where the explanation breaks down, but I think it points at a useful concept which is summarized by this adorable graphic:
The essay uses the word “personhood”. In the original context, this was useful: it gets at why cultures develop, why it matters whether you’re able to demonstrate reliably, trust, etc. It helps explain outgroups and xenophobia: outsiders do not share your social norms, so you can’t reliably interact with them, and it’s easier to think of them as non-people than try to figure out how to have positive interactions.
But what I’m most interested in is “how can we use this to make it easier for groups with different norms to interact with each other”? And for that, I think using the word “personhood” makes it way more likely to veer into judging each other for having different preferences and communication styles.
What makes a person is… arbitrary, but not fully arbitrary.
Rationalist culture tends to attract people who prefer a particular style of “social interface”, often favoring explicit communication and discussing ideas in extreme detail. There’s a lot of value to those things, but they have some problems:
a) this social interface does NOT mesh well with much of the rest of world (this is a problem if you have any goals that involve the rest of the world)
b) this goal does not uniformly mesh well with all people interested in and valuable to the rationality community.
I don’t actually think it’s possible to develop a set of assumptions that fit everyone’s needs. But I do think it’s possible to develop better tools for navigating different social contexts. I think it may be possible both to tweak sets-of-norms so that they mesh better together, or at least when they bump into each other, there’s greater awareness of what’s happening and people’s default response is “oh, we seem to have different preferences, let’s figure out how
Maybe we can end up with something that looks kinda like this:
Having said a bunch of things about different cultural interfaces, I think this post by Brienne is really important, and highlights the end goal of all of this.
“Cultures” are a crutch. They are there to help you get your bearings. They’re better than nothing. But they are not a substitute for actually having the skills needed to navigate arbitrary social situations as they come up so you can achieve whatever it is you want to achieve.
To master communication, you can’t just be like, “I prefer Tell Culture, which is better than Guess Culture, so my disabilities in Guess Culture are therefore justified.” Justified shmustified, you’re still missing an arm.
My advice to you—my request of you, even—if you find yourself fueling these debates [about which culture is better], is to (for the love of god) move on. If you’ve already applied cognitive first aid, you’ve created an affordance for further advancement. Using even more tourniquettes doesn’t help.
ii. Game Theory, Recursion, Trust
(or, “Social dynamics are complicated, you are not getting away with the things you think you are getting away with, stop trying to be clever, manipulative, act-utilitarian or naive-consequentialist without actually understanding what is going on”)
Critch argues why it is not just “morally wrong”, but an intellectual mistake, to violate someone’s trust (even when you don’t expect any repercussions in the future).
When someone decides whether to trust you (say, giving you a huge opportunity), on the expectation that you’ll refrain from exploiting them, they’ve already run a low-grade simulation of you in their imagination. And the thing is that you don’t know whether you’re in a simulation or not when you make the decision whether to repay them.
Some people argue “but I can tell that I’m a conscious being, and they aren’t a literal super-intelligent AI, they’re just a human. They can’t possibly be simulating me in this high fidelity. I must be real.” This is true. But their simulation of you is not based on your thoughts, it’s based on your actions. It’s really hard to fake.
One way to think about it, not expounded on in the article: Yes, if you pause to think about it you can notice that you’re conscious and probably not being simulated in their imagination. But by the time you notice that, it’s too late. People build up models of each other all the time, based on very subtle cues such as how fast you respond to something. Conscious you knows that you’re conscious. But their decision of whether to trust you was based off the half-second it took for unconscious you to reply to questions like “Hey, do you think you handle Project X while I’m away?”
The best way to convince people you’re trustworthy is to actually be trustworthy.
This is short enough to just include the whole thing:
Consider an “ask culture” where employees consider themselves totally allowed to say “no” without repercussions. The boss would prefer people work unpaid overtime so ey gets more work done without having to pay anything, so ey asks everyone. Most people say no, because they hate unpaid overtime. The only people who agree will be those who really love the company or their job—they end up looking really good. More and more workers realize the value of lying and agreeing to work unpaid overtime so the boss thinks they really love the company. Eventually, the few workers who continue refusing look really bad, like they’re the only ones who aren’t team players, and they grudgingly accept.
Only now the boss notices that the employees hate their jobs and hate the boss. The boss decides to only ask employees if they will work unpaid overtime when it’s absolutely necessary. The ask culture has become a guess culture.
How this applies to friendship is left as an exercise for the reader.
A fairly in depth look into how common knowledge, signaling, newcomb-like problems and recursive modeling of each other interact to produce “regular social interaction.”
I think there’s a lot of interesting stuff here—I’m not sure if it’s exactly accurate but it points in directions that seem useful. But I actually think the most important takeaway is the warning at the beginning:
WARNING: An easy instinct, on learning these things, is to try to become more complicated yourself, to deal with the complicated territory. However, my primary conclusion is “simplify, simplify, simplify”: try to make fewer decisions that depend on other people’s state of mind. You can see more about why and how in the posts in the “Related” section, at the bottom.
When you’re trying to make decisions about people, you’re reading a lot of subtle cues off them to get a sense of how you feel about that. When you [generic person you, not necessarily you in particular] can tell someone is making complex decisions based on game theory and trying to model all of this explicitly, it a) often comes across as a bit off, and b) even if it doesn’t, you still have to invest a lot of cognitive resources figuring out how they are modeling things and whether they are actually doing a good job or missing key insights or subtle cues. The result can be draining, and it can output a general response of “ugh, something about this feels untrustworthy.”
Whereas when people are able to cache this knowledge down into a system-1 level, you’re able to execute a simpler algorithm that looks more like “just try to be a good trustworthy person”, that people can easily read off your facial expression, and which reduces overall cognitive burden.
There’s some confusion over what “moral” means, because there’s two kinds of morality:
System 1 morality is noticing-in-realtime when people need help, or when you’re being an asshole, and then doing something about it.
System 2 morality is when you have a complex problem and a lot of time to think about it.
System 1 moralists will pay back Parfit’s Hitchhiker because doing otherwise would be being a jerk. System 2 moralists invent Timeless [Functional?] decision theory.
You want a lot of people with System 2 morality in the world, trying to fix complex problems. You want people with System 1 morality in your social circle.
The person who wrote this post eventually left the rationality community, in part due to frustration due to people constantly violating small boundaries that seemed pretty obvious (things in the vein of “if you’re going to be 2 hours late, text me so I don’t have to sit around waiting for you.”)
Final Remarks
I want to reiterate—all models are wrong. Some models are useful. The most important takeaway from this is not that any particular one of these perspectives is true, but that social dynamics has a lot of stuff going on that is more complicated than you’re naively imagining, and that this stuff is important enough to put the time into getting right.
Background Reading: The Real Hufflepuff Sequence Was The Posts We Made Along The Way
This is the fourth post of the Project Hufflepuff sequence. Previous posts:
Introduction
Who exactly is the Rationality Community? (i.e. who is Project Hufflepuff trying to help)
Straw Hufflepuffs and Lone Heroes
Epistemic Status: Tries to get away with making nuanced points about social reality by using cute graphics of geometric objects. All models are wrong. Some models are useful.
Traditionally, when nerds try to understand social systems and fix the obvious problems in them, they end up looking something like this:
Social dynamics is hard to understand with your system 2 (i.e. deliberative/logical) brain. There’s a lot of subtle nuances going on, and typically, nerds tend to see the obvious stuff, maybe go one or two levels deeper than the obvious stuff, and miss that it’s in fact 4+ levels deep and it’s happening in realtime faster than you can deliberate. Human brains are pretty good (most of the time) at responding to the nuances intuitively. But in the rationality community, we’ve self-selected for a lot of people who:
Don’t really trust things that they can’t understand fully with their system 2 brain.
Tend not to be as naturally skilled at intuitive mainstream social styles.
Are trying to accomplish things that mainstream social interactions aren’t designed to accomplish (i.e. thinking deeply and clearly on a regular basis).
This post is an overview of essays that rationalist-types have written over the past several years, that I think add up to a “secret sequence” exploring why social dynamics are hard, and why they are important to get right. This may useful both to understand some previous attempts by the rationality community to change social dynamics on purpose, as well as to current endeavors to improve things.
(Note: I occasionally have words in [brackets], where I think original jargon was pointing in a misleading direction and I think it’s worth changing)
To start with, a word of caution:
Armchair sociolology can be harmful [link] - Ozy’s post is pertinent—most essays below fall into the category of “armchair sociology”, and attempts by nerds to understand and articulate social-dynamics that they aren’t actually that good at. Several times when an outsider has looked in at rationalist attempts to understood human interaction they’ve said “Oh my god, this is the blind leading the blind”, and often that seemed to me like a fair assessment.
I think all the essays that follow are useful, and are pointing at something real. But taken individually, they’re kinda like the blind men groping at the elephant, each coming away with the distinct impression an elephant is like a snake, tree, a boulder, depending on which aspect they’re looking at.
[Fake Edit: Ozy informs me that they were specifically warning against amateur sociology and not psychology. I think the idea still roughly applies]
i. Cultural Assumptions of Trust
Guess [Infer] Culture, Ask Culture, and Tell [Reveal] Culture (link)
(by Malcolm Ocean)
Different people have different ways of articulating their needs and asking for help. Different ways of asking require different assumptions of trust. If people are bringing different expectations of trust into an interaction, they may feel that that trust is being violated, which can seem rude, passive aggressive or oppressive.
I’m listing this article, instead of numerous others about Ask/Guess/Tell, because I think: a) Malcolm does a good job of explaining how all the cultures work, and b) I think his presentation of Reveal culture is a good, clearer upgrade for Brienne’s Tell culture, and I’m a bit sad it didn’t seem to make it into the zeitgeist yet.
I also like the suggestion to call Guess Culture “Infer Culture” (implying a bit more about what skills the culture actually emphasizes).
Guess Culture Screens for Trying to Cooperate (link) (Ben Hoffman)
Rationality folk (and more generally, nerds), tend to prefer explicit communication over implicit, and generally see Guess culture as strictly inferior to Ask culture once you’ve learned to assert yourself.
But there is something Guess culture does which Ask culture doesn’t, which is give you evidence of how much people understand you and are trying to cooperate. Guess cultures filters for people who have either invested effort into understanding your culture overall, or people who are good at inferring your own wants.
Sharp Culture and Soft Culture (link) (Sam Rosen)
[WARNING: It turned out lots of people thought this meant something different than what I thought it meant. Some people thought it meant soft culture didn’t involve giving people feedback or criticism at all. I don’t think Soft/Sharp are totally-naturally clusters in the first place, and the distinction I’m interested in (as applies to rationality-culture), is how you give feedback.
(i.e. “Dude, your art sucks. It has no perspective.” vs “oh, cool. Nice colors. For the next drawing, you might try incorporating perspective”, as a simplified example)
Somewhat orthogonal to Infer/Ask/Reveal culture is “Soft” vs “Sharp” culture. Sharp culture tends to have more biting humor, ribbing each other, and criticism. Soft culture tends to value kindness and social harmony more. Sam says that Sharp culture “values honesty more.” Robby Bensinger counters in the comments: “My own experience is that sharp culture makes it more OK to be open about certain things (e.g., anger, disgust, power disparities, disagreements), but less OK to be open about other things (e.g., weakness, pain, fear, loneliness, things that are true but not funny or provocative or badass).”
Handshakes, Hi, and What’s New: What’s Going on With Small Talk? (Ben Hoffman)
Small talk often sounds nonsensical to literally-minded people, but it serves a fairly important function: giving people a structured path to figure out how much time/sympathy/interest they want to give each other. And even when the answer is “not much”, it still is, significantly, nonzero—you regard each other as persons, not faceless strangers.
Personhood [Social Interfaces?] (Kevin Simler)
This essays gets a lot of mixed reactions, much of which I think has to do with its use of the word “Person.” The essay is aimed at explaining how people end up treating each other as persons or nonpersons, without making any kind of judgement about it. This includes noting some things human tend to do that you might consider horrible.
Like many grand theories, I think it overstates it’s case and ignores some places where the explanation breaks down, but I think it points at a useful concept which is summarized by this adorable graphic:
The essay uses the word “personhood”. In the original context, this was useful: it gets at why cultures develop, why it matters whether you’re able to demonstrate reliably, trust, etc. It helps explain outgroups and xenophobia: outsiders do not share your social norms, so you can’t reliably interact with them, and it’s easier to think of them as non-people than try to figure out how to have positive interactions.
But what I’m most interested in is “how can we use this to make it easier for groups with different norms to interact with each other”? And for that, I think using the word “personhood” makes it way more likely to veer into judging each other for having different preferences and communication styles.
What makes a person is… arbitrary, but not fully arbitrary.
Rationalist culture tends to attract people who prefer a particular style of “social interface”, often favoring explicit communication and discussing ideas in extreme detail. There’s a lot of value to those things, but they have some problems:
a) this social interface does NOT mesh well with much of the rest of world (this is a problem if you have any goals that involve the rest of the world)
b) this goal does not uniformly mesh well with all people interested in and valuable to the rationality community.
I don’t actually think it’s possible to develop a set of assumptions that fit everyone’s needs. But I do think it’s possible to develop better tools for navigating different social contexts. I think it may be possible both to tweak sets-of-norms so that they mesh better together, or at least when they bump into each other, there’s greater awareness of what’s happening and people’s default response is “oh, we seem to have different preferences, let’s figure out how
Maybe we can end up with something that looks kinda like this:
Against Being Against or For Tell Culture (Brienne Yudkowsky)
Having said a bunch of things about different cultural interfaces, I think this post by Brienne is really important, and highlights the end goal of all of this.
“Cultures” are a crutch. They are there to help you get your bearings. They’re better than nothing. But they are not a substitute for actually having the skills needed to navigate arbitrary social situations as they come up so you can achieve whatever it is you want to achieve.
My advice to you—my request of you, even—if you find yourself fueling these debates [about which culture is better], is to (for the love of god) move on. If you’ve already applied cognitive first aid, you’ve created an affordance for further advancement. Using even more tourniquettes doesn’t help.
ii. Game Theory, Recursion, Trust
(or, “Social dynamics are complicated, you are not getting away with the things you think you are getting away with, stop trying to be clever, manipulative, act-utilitarian or naive-consequentialist without actually understanding what is going on”)
Grokking Newcomb’s Problem and Deserving Trust (Andrew Critch)
Critch argues why it is not just “morally wrong”, but an intellectual mistake, to violate someone’s trust (even when you don’t expect any repercussions in the future).
When someone decides whether to trust you (say, giving you a huge opportunity), on the expectation that you’ll refrain from exploiting them, they’ve already run a low-grade simulation of you in their imagination. And the thing is that you don’t know whether you’re in a simulation or not when you make the decision whether to repay them.
Some people argue “but I can tell that I’m a conscious being, and they aren’t a literal super-intelligent AI, they’re just a human. They can’t possibly be simulating me in this high fidelity. I must be real.” This is true. But their simulation of you is not based on your thoughts, it’s based on your actions. It’s really hard to fake.
One way to think about it, not expounded on in the article: Yes, if you pause to think about it you can notice that you’re conscious and probably not being simulated in their imagination. But by the time you notice that, it’s too late. People build up models of each other all the time, based on very subtle cues such as how fast you respond to something. Conscious you knows that you’re conscious. But their decision of whether to trust you was based off the half-second it took for unconscious you to reply to questions like “Hey, do you think you handle Project X while I’m away?”
The best way to convince people you’re trustworthy is to actually be trustworthy.
You May Not Believe In Guess [Infer] Culture But It Believes In You (Scott Alexander)
This is short enough to just include the whole thing:
Consider an “ask culture” where employees consider themselves totally allowed to say “no” without repercussions. The boss would prefer people work unpaid overtime so ey gets more work done without having to pay anything, so ey asks everyone. Most people say no, because they hate unpaid overtime. The only people who agree will be those who really love the company or their job—they end up looking really good. More and more workers realize the value of lying and agreeing to work unpaid overtime so the boss thinks they really love the company. Eventually, the few workers who continue refusing look really bad, like they’re the only ones who aren’t team players, and they grudgingly accept.
Only now the boss notices that the employees hate their jobs and hate the boss. The boss decides to only ask employees if they will work unpaid overtime when it’s absolutely necessary. The ask culture has become a guess culture.
The Social Substrate (Lahwran)
A fairly in depth look into how common knowledge, signaling, newcomb-like problems and recursive modeling of each other interact to produce “regular social interaction.”
I think there’s a lot of interesting stuff here—I’m not sure if it’s exactly accurate but it points in directions that seem useful. But I actually think the most important takeaway is the warning at the beginning:
When you’re trying to make decisions about people, you’re reading a lot of subtle cues off them to get a sense of how you feel about that. When you [generic person you, not necessarily you in particular] can tell someone is making complex decisions based on game theory and trying to model all of this explicitly, it a) often comes across as a bit off, and b) even if it doesn’t, you still have to invest a lot of cognitive resources figuring out how they are modeling things and whether they are actually doing a good job or missing key insights or subtle cues. The result can be draining, and it can output a general response of “ugh, something about this feels untrustworthy.”
Whereas when people are able to cache this knowledge down into a system-1 level, you’re able to execute a simpler algorithm that looks more like “just try to be a good trustworthy person”, that people can easily read off your facial expression, and which reduces overall cognitive burden.
System 1 and System 2 Morality (Sophie Grouchy)
There’s some confusion over what “moral” means, because there’s two kinds of morality:
System 1 morality is noticing-in-realtime when people need help, or when you’re being an asshole, and then doing something about it.
System 2 morality is when you have a complex problem and a lot of time to think about it.
System 1 moralists will pay back Parfit’s Hitchhiker because doing otherwise would be being a jerk. System 2 moralists invent Timeless [Functional?] decision theory.
You want a lot of people with System 2 morality in the world, trying to fix complex problems. You want people with System 1 morality in your social circle.
The person who wrote this post eventually left the rationality community, in part due to frustration due to people constantly violating small boundaries that seemed pretty obvious (things in the vein of “if you’re going to be 2 hours late, text me so I don’t have to sit around waiting for you.”)
Final Remarks
I want to reiterate—all models are wrong. Some models are useful. The most important takeaway from this is not that any particular one of these perspectives is true, but that social dynamics has a lot of stuff going on that is more complicated than you’re naively imagining, and that this stuff is important enough to put the time into getting right.