“Brainwashing”, as popularly understood, does not exist or is of almost zero effectiveness. The belief stems from American panic over Communism post-Korean War combined with fear of new religions and sensationalized incidents; in practice, “cults” have retention rates in the single percentage point range and ceased to be an issue decades ago. Typically, a conversion sticks because an organization provides value to its members.
Until I read this, I didn’t realize there are different possible claims about the dangers of cults. One claim—the one gwern is debunking—is that cults are a large-scale danger, and practically anyone can be taken over by a cult.
The other less hyperbolic claim is that cults can seriously screw up people’s lives, even if it’s a smallish proportion of people. I still think that’s true.
This point is more about framing, and it connects, for me, to covid and so on. The central issue in both cases seems to be that some people who consider “the herd” to be the only object of value and the only way to really justify “policies or norms or advice”.
By contrast, I think a herd that doesn’t serve its members in a clean and honest way is simply not a good herd. Either a “functioning herd” unpacks logically into a large number of small observable countable “happy lives full of happy events” or else the herd is shit. Make the lives good, and the herd is good. Done.
I. For Example: Masks Working Locally But Not Globally
Do “masks” work for covid?
On one level, the level of keeping viruses away from “one’s own” personal airholes… masks just OBVIOUSLY work to keep your body momentarily safe from being infiltrated by viral particles larger the the very tiny holes in the mask that let through individual O2 and N2 molecules (but not larger things like sooty carbon-based smoke particles (which are themselves smaller than even bigger viral particles)).
On a bus full of coughing people: properly manufactured and tested and worn masks are obviously selfishly useful (and make a positive contribution in a suite of similar precautions).
Now suppose someone only cares about the herd, and their only possible intervention is to go on TV and say “Yay for masks, m’kay?”
Then we look at whether the words being said on TV causes covid to not be endemic… and we expect a negative result. So “saying yay” here… might not be a cure-all for society. Why will it not work? Many reasons.
Some people don’t even have TV to hear the message, and some people rightly distrust most things the TV tells them (it fully of lies and ads and partisan bickering and infomercials after all), and some people just don’t believe in the germ theory of disease (weirdly, many microbiologists don’t believe it in practice), and some people never get to Piaget Level 4 such that they can “understand and believe ANY theory” in a meaningful way that links abstractions to concrete details in a rigorous fashion.
So there will be failures in adherence at adequate protective practices in “parts of the herd of people”… and then the thing is infectious, so it races through some of the herd, but maybe not all of it, focusing mostly on germophobically incapable folk.
Then it could persist in cliques of people that contain a member whose carefulness is not completely adequate eventually… and then hop from clique to clique forever, with spikes. And in general the herd will NOT be protected by the mere statements on TV having been uttered.
On this reasoning, “masks don’t work” is actually mechanistically sensible… if you think of sociological predictions as a mechanistic practice.
(To actually eradicate covid would take roughly two interventions: (1) regular mass testing and (2) involuntary quarantine of the infectious. Then maintenance involves testing and quarantine at ports. Masks aren’t on this lists. Neither are vaccines. These other things are both just ways to selfishly cope with covid in a endemically infected region of the world whose government can’t do the two things that actually would work.)
II. Totalizing Social Groups Devouring Individuals But Not The World
I think something similar to the covid model is likely true with high-demand (totalitarian? totalizing? monopolizing?) social groups that mostly recruit a small percent of the mentally vulnerable and use them up after two to five years.
These high-demand groups are not dangerous to the herd necessarily.
If herd level interests and actions are your only lens for thinking about goodness and badness, then gwern’s position is that “cults are not bad” (and I hear this with the coda ”...in the same way that masks are not good”).
But in the meantime, cults are probably not safe to join for actual people whose lives are individually worthwhile, and who deserve the modicum of practical caring concern that each individual human naturally deserves just for being a human in a healthy and functional society (which ours is maybe not (because many don’t get such treatment)).
If the cult said on its tin what it does, and then did what it said on the tin, would the people have joined? Probably not.
So what happened might not have been a LIE, but it was literally “mis-leading”. The people were led… BADLY. They trusted someone to guide their behavior… and then regretted it.
III. Overreacting To Extremes When Normal Badness Is Common
A deeper thing: I’ve been told by two different CEOs of two different startups I worked at, in speeches they gave to the entire company, that working at their startup was “the most important thing that any of us will ever be involved in”.
At least one of them was wrong, but probably both of them were wrong, and this kind of thing seems to be part and parcel of very much of modern American culture… it just happens to be rife with insanity and lies.
When I heard this line, both times (but more the second time) I just rolled my eyes and remembered to cash my paycheck.
In retrospect I feel that I learned lots of TECHNICAL skills on the job that I wanted to learn (probably because wiser people than I was at that time all passed on these jobs), but in addition to this I learned a lot about business functioning and dysfunction by direct observation (hanging out just below and outside the C-suite, working as a data scientist)...
...but I imagine that a lot of people (perhaps less prone to going meta and making theories) do see in glimmers of these same general kinds of “crazy CEO deals” in their own lives, and vaguely undertand how broken a lot of things are in our post-post-modern “normally accepted everyday life”...
...and then they react to slightly more extreme things than they see in their daily experience with only “normal amounts of brokenness in it” by over-reacting to the specific extreme instance …which seems like maybe a sort of a way to try to cope with the totality of an imperfect global experience?
Maybe overreacting to new vivid instances of a bad trend is useful sometimes? Who knows! Not me. Not for sure ;-)
Previously on Less Wrong: gwern’s “Notes on Brainwashing & ‘Cults’”. The summary in his own words:
Thanks for that link! I missed that one, and it is pretty good.
I think Nancy’s response is the one I personally resonate with.
This point is more about framing, and it connects, for me, to covid and so on. The central issue in both cases seems to be that some people who consider “the herd” to be the only object of value and the only way to really justify “policies or norms or advice”.
By contrast, I think a herd that doesn’t serve its members in a clean and honest way is simply not a good herd. Either a “functioning herd” unpacks logically into a large number of small observable countable “happy lives full of happy events” or else the herd is shit. Make the lives good, and the herd is good. Done.
I. For Example: Masks Working Locally But Not Globally
Do “masks” work for covid?
On one level, the level of keeping viruses away from “one’s own” personal airholes… masks just OBVIOUSLY work to keep your body momentarily safe from being infiltrated by viral particles larger the the very tiny holes in the mask that let through individual O2 and N2 molecules (but not larger things like sooty carbon-based smoke particles (which are themselves smaller than even bigger viral particles)).
On a bus full of coughing people: properly manufactured and tested and worn masks are obviously selfishly useful (and make a positive contribution in a suite of similar precautions).
Now suppose someone only cares about the herd, and their only possible intervention is to go on TV and say “Yay for masks, m’kay?”
Then we look at whether the words being said on TV causes covid to not be endemic… and we expect a negative result. So “saying yay” here… might not be a cure-all for society. Why will it not work? Many reasons.
Some people don’t even have TV to hear the message, and some people rightly distrust most things the TV tells them (it fully of lies and ads and partisan bickering and infomercials after all), and some people just don’t believe in the germ theory of disease (weirdly, many microbiologists don’t believe it in practice), and some people never get to Piaget Level 4 such that they can “understand and believe ANY theory” in a meaningful way that links abstractions to concrete details in a rigorous fashion.
So there will be failures in adherence at adequate protective practices in “parts of the herd of people”… and then the thing is infectious, so it races through some of the herd, but maybe not all of it, focusing mostly on germophobically incapable folk.
Then it could persist in cliques of people that contain a member whose carefulness is not completely adequate eventually… and then hop from clique to clique forever, with spikes. And in general the herd will NOT be protected by the mere statements on TV having been uttered.
On this reasoning, “masks don’t work” is actually mechanistically sensible… if you think of sociological predictions as a mechanistic practice.
(To actually eradicate covid would take roughly two interventions: (1) regular mass testing and (2) involuntary quarantine of the infectious. Then maintenance involves testing and quarantine at ports. Masks aren’t on this lists. Neither are vaccines. These other things are both just ways to selfishly cope with covid in a endemically infected region of the world whose government can’t do the two things that actually would work.)
II. Totalizing Social Groups Devouring Individuals But Not The World
I think something similar to the covid model is likely true with high-demand (totalitarian? totalizing? monopolizing?) social groups that mostly recruit a small percent of the mentally vulnerable and use them up after two to five years.
These high-demand groups are not dangerous to the herd necessarily.
If herd level interests and actions are your only lens for thinking about goodness and badness, then gwern’s position is that “cults are not bad” (and I hear this with the coda ”...in the same way that masks are not good”).
But in the meantime, cults are probably not safe to join for actual people whose lives are individually worthwhile, and who deserve the modicum of practical caring concern that each individual human naturally deserves just for being a human in a healthy and functional society (which ours is maybe not (because many don’t get such treatment)).
If the cult said on its tin what it does, and then did what it said on the tin, would the people have joined? Probably not.
So what happened might not have been a LIE, but it was literally “mis-leading”. The people were led… BADLY. They trusted someone to guide their behavior… and then regretted it.
III. Overreacting To Extremes When Normal Badness Is Common
A deeper thing: I’ve been told by two different CEOs of two different startups I worked at, in speeches they gave to the entire company, that working at their startup was “the most important thing that any of us will ever be involved in”.
At least one of them was wrong, but probably both of them were wrong, and this kind of thing seems to be part and parcel of very much of modern American culture… it just happens to be rife with insanity and lies.
When I heard this line, both times (but more the second time) I just rolled my eyes and remembered to cash my paycheck.
In retrospect I feel that I learned lots of TECHNICAL skills on the job that I wanted to learn (probably because wiser people than I was at that time all passed on these jobs), but in addition to this I learned a lot about business functioning and dysfunction by direct observation (hanging out just below and outside the C-suite, working as a data scientist)...
...but I imagine that a lot of people (perhaps less prone to going meta and making theories) do see in glimmers of these same general kinds of “crazy CEO deals” in their own lives, and vaguely undertand how broken a lot of things are in our post-post-modern “normally accepted everyday life”...
...and then they react to slightly more extreme things than they see in their daily experience with only “normal amounts of brokenness in it” by over-reacting to the specific extreme instance …which seems like maybe a sort of a way to try to cope with the totality of an imperfect global experience?
Maybe overreacting to new vivid instances of a bad trend is useful sometimes? Who knows! Not me. Not for sure ;-)