I want to bring up a concept I found very useful for thinking about how to become less susceptible to these sorts of things.
(NB that while I don’t agree with much of the criticism here, I do think “the community” does modestly increase psychosis risk, and the Ziz and Vassar bubbles do so to extraordinary degrees. I also think there’s a bunch of low-hanging fruit here, so I’d like us to take this seriously and get psychosis risk lower than baseline.)
(ETA because people bring this up in the comments: law of equal and opposite advice applies. Many people seem to not have the problems that I’ve seen many other people really struggle with. That’s fine. Also I state these strongly—if you took all this advice strongly, you would swing way too far in the opposite direction. I do not anticipate anyone will do that but other people seem to be concerned about it so I will note that here. Please adjust the tone and strength-of-claim until it feels right to you, unless you are young and new to the “community” and then take it more strongly than feels right to you.)
Anyways, the concept: I heard the word “totalizing” on Twitter at some point (h/t to somebody). It now seems fundamental to my understanding of these dynamics. “Totalizing” was used in the sense of a “totalizing ideology”. This may just be a subculture term without a realer definition, but it means something like “an ideology that claims to affect/define meaning for all parts of your life, rather than just some”—and implicitly also that this ideology has a major effect and causes some behaviors at odds with default behavior.
This definition heavily overlaps with the stuff people typically associate with cults. For example, discouraging contact with family/outside, or having a whole lot hanging on whether the leaders approve of you. Both of these clearly affect how much you can have going on in your “outside” life.
Note that obviously totalization is on an axis. It’s not just about time spent on an ideology, but how much mental space that ideology takes up.
I think some of the biggest negative influences on me in the rationality community also had the trait of pushing towards totalization, though were unalike in many other ways. One was ideological and peer pressure to turn socializing/parties/entertainment into networking/learning, which meant that part of my life also could become about the ideology. Another was the idea of turning my thoughts/thinking/being into more fodder to think about thinking processes and self-improve, which cannibalized more of my default state.
I think engaging with new, more totalizing versions of the ideology or culture is a major way that people get more psychotic. Consider the maximum-entropy model of psychosis, so named because you aren’t specifying any of the neural or psychological mechanisms, you’re taking strictly what you can verify and being maximally agnostic about it. In this model, you might define psychosis as when “thought gets too far away from normal, and your new mental state is devoid of many of the guardrails/protections/negative-feedback-loops/sanity-checks that your normal mental states have.” (This model gels nicely with the fact that psychosis can be treated so well via drinking water, doing dishes, not thinking for awhile, tranquilizers, socializing, etc. (h/t anon).) In this max-ent model of psychosis, it is pretty obvious how totalization leads to psychosis. Changing more state, reducing more guardrails, rolling your own psychological protections that are guaranteed to have flaws, and cutting out all the normal stuff in your life that resets state. (Changing a bunch of psychological stuff at once is generally a terrible idea for the same reason, though that’s a general psychosis tip rather than a totalization-related one.)
I still don’t have a concise or great theoretical explanation for why totalization seems so predictive of ideological damage. I have a lot of reasons for why it seems clearly bad regarding your belief-structure, and some other reasons why it may just be strongly correlated with overreach in ways that aren’t perfectly causal. But without getting into precisely why, I think it’s an important lens to view the rationalist “community” in.
So I think one of the main things I want to see less of in the rationalist/EA “communities” is totalization.
This has a billion object-level points, most of which will be left as an exercise to the reader:
Don’t proselytize EA to high schoolers. Don’t proselytize other crazy ideologies without guardrails to young people. Only do that after your ideology has proven to make a healthy community with normal levels of burnout/psychosis. I think we can get there in a few years, but I don’t think we’re there yet. It just actually takes time to evolve the right memes, unfortunately.
To repeat the perennial criticism… it makes sense that the rationality community ends up pretty insular, but it seems good for loads of reasons to have more outside contact and ties. I think at the very least, encouraging people to hire outside the community and do hobbies outside the community are good starting points.
I’ve long felt that at parties and social events (in the Bay Area anyways) less time should be spent on model-building and networking and learning, and more time should be spent on anything else. Spending your time networking or learning at parties is fine if those are pretty different than your normal life, but we don’t really have that luxury.
Someone recently tried to tell me they wanted to put all their charitable money into AI safety specifically, because it was their comparative advantage. I disagree with this even on a personal basis with small amounts. Making donations to other causes helps you take them seriously, in the way that trading with real-but-trivial amounts of money instead of paper trading moves you strongly from Far Mode into Near Mode. I think paying 10% overhead of charitable money to lower-EV causes is going to be much better for AI safety in the long-run due to seriousness-in-exploration, AND I shouldn’t even have to justify it as such—I should be able to say something like “it’s just unvirtuous to put all eggs in one basket, don’t do it”. I think the old arguments about obviously putting all your money into the highest-EV charity at a given time are similarly wrong.
I love that Lightcone has a bunch of books outside the standard rationalist literature, about Jobs, Bezos, LKY, etc etc.
In general, I don’t like when people try to re-write social mechanisms (I’m fine with tinkering, small experiments, etc). This feels to me like one of the fastest ways to de-stabilize people, as well as the dumbest Chesterton’s fence to take down because of how socializing is in the wheelhouse of cultural gradient descent and not at all remotely in the wheelhouse of theorizing.
I’m much more wary of psychological theorizing, x-rationality, etc due to basically the exact points in the bullet above—your mind is in the wheelhouse of gradient descent, not guided theorizing. I walk this one—I quit my last project in part because of this. Other forms of tinkering-style psychological experimentation or growth are likely more ok. But even “lots of debugging” seems bad here, basically because it gives you too much episteme of how your brain works and not enough techne or metis to balance it out. You end up subtly or not-subtly pushing in all sorts of directions that don’t work, and it causes problems. I think the single biggest improvement to debugging (both for ROI and for health) is if there was a culture of often saying “this one’s hopeless, leave it be” much earlier and explicitly, or saying “yeah almost all of this is effectively-unchangeable”. Going multiple levels down the tree to solve a bug is going too far. It’s too easy to get totalized by the bug-fixing spirit if you regard everything as mutable.
As dumb as jobs are, I’m much more pro-job than I used to be for a bunch of reasons. The core reasons are obv not because of psychosis, but other facets of totalization-escape seems like a major deal.
As dumb as regular schedules, are, ditto. Having things that you repeatedly have to succeed in doing leaves you genuinely much less room for going psychotic. Being nocturnal and such are also offenders in this category.
I’d like to see Bay Area rationalist culture put some emphasis on real holidays rather than only rolling their own. E.g. Solstice instead of Christmas seems fine, but also we should have a lot of emphasis on Christmas too? I had a housemate who ran amazing Easter celebrations in Event Horizon that were extremely positive, and I loved that they captured the real spirit of Easter rather than trying to inject the spirit of Rationality into the corpse of Easter to create some animated zombie holiday. In this vein I also love Petrov Day but slightly worry that we focus much less on July 4th or Thanksgiving or other holidays that are more shared with others. I guess maybe I should just be glad we haven’t rationalized those...
Co-dependency and totalizing relationships seem relevant here although not much new to say.
Anna’s crusade for hobbies over the last several years has seemed extremely useful on this topic directly and indirectly.
I got one comment on a draft of this about how someone basically still endorsed years later their totalization after their CFAR workshop. I think this is sort of fine—very excitable and [other characterizations] people can easily become fairly totalized when entering a new world. However, I still think that a culture which totalized them somewhat less would have been better.
Also, lots of people totalize themselves—I was one of those people who got very excited about rationality and wanted to push it to new heights and such, unendorsed by anyone in the “community” (and even disendorsed). So this isn’t a question of “leadership” of some kind asking too much from people (except Vassar)—it’s more a question of building a healthy culture. Let us not confuse blame with seeking to become better.
The folk theory of lying is a tiny bit wrong and I agree it should be patched. I definitely do not agree we should throw it out, or be uncertain whether lying exists.
Lying clearly exists.
1. Oftentimes people consider how best to lie about e.g. them being late. When they settle on the lie of telling their boss they were talking to their other boss and they weren’t, and they know this is a lie, that’s a central case of a lie—definitely not motivated cognition.
To expand our extensional definition to noncentral cases, you can consider some other ways people might tell maybe-lies when they are late. Among others, I have had the experiences [edit: grammar] of
2. telling someone I would be there in 10 minutes when it was going to take 20, and if you asked me on the side with no consequences I would immediately have been able to tell you that it was 20 even though in the moment I certainly hadn’t conceived myself as lying, and I think people would agree with me this is a lie (albeit white)
3. telling someone I would be there in 10 minutes when it was going to take 20, and if you asked me on the side with no consequences I would have still said 10, because my model definitely said 10, and once I started looking into my model I would notice that probably I was missing some contingencies, and that maybe I had been motivated at certain spots when forming my model, and I would start calculating… and I think most people would agree with me this is not a lie
4. telling someone I would be there in 10 minutes when it was going to take 20, and my model was formed epistemically virtuously despitely obviously there being good reasons for expecting shorter timescales, and who knows how long it would take me to find enough nuances to fix it and say 20. This is not a lie.
Ruby’s example of the workplace fits somewhere between numbers 1 and 2. Jessica’s example of short AI timelines I think is intended to fit 3 (although I think the situation is actually 4 for most people). The example of the political fact-checking doesn’t fit cleanly because politically we’re typically allowed to call anything wrong a “lie” regardless of intent, but I think is somewhere between 2 and 3 and I think nonpartisan people would agree that, unless the perpetrators actually could have said they were wrong about the stat, the case was not actually a lie (just a different type of bad falsehood reflecting on the character of those involved). There are certainly many gradations here, but I just wanted to show that there is actually a relatively commonly accepted implicit theory about when things are lies that fits with the territory and isn’t some sort of politicking map distortion as it seemed you were implying.
The intensional definition you found that included “conscious intent to deceive” is not actually the implicit folk theory most people operate under: they include number 2′s “unconscious intent to deceive” or “in-the-moment should-have-been-very-easy-to-tell-you-were-wrong obvious-motivated-cognition-cover-up”. I agree the explicit folk theory should be modified, though.
I also want to point out that this pattern of explicit vs implicit folk theories applies well to lots of other things. Consider “identity”—the explicit folk theory probably says something about souls or a real cohesive “I”, but the implicit version often uses distancing or phrases like “that wasn’t me” [edit: in the context of it being unlike their normal self, not that someone else literally did it] and things such that people clearly sort of know what’s going on. Other examples include theory of action, “I can’t do it”, various things around relationships, what is real as opposed to postmodernism, etc etc. To not cherry-pick, there are some difficult cases to consider like “speak your truth” or the problem of evil, but under nuanced consideration these fit with the dynamic of the others. I just mention this generalization because LW types (incl me) learned to tear apart all the folk theories because their explicit version were horribly contradictive, and while this has been very powerful for us I feel like an equally powerful skill is figuring out how to put Humpty-Dumpty back together again.