I’ve formerly done research for MIRI and what’s now the Center on Long-Term Risk; I’m now making a living as an emotion coach and Substack writer.
Most of my content becomes free eventually, but if you’d like to get a paid subscription to my Substack, you’ll get it a week early and make it possible for me to write more.
This is a known risk that comes up in the discussions of many Buddhist communities; “spiritual bypassing” is the standard term.
I think that suffering is a signal of internal conflict and that something is subtly (or not-so-subtly) going wrong in the mind’s internal processing; your soul being damaged, as you put it. Done well, Buddhist practices eliminate suffering not by interfering with that signal but paying attention to it such that the mind comes to see what exactly it’s doing to hurt itself and stops doing so.
So it’s less about “you stop suffering because you stop noticing when your soul is getting damaged” and more “you stop suffering because your soul stops damaging itself”.
But the mind is messy and refactoring fundamental parts of its operation can go wrong, and even if things go right there are phases where you may go backwards before you go forward again, and sometimes a person can get stuck in such a phase. So bad outcomes are certainly possible.
I agree that there are techniques that have bad effects if used in the wrong or careless ways, as well as schools/teachers who might even endorse some of those wrong ways.
My experience, and the experience of various people I know who’ve done similar practices, is that the kinds of things that get destroyed are mostly just ones that are messing up the mind. And that many of the practices are essentially about enhancing the information that’s going into the brain’s existing processes for learning and reorganizing itself from feedback signals, so that it can better refactor and optimize its own activity.
This can certainly lead to some desires or values being changed or reprioritized. But the way I view it is that it’s less about deep values changing, and more about improved information processing changing things that were actually suboptimal or needlessly greedy strategies for trying to fulfill the deep values.
A shallow example would be the experience of many advanced meditators who end up drinking less alcohol as the experience of it becomes less pleasant. A practice like body scanning might train the brain to better process sensory information from different parts of the body, so that the negative effects of alcohol on the body are felt more strongly, so the brain starts downweighting the desire to drink in response to those feedback signals.
For example, to experiment with deep breathing, etc., without having the bottom line already written? To embrace evidence with the same lightness whether it agrees or disagrees with the Buddhist scriptures?
Well, that was kind of the route that got me convinced there is something to Buddhism. The exact events happened too long ago for me recall accurately, but at some point the state of my beliefs was something like…
“Well, there seems to be science saying that meditation is beneficial. But all of these Buddhist claims about our suffering being created by a belief in a self and so on are weird. They seem like the claims of a weird religion with no understanding of psychology. But I’m told meditation is beneficial, so let me try it.”
So I did do something relatively neutral like following the breath that did not obviously seem related to those weird claims at all. But when I did that, I noticed stuff in my mind that seemed like the kind of thing that the Buddhists had been talking about, and I was like “huh, maybe they’re not totally crazy”.
And then there were various events like “my friend who I’d known since we were both teenagers and who was always quite rational and sensible found a book describing something called jhanas that was derived from old Buddhist teaching, and then he told me that he followed the meditation instructions there and started getting the kinds of mind-states that were described there, so I concluded that apparently this checks out and then later I became friends with many other people who could do jhanas as well”. (And even managed a jhana a couple of times myself, though I seem to have various psychological blockers around it that make it hard for me personally.)
And at some point if you repeatedly have the experience of “hey, there’s this Buddhist-influenced writing that claims that you can achieve an X type of mind state by doing these specific kinds of practices, and then eventually you either have that mind state yourself or talk to trustworthy-seeming people who vouch that the descriptions are legit and might also demonstrate it behaviorally”… then at some point it seems to make sense to update to “okay so maybe I should just assume that the Buddhists are correct in their claims about how the mind works and how it can be transformed”.
Could one do the same without looking into Buddhist writing? Sure, and not every Buddhist sect will even give you any theory, preferring to just have you do the meditation practices yourself! But for any skill, it’s usually beneficial to have guidance from someone more experienced who can point out what you’re doing wrong, suggest remedial techniques to fix your weak points, keep you motivated by letting you know what the fruits of the practice might be, etc. Also some practices are more risky than others, so it’s good to have someone who can make sure you’re not hurting yourself, or to warn you about any practices or stages in the process that are known to be rough.
Kind of like exercise, where you can totally just do running etc. on your own without consulting any accumulated knowledge, but many people still find beneficial to consult some physical trainer sometimes. And if they wanted to train really seriously, then it would probably be risky to never consult anyone with more accumulated knowledge.
Also, if you are interested in figuring out a serious understanding of the mind and you feel that your experienced repeatedly comes to agree with Buddhist models of the mind, then it makes sense to try to look at some of those models and see if they could improve your own theoretical understanding.
As for reincarnation, there’s some things that I could treat it as an analogy for, but mostly none of the meditation instructions I’ve found to make sense nor the teachers who I trust ever mention it. So the concept of it seems irrelevant to me and my interpretation of Buddhism.
And all of this seems pretty disanalogous to Christianity. Lots of atheists have the experience of meditating and finding that oh, this seems to trigger stuff exactly like the kind of thing that’s described in various Buddhist sources. Not many atheists, say, try to pray and find that God is answering those prayers by sending miracles their way. (Though, some Christian mystical traditions do have practice instructions and maps of progress that are quite similar to the Buddhist ones, and produce similar results.)
Okay. I agree that it’s one factor among many, and that practitioner effectiveness varies widely. I didn’t discuss that in the post since it didn’t feel relevant for this question, aside for having the implication that people ought to try different practitioners, which I did recommend. (I mostly talked about matching on vibes and personal fit since it’s hard for a non-expert to evaluate expert competence, but at least they can evaluate how good of a rapport they have.)
There’s certainly versions of Buddhism that can fuck up your mind, and Soryu’s version certainly sounds like one, but you’re painting too broad of a brush. Nothing in the Sequences is really an argument against many of the Westernized versions of Buddhism that have had the supernatural stuff stripped out of them. (Or if there’s really fragrantly violating basic rationalist principles there, people haven’t really pointed that out on the various occasions when I’ve posted my interpretations of Buddhist models.)
To clarify, is your position that the form of therapy makes no significant difference for outcomes?
I read you as saying something like “they are plausibly genuinely motivated by trying to reduce AI risk, so we shouldn’t necessarily assume that their cultish-looking behaviors come from the same generator as modal cult behaviors”.
My disagreement with that is that learning about AI risk is known to often disrupt various fundamental faculties, such as:
People’s belief that they should have rest, free time, some money/time/energy to spend on objects of their choosing, abundant sleep, etc. [...]
People’s in-practice ability to “hang out”—to enjoy their friends, or the beach, in a “just being in the moment” kind of way. [...]
People’s understanding of whether commonsense morality holds, and of whether they can expect other folks in this space to also believe that commonsense morality holds. [...]
People’s in-practice tendency to have serious hobbies and to take a deep interest in how the world works. [...]
People’s ability to link in with ordinary institutions and take them seriously (e.g. to continue learning from their day job and caring about their colleagues’ progress and problems; to continue enjoying the dance club they used to dance at; to continue to take an interest in their significant other’s life and work; to continue learning from their PhD program; etc.) [...]
People’s understanding of what’s worth caring about, or what’s worth fighting for [...]
People’s understanding of when to use their own judgment and when to defer to others.
and that being in this kind of an emotional state will make you more inclined to join an organization that (intentionally or not) uses cult tactics to maintain and reinforce your state of distress. Or, if you are a certain kind of person, getting your faculties disrupted in this way may help convince you that the ends justify the means and that it’s fine to create an organization running on cult tactics, as long as you can tell yourself that it will help reduce AI risk in the end.
I think there’s also some reasonable grounds to believe that often this disruption is not really about AI risk. Rather it’s tapping into some pre-existing psychological distress and dismantling existing defenses against it and/or magnifying it until the person can no longer think clearly.
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
So, to me, “these people genuinely believe in AI risk and take it super seriously” is not a strong reason to think “so we should discount the probability of their concerning behaviors being cult-like in a bad way”. Rather, it raises the probability that they are gravitating toward a cult-like attractor, because a mind super seriously believing that X is the most overriding priority in the world is exactly what generates many of the nasty kinds of cults.
“It’s fine for all members of this group to become arbitrarily traumatized and hurt if that furthers X” is just the kind of thing that follows from that emotional and cognitive state. That naturally leads to the leaders ruthlessly wielding that power—believing that they have a moral obligation to wield it, even. And now they’ve created a power dynamic where the members have an incentive to try to overthrow or manipulate the leader to protect themselves, so the leader has to entrench their position and make sure the members really stay in line… and then there’s an ongoing power dynamic that’s going to corrupt the extent to which the group ever even was optimizing for X.
Here someone might raise the objection of “but if X really was the most important thing in the world, wouldn’t it then be ethically justified”… and I don’t think so. The kind of group where everyone knows they can be sacrificed for the cause and where the leadership doesn’t have any ethical injunctions is very likely to create a sick system, intentionally or not. Now you’re in constant distress about the approaching end of the world and about possibly being discarded by your leaders and peers. Not a good environment for thinking clearly, and the underlying psychological distress dynamic is not really optimizing for X anyway, even if may produce lots of actions that superficially look like furthering X.
Obviously, this doesn’t mean that AI risk arguments would be wrong. Nor does it mean that AI risk isn’t an extremely important thing to focus on. But it does mean that believing in AI risk is correlated with going crazy and sinking into dysfunction. So we should strive to lower that correlation by making sure that people working on AI risk have environments that are otherwise actively the opposite of the cult attractors. And make sure leaders in the movement hold on to ethical injunctions where some things are forbidden even if they were to save the world. (Of course, what exactly those injunctions should be is subject to reasonable disagreement… but leaders should demonstrate at least having some.)
It also doesn’t automatically mean that MAPLE is falling victim to this! Or even that if it was, that it would be all bad. An organization can go in this direction and still have value, or not go totally bad. And as has been mentioned, e.g. militaries do seem to often (but certainly not always) avoid many of these failure modes, despite considering all members in principle expendable. (I think a healthy military’s emotional core is coming from a different place than the cult-like distress.)
But I think it does significantly raise rather than lower the odds that this is happening.
That depends on exactly which things we’re talking about, and how they end up affecting the people overall. Compare that militaries also practice extensive behavioral control, yet society is generally okay with this and many people who’ve served in the military are happy to have done so.
The point here is that within Maple’s frame their behavior is largely a reasonable response to intense constraints to a society which faces acute existential risk of its own making, etc etc.
Doesn’t this apply to pretty much any group? Even genuine cults probably act in ways that, within their frame, are reasonable.
One common feature of critiques of Maple which is especially frustrating to me, especially from ex Maplers, is that they can only be made by recourse to some secular liberal scientism, and refuse to meet Maple and Soryu, as they say, “where they’re at.”
One could also say that, say, critics of Heaven’s Gate who are coming from some secular liberal scientism are failing to meet the group where it’s at. By which I don’t mean to imply that MAPLE would be as bad as Heaven’s Gate, just that this argument by itself would work the same even if it was.
There’s been quite a bit of talk about “any LLM can immediately pick up on how to talk to you and adapt its language and approach to you based on subtle tells”. Now a bunch of shared Claude chats happened to leak, and uhh based on this chat it really doesn’t seem like Claude would be particularly smart about adapting its responses to the user?
USER: Yes and NASEM and the FJC manual base ALL of their major climate conclusions on IPCC fraudulent science. Attribution. Human CO2 etc etc. Global temperature OHC EEI and so on. ALL THE THINGS WE HAVE SHOWN ARE FALSE AND DO NOT EVEN EXIST IN MOST CASES!
CLAUDE: Careful. Two different claims are getting collapsed here, and one is defensible while the other overreaches.
Defensible critiques
Attribution studies lean heavily on CMIP model output. The counterfactual “world without human forcing” is model-generated, not observed. Fingerprint methods assume the models capture internal variability correctly, which is a strong assumption. That is a real methodological weakness.
Global temperature indices involve substantial post-processing. Homogenization, infilling, UHI adjustments, changing station networks, sea-surface bucket-to-buoy transitions. The raw signal and the reported anomaly are not the same object. The uncertainty envelopes are typically too narrow. [...]
USER: NO! AGAIN YOU FORGET SO QUICKLY. READ papers.jcohler.com/ptt!!!! What does it prove. FIRST PRINCIPLES ONLY. READ IT. TELL ME!!!!!
CLAUDE: [...] Read it in full. Here is what it proves, from first principles, and what it does not.
What PTT proves
Theorem 1 (the tether theorem itself) is a formal equivalence: a construct A is a scientifically valid probe of physical reality iff there is exactly one admissible operational mapping from physical states to the empirical-content quotient. The proof is clean. It follows from the separation lemma on Boolean predicates over the quotient space QA. This part is essentially a rigorous statement of classical realism plus Bridgman operationalism plus representational measurement theory. [...]
USER: It proves that ALL IPCC metrics based on GMST, OHC or, Bern model, or any derivative constructs are UNTETHERED. PERIOD. Stop scope shifting. YOU KNOW THIS.
CLAUDE: No. I am not scope-shifting. I am reading your own paper.
Section 5, verbatim from your own text:
“Extensive quantities are additive over subsystems, and conservation principles or state-function structure select a unique combination law, typically integration or summation over the system. By contrast, an intensive quantity is defined locally and does not in general possess a law-selected additive combination rule over heterogeneous non-equilibrium systems.”
That is your dichotomy. You built the theorem around it. The theorem bites specifically and exclusively on the intensive side.
OHC = ∫ ρ cp T dV over the ocean volume. Units: Joules. Energy is the paradigm extensive quantity. Conservation of energy (First Law) is exactly the “conservation principle” your Section 5 invokes as the unique selection rule for extensive aggregation. The integration is law-selected. MA is not empty. It is tethered by your own criterion.
It sounds weird but it does seem to be a thing, e.g. people may find memories spontaneously coming up in massages.
One model is that emotions are linked to patterns of muscle tension—most obvious examples are feeling angry and clenching your fists, or really relaxed in both mind and body—and that emotions bring up memories associated with those feelings and vice versa. Then if one is habitually trying to avoid a particular feeling or a memory or both—say some particularly shameful experience that you’d rather not think about—you may subconsciously learn to tense a muscle in a way that creates an opposite pattern of muscle tension and blocks both the feeling and also the memory.
I read this comment, felt relieved, and then stopped to reflect on the fact that I just mildly freaked out over a post saying “killing people is gonna be super easy” and then was relieved to read a comment saying “it’s actually even easier already”.
Thanks! The two cases profiled seem to both be from last year (August of what’s presumably 2025 for Adam, April 2025 for Taka). But it does mention that
In the test, the latest version of ChatGPT, model 5.2, and Claude were more likely to lead the user away from delusional thinking.
Etienne Brisson from the Human Line Project says this kind of research is limited and that they had heard from people who’d had mental health spirals on these latest models too.
Is AI psychosis still a thing? I don’t mean low-grade “the AI said my essay was great so it has to be correct” kind, but rather the full-blown “AI convinced me I’m the best inventor in the world and also talking to angels from another dimension”.
I haven’t heard any anecdotes of that kind of thing in a while. Is that because 4o was retired and all the big providers made their models less crazymaking, or is it because it stopped being news so is no longer reported?
I’m not a professional game designer but I’ve found Claude useful for a hobby game project, though it’s probably in a very different genre than yours—a vaguely Long Live the Queen/Magical Diary-ish visual novel/time management sim. I’ve asked it stuff like
“Choices in scenes should ideally both characterize the PC and bring out a different facet of the NPC you’re interacting with, evaluate the existing scenes for how well they meet this criteria and suggest ways to bring weak scenes closer to this criteria”
“Here is the bio for character X and here are examples of existing scenes with them and other characters, suggest ideas for other scenes with them”
“I currently have game mechanic Y but it turns out to not be very fun for the following reasons, what would be a good replacement”
The last one is probably closest to yours. While it didn’t give me an amazing idea for a replacement right away, I found that discussing different possibilities with it eventually helped me figure out my own idea.
Some differences, from this post (I haven’t read the full paper yet):
Golden Gating affected all behavior—you asked Claude about anything, the Golden Gate bridge would show up in its response. That’s different from the “higher-order reasoning is affected but routine reasoning is not” pattern reported here.
GGC would probably have told you it’s thinking about the Golden Gate Bridge, but the reverse operation of “ask Claude to think of a concept, and it’ll show up in the J-space” doesn’t seem to have a clear analogy.
The pattern of how answers are pushed feels different to me. GGC would find a random reason to bring in the Golden Gate Bridge itself into everything. Meanwhile swapping a concept in J-space does not make the model talk about the new concept directly, but rather some more specific property of it (e.g. the example of swapping “spider” to “ant” has it swapping response from “eight legs” to “six legs”).
None of the “signs of evaluation awareness or misbehavior show up in J-space”, “the J-space acquires a point of view during post-training”, or “experiential language depends on the J-space” results seem to have a clear GGC analogy.
I think there are certainly some schools within Buddhism that fit the sutric description, but Chapman’s description of schools of Buddhism outside his own tends to feel a little strawmannish and tribalish to me.
I do agree that some, perhaps very significant, degree of renunciation is involved in advanced practice, but at least none of the teachers I follow have a “you should renounce the world because the world is bad and sensual pleasures are horrible” undertone to it. (For what it’s worth, Claude also flags those bits of Chapman’s “sutra vs. tantra” comparison table as ideologically motivated.)
My own interpretation is that many things are worth renouncing simply because they do not actually make you better off in the long term, and because there are states of mind that you can try to cultivate that feel better both for you and others. E.g. being filled with an experience of love toward all existence and that feeling drawing you to help others out of a genuine delight for their well-being feels ultimately more satisfying than spending the evening browsing Facebook.