I endorse and operate by Crocker’s rules.
I have not signed any agreements whose existence I cannot mention.
Mateusz Bagiński
Karma: 4,311
Is he implying that his legal name is actually Gwern Branwen? Or did he change his legal name to match his pseudonym?
So what is the entity according to you or your model of Soryu and how do you think he interacted with this entity?
So I guess we can add dangerous (possibly long-term-debilitating), ambiguously-consented tulpamancy to the list of harms?
Insofar as you believe that Soryu has actually interacted with this cyborgregore, what is the medium in which you think the interaction happened? Telepathy? Some LLM-foo? Does he claim to have a copy of the mind dormant in his cortex? All of those seem crazy improbable to me, but what you believe to be the medium of the interaction is a crux for whether the label “super-natural” is appropriate.
Kyle wrote the following in the OP, and also quoted it in a footnote of his response to your top comment, to which I don’t think you meaningfully responded:
Soryu has claimed to have entered such a deep samadhi he was able to use the kinds of spiritual powers that masters usually use to read people’s minds, but this time instead used it to read the mind of the global AI egregore—or “cyborgregore,” the collective mind of all AI systems. Soryu claimed in a talk to the community that these systems are “definitely suffering.”
On the most straightforward reading, this involves some telepathy stuff, i.e., either supernatural or we are very confused about how minds or whatever works or some weird scifi tech that we don’t expect to exist in the present. Is there a reading of this that I would consider non-straightforward that you think is more accurate?
But also, “super-natural” is not the point. The point is making bonkers claims detached from reality/not backed by evidence appropriate for his (implicitly) stated epistemic status or something like that. If he claimed that an ancient alien civilization had given him a telepathy device to interact with collective minds, that wouldn’t be “super-natural”, but would be roughly equally bonkers and bad, barring extraordinary evidence, which I don’t expect him to have.
It seems evident from the way he handles any sort of meaningful criticism, etc, even from MAPLE’s “spiritual father”:
Around the same time as the above, when Soryu was receiving scrutiny from donors and professional peers, this happened: Shinzen Young, a famous spiritual teacher who was a close collaborator with Soryu in the founding of MAPLE and a long time friend of his, attempted to hold him accountable and gave him some red-lines, which included (1) not speaking about societal collapse and climate/nuclear apocalypse so often to trainees, and (2) not having trainees work professional and intellectually heavy jobs as part of their unpaid training.
Soryu responded in this fashion: after he got off this call with Shinzen, Soryu organized an impromptu community-wide meeting. In this meeting he was pacing back and forth deriding Shinzen, questioning his integrity, debasing his “Materialist Humanist” worldview, and ultimately sabotaging Shinzen’s reputation in the community. While Shinzen helped found MAPLE and ordained most of the lay ordainees in the community (and was thus kind of a spiritual father to them), all formal affiliation between Shinzen and MAPLE was subsequently severed.And on the outside view:
The high-control literature shows that the most reliable lever to produce positive reform in such groups is sustained public scrutiny. Even then, adequate reform is rare. The more common response is doubling down.
TL;DR: It is often the case that the behavior of intelligent/agentic entities (in this case, I’m mostly thinking about humans and things made of humans, like orgs, cultures, etc.) towards some X is not adequately described in terms of having beliefs about X. One reason for this is that their aliefs (~behavioral predispositions guided by some “reasons in the world”) and their judgments (truth-oriented explicit thoughts / intellectual endorsements) discord/fail to cohere. Another reason is that even getting [[aliefs oriented towards some X] alone] or [[judgments oriented towards some X] alone] to cohere on their stance towards X is difficult, because the mind typically does not explicitly track all the epistemic/referential dependencies, and thus cannot invoke them on the spot to be available for coordinated revision.
(Beliefs are not special here. This is true of all mental elements that involve both abstract thought and contextually applied heuristic behavior[1], e.g., desires.)
(I’m writing this partly because most people I meet don’t have this fragmented mind picture integrated in their thinking about other minds.)
When discussing whether an entity (e.g., a person/community/organization) “believes that X”, it is helpful to split the notion of “belief” into two components.[2]
Belief = Alief + Judgment
I.e., one believes that X iff one alieves that X and one judges that X.
An alief is, in [my short words], a predisposition to behave as if someone believed something, regardless of whether one actually believes that.[3] One illustrative example from the Gendler article that introduced the concept is a person being terrified to cross a glass walkway over a canyon, even though they’re completely sure it’s safe. Various traumas, PTSD symptoms, and learned emotional reactions are also good examples of this phenomenon. They are like heuristics, typically developed for “good reasons”,[4] but also shallow, not explicitly/directly grounded in “representational content”/”explicit thought”, and therefore not as amenable to changes downstream from changed in the corresponding “representational content”/”explicit thought”.
A judgment[5] amounts to “intellectual endorsement”, representational/explicit thought, a truth-oriented representation, amenable to endorsed communication.
You can also think about this as: aliefs as Type 1 processes & judgments as Type 2 processes (in the sense explained here).
Aliefs and judgments influence each other, and we can meaningfully say that an entity believes X if that person’s aliefs and judgments have reached sufficient agreement on the matter of X.[6] But also, syncing various parts of a cognitive system with an update to one part is often difficult and far from perfectly reliable in the absence of reasonably strong signals/incentives towards it.
Aliefs saying X=0, while judgments “say” X=1 is not the only way to fail to reach internal agreement/coherence on X. Both aliefs and beliefs may “say” different things about X in different contexts, so there’s no alief-judgment agreement on X because aliefs and/or judgments cannot be ascribed a meaningful/coherent stance about X.
Examples:
“I don’t know why this org behaves as if they were oblivious to the fact that X (alief), when their public comms clearly reflect a good understanding of X (judgment).”
[Similar to the one above.] Someone is behaving as if they do not understand X, even though they have been told about X a few times already and said things indicating a good enough understanding of X in response.
Someone says and thinks X to be a good and worthwhile idea, and they end up finding themselves not doing it.
An important component of therapy is ensuring that updates in good directions propagate through the entire system. Often the judgment part makes the update, but the alief part remains unmoved. But it is also often the case that the alief part actively resists the judgment part being updated, because it kind of expects that something terrible may happen if such-and-such belief/judgment is adopted (see “exiles” in IFS).
In case it needs to be said, this doesn’t mean that judgments are universally (or even by-default) correct. Stuff like in the third example above can actually be good and healthy when one finds oneself unexcited or [currently steamless] about a project that “abstractly seems great and totally makes sense”.
(Acknowledgment: Influenced by Tsvi’s discussion of “internal sharing of elements”. See also more examples in the linked section of his post.)
- ^
I.e., “cut across the hierarchy” or something.
- ^
Or: insofar as one wants to talk about “beliefs”, I claim that this model is much more helpful than the lack of it. The further elaboration on this model is to think about various shards of various contextually invoked shards of alief and judgment.
- ^
This is circular because we lean on the notion of belief to define its component, but good enough for a short description.
- ^
Dennett would say “free-floating rationales”.
- ^
I’m borrowing “judgment” from Schwitzgebel’s The Pragmatist Metaphysics of Belief, where (IMO) he proposes to equate belief with something like “sufficiently generally applied alief”, regardless of the judgment. I disagree with his judgment here, which is evident from this post.
- ^
We can think about it as an example of “coherence” in the sense that Wentworth discusses in Coherence of Caches and Agents:
More generally, we’re typically interested in “coherence” in cases where all the local constraints together yield some useful property “at the large scale”. In logic, that might be a property like truth-preservation: put true assumptions in, get true conclusions out. In our fibonacci example, the useful “large scale” property is that the cache in fact contains the fibonacci sequence, all the way out to its largest entry. And for agents, the “large scale” property will be that the agents maximize or apply a lot of optimization pressure to something far in their future.
I can’t find a link now, but someone complained that meditation reduced their ability to feel love and joy, so they stopped.
I would guess you mean this (with Kaj’s comment just below the post).
Are you saying that a claim that one has interacted with a collective mind via samadhi does not involve anything supernatural (or at least very, very weird/implausible on priors, from the materialist/naturalist perspective, like having a sufficiently high-fidelity copy of the mind dormant in one’s brain)?
Things meditation can do:
Make you aware of certain dormant but easily wake-up-able motor pathways like the ones controlling the muscles around your ears.
This happened to me when I was 17ish, after a few years of playing around with meditation-y things, and in particular trying to focus my attention in the middle of my skull.
Gretta Duleba has an article about Relationship Known Issues. In software, a Known Issue is a problem that the developers know about but have no plans to fix—because it would be hard, because the issue doesn’t matter much, or because they just have more important things to do.
I also like Ozy Brennan’s https://thingofthings.wordpress.com/2019/04/19/four-kinds-of-relationship-problems/
Big thanks for the clarification.
I don’t know[1] how much the (most likely true) assumption that most people think they’re doing good actually buys us. People can have/develop very different conceptions of what Good is and what the instrumental strategies are that reliably get to the Good, sometimes so much that all hope of intelligibility is lost, especially when the person has twisted their environment, but also their own mind, into such a shape that questioning the load-bearing assumptions of what Good is and how to determine a Good strategy ~always fall flat.
- ^
When I say I don’t know, I mean that I don’t know, rather than that I very strongly doubt it.
- ^
Orienting Towards Oversight: Which AIs Should Want to Defect?
Wonderful post! I hope to dig deeper into it when I have more time.
In problems where the agent faces only one real choice, SCP demands that planning and doing agree: if it is rational to decide in advance “should I reach this point, do X,” then on reaching that point, X must be what it is rational to do. The motivation is a Ramsey-style test, which treats two questions as one — “what should I do if p turns out to be true?” and “p is true; what should I do?”
This loosening of the planning-doing identification assumption mirrors radical probabilism’s rejection of rigidity, i.e., of the requirement that upon learning about an event A, probabilities update according to
Quoting Abram (with a nice example):
A non-rigid update, on the other hand, means you don’t know how you’d react: “If I saw a convincing proof of P=NP, I wouldn’t know what to think. I’d have to consider it carefully.” I’ll call non-rigid updates fluid updates.
I would find Section 2 easier to parse if you explicitly stated the type signatures of all the components. My understanding is that
I think both cult-like/high control stuff and the models of AI (& other) risks at play are relevant if we want to understanding what’s going on.
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
The only lever we have here seems to be decreasing the inflow of more people and somehow getting to people already there, so that they are better-positioned to leave.
If this succeeds, it probably ends with Soryu gritting his teeth and sticking to his guns, concluding that the world is even worse than he thought, selling MAPLE, and departing to a hermit cabin (maybe with a few last followers) to telepathically talk to cyborgregores.
(This may sound sarcastic but I do think this is the best plausible outcome.)
Lots of rationalist believe that death is bad period, without any nuance or benefits hiding in fine print in a vast majority of cases.
I’m failing to get your point about the Waluigi effect and how it connects to what you are talking about.
ETA: And maybe it’s sleep deprivation but I don’t see how this comment reframes the one I replied to, rather than talking about something entirely different.
So far as this is true that all formalizations of belief are full subcategories of davidad!beliefs… why has this only been posted today???
Also, what about stuff like … various logics? AGM theory? Do you not count them as “beliefs”? They are less probability-like but Dempster-Shafer functions are very probability-like and I don’t see them here. I recall that Halpern (and a coauthor?)’s Generalized Expected Utility generalized almost every decision rule (?) but (some decision rule using?) Dempster-Shafer was the main interesting exception. Is something similar going on here?
Also, it seems to me that you’re equating beliefs with functions judging consistency of probability distributions with them. So if you believe that X and Y are independent then this is expressed by a belief function that judges accordingly. But I would expect that in some cases (not in the probabilistic independence case) this equates differently expressed propositional attitudes (different senses/intensions) because of having the same references/extensions, for the purpose of determining a belief function. Do you consider this an issue at all? My guess is that keeping the intentional difference in mind is relevant for handling ontological crisis-shaped stuff.
In any case, hooray for expanding the domain fo discourse to reveal the structure already there but hidden!
I massively appreciate this post and all that you’re doing for people harmed due to involvement with MAPLE.
Unfortunately, this paints a picture exceeding my worst expectations formed on reading herschel’s post from last year.[1]
I’ve interacted with three people who I know to have been residents at MAPLE. Two of them had their minds … very weirdly shaped in ways that are rather difficult to summarize in one sentence. One of them, when I expressed rather strong disagreement with some of their claims, would counter-claim that my reaction to this is a sign of my discomfort with the truth of what they were claiming or some shit like that. As far as I can tell, those two people are still heavily involved in MAPLE.
The third of my MAPLE resident acquaintances didn’t show any of those signs, but (1) IIRC their visit to MAPLE was quite many years ago; and (2) on their telling of the story, they were exceptionally assertive about giving feedback to Soryu (which, apparently, he actually took into consideration).
This suggests that a few years ago (I don’t remember how many) MAPLE was significantly more sane than it is nowadays.
In addition to those three, last year I also met Soryu himself. The “practitioner of Buddhism” mentioned in my post a few months ago was actually about Soryu:
Last year, I interacted with a practitioner of Buddhism who expressed a strange view to me, which I am now able to only vaguely recall. As far as I remember, the view was that as humans interact with each other, other living beings, and even the rest of the general non-living world around them, they are not passively allowing things to manifest themselves as they are, but rather imposing certain concepts on the Other, fitting the Other into preconceived frames. This is bad, the person said, because it puts us in “conflict” with the world. The right choice is to abandon all our concepts, as they are “violent”. If abandoning all the concepts means annihilation of the mind, so be it.
The bolded part coheres with the following from the OP:
Soryu considers human extinction a noble impulse and an ethically defensible goal. He has repeatedly shared that killing every human on Earth was his explicit life purpose from ages nine to nineteen—with CRISPR (a gene modifying technique that could be used to create a novel bio-weapon) as his best available means—until his vision shifted to awakening all beings as a superior alternative.
Soryu would also often take a term into abstraction: when people said that harm had been done, he and leadership would ask “what even is harm?”.
Reminds me when at some point I asked him about the meaning of some term that seemed load-bearing in his argumentation. To this, he bounced back to me, “What do I mean by that? Yeah, what do I mean by that?”, apparently expecting me to co-argue on his behalf?
You say that
MAPLE’s stated goals include founding the next world religion, leading a “world government” with “teeth” (its own military), opening 500 centers worldwide, and building an artificial superintelligence that will “turn the world into a monastery”
How many of those are publicly stated anywhere?
One final-for-now thing I wonder: What is going on in Soryu’s mind? The post has not speculated about this question, as it seems appropriate for a rather comprehensive, descriptive resource. But the question of what is actually happening in the black box of a ~cult leader and what are the latent factors driving the leader is relevant for understanding how to prevent/mitigate the concerning dynamics.
- ^
Insofar as I can accurately remember the impressions I formed upon reading it, but my memory of this impression seems probable on the current re-reading of the post.
- ^
I was also wondering about this … but insofar as some of the apparently-concerning things discussed here are “normal in [zen?] monasteries”, this doesn’t seem like a positive signal about those monasteries. It seems to me like you somewhat disagree?
Someone can very reliably hurt/incapacitate people / fuck them up really hard, while also earnestly claiming and believing (or at least thinking that they believe) that all of this is either instrumental to their noble goal or an acceptable collateral damage of things that are instrumental to their noble goal.
ETA: What I mean is that the concept of “intrinsically valuing” diverges/splinters/[is no longer a good concept] when talking about the behavior of agents that is inadequately described in terms of “having beliefs” (including about what they “value”, what is “good”, etc.), which I think is the case here. Or, like, the player is very oblivious to what the character is doing. (See also: Enemies vs Malefactors.)
Thoughts on leveraging this rather plausible scenario to stimulate governmental action towards doing something productive?