Mostly good point. Maybe a crux here is that Maple’s claims re risk are uh sort of plausible ish. (Sort of, I think Soryu’s understanding is kind of technically inept and whatever his spiritual interpretation is seems obviously wrong, but still.)
A big part of what I’m getting at is like, “in context re ai risk etc, and both their posture wrt it and their their adjacency, we should be taking them more seriously, rather than rounding them to modal cult.”
Or like. I think it’s entirely reasonable to ignore them, my claim is more like, if we’re going to engage, the uh naive anti cult model seems unfair/inappropriate.
I read you as saying something like “they are plausibly genuinely motivated by trying to reduce AI risk, so we shouldn’t necessarily assume that their cultish-looking behaviors come from the same generator as modal cult behaviors”.
People’s belief that they should have rest, free time, some money/time/energy to spend on objects of their choosing, abundant sleep, etc. [...]
People’s in-practice ability to “hang out”—to enjoy their friends, or the beach, in a “just being in the moment” kind of way. [...]
People’s understanding of whether commonsense morality holds, and of whether they can expect other folks in this space to also believe that commonsense morality holds. [...]
People’s in-practice tendency to have serious hobbies and to take a deep interest in how the world works. [...]
People’s ability to link in with ordinary institutions and take them seriously (e.g. to continue learning from their day job and caring about their colleagues’ progress and problems; to continue enjoying the dance club they used to dance at; to continue to take an interest in their significant other’s life and work; to continue learning from their PhD program; etc.) [...]
People’s understanding of what’s worth caring about, or what’s worth fighting for [...]
People’s understanding of when to use their own judgment and when to defer to others.
and that being in this kind of an emotional state will make you more inclined to join an organization that (intentionally or not) uses cult tactics to maintain and reinforce your state of distress. Or, if you are a certain kind of person, getting your faculties disrupted in this way may help convince you that the ends justify the means and that it’s fine to create an organization running on cult tactics, as long as you can tell yourself that it will help reduce AI risk in the end.
I think there’s also some reasonable grounds to believe that often this disruption is not really about AI risk. Rather it’s tapping into some pre-existing psychological distress and dismantling existing defenses against it and/or magnifying it until the person can no longer think clearly.
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
So, to me, “these people genuinely believe in AI risk and take it super seriously” is not a strong reason to think “so we should discount the probability of their concerning behaviors being cult-like in a bad way”. Rather, it raises the probability that they are gravitating toward a cult-like attractor, because a mind super seriously believing that X is the most overriding priority in the world is exactly what generates many of the nasty kinds of cults.
“It’s fine for all members of this group to become arbitrarily traumatized and hurt if that furthers X” is just the kind of thing that follows from that emotional and cognitive state. That naturally leads to the leaders ruthlessly wielding that power—believing that they have a moral obligation to wield it, even. And now they’ve created a power dynamic where the members have an incentive to try to overthrow or manipulate the leader to protect themselves, so the leader has to entrench their position and make sure the members really stay in line… and then there’s an ongoing power dynamic that’s going to corrupt the extent to which the group ever even was optimizing for X.
Here someone might raise the objection of “but if X really was the most important thing in the world, wouldn’t it then be ethically justified”… and I don’t think so. The kind of group where everyone knows they can be sacrificed for the cause and where the leadership doesn’t have any ethical injunctions is very likely to create a sick system, intentionally or not. Now you’re in constant distress about the approaching end of the world and about possibly being discarded by your leaders and peers. Not a good environment for thinking clearly, and the underlying psychological distress dynamic is not really optimizing for X anyway, even if may produce lots of actions that superficially look like furthering X.
Obviously, this doesn’t mean that AI risk arguments would be wrong. Nor does it mean that AI risk isn’t an extremely important thing to focus on. But it does mean that believing in AI risk is correlated with going crazy and sinking into dysfunction. So we should strive to lower that correlation by making sure that people working on AI risk have environments that are otherwise actively the opposite of the cult attractors. And make sure leaders in the movement hold on to ethical injunctions where some things are forbidden even if they were to save the world. (Of course, what exactly those injunctions should be is subject to reasonable disagreement… but leaders should demonstrate at least having some.)
It also doesn’t automatically mean that MAPLE is falling victim to this! Or even that if it was, that it would be all bad. An organization can go in this direction and still have value, or not go totally bad. And as has been mentioned, e.g. militaries do seem to often (but certainly not always) avoid many of these failure modes, despite considering all members in principle expendable. (I think a healthy military’s emotional core is coming from a different place than the cult-like distress.)
But I think it does significantly raise rather than lower the odds that this is happening.
Feels maybe pedantic on my part to keep responding? In any case:
“Maple is worried about ai risk and trying to do something about it” isn’t necessarily the whole argument, and I’m not even sure I fully support whatever the steelman would be. But here’s some attempt at such a steelman:
“Maple’s account of the central upstream causes of ai risk (let’s say, perverse market incentives + rivalrous dynamics betw state actors, though maybe this is giving them too much credit) actually seems plausible, and sort of in a way that even Eliezer would agree with, and they are… trying to address those causes. We should be addressing them in these terms, and pointing out if they are object level confused about the causality, or their theory of change is implausible, or they are ineffective, and separately if they are meanwhile harmful. Saying they fit the mold of a classic high-control group and leaving members traumatized doesn’t really get at their cruxes, even if it’s certainly a bad look.”
(This is a better way of expressing it than I had last year or when I was writing my top level comment, actually.)
I think both cult-like/high control stuff and the models of AI (& other) risks at play are relevant if we want to understanding what’s going on.
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
The only lever we have here seems to be decreasing the inflow of more people and somehow getting to people already there, so that they are better-positioned to leave.
If this succeeds, it probably ends with Soryu gritting his teeth and sticking to his guns, concluding that the world is even worse than he thought, selling MAPLE, and departing to a hermit cabin (maybe with a few last followers) to telepathically talk to cyborgregores.
(This may sound sarcastic but I do think this is the best plausible outcome.)
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
It seems evident from the way he handles any sort of meaningful criticism, etc, even from MAPLE’s “spiritual father”:
Around the same time as the above, when Soryu was receiving scrutiny from donors and professional peers, this happened: Shinzen Young, a famous spiritual teacher who was a close collaborator with Soryu in the founding of MAPLE and a long time friend of his, attempted to hold him accountable and gave him some red-lines, which included (1) not speaking about societal collapse and climate/nuclear apocalypse so often to trainees, and (2) not having trainees work professional and intellectually heavy jobs as part of their unpaid training.
Soryu responded in this fashion: after he got off this call with Shinzen, Soryu organized an impromptu community-wide meeting. In this meeting he was pacing back and forth deriding Shinzen, questioning his integrity, debasing his “Materialist Humanist” worldview, and ultimately sabotaging Shinzen’s reputation in the community. While Shinzen helped found MAPLE and ordained most of the lay ordainees in the community (and was thus kind of a spiritual father to them), all formal affiliation between Shinzen and MAPLE was subsequently severed.
And on the outside view:
The high-control literature shows that the most reliable lever to produce positive reform in such groups is sustained public scrutiny. Even then, adequate reform is rare. The more common response is doubling down.
Soryu responded in this fashion: after he got off this call with Shinzen, Soryu organized an impromptu community-wide meeting. In this meeting he was pacing back and forth deriding Shinzen, questioning his integrity, debasing his “Materialist Humanist” worldview, and ultimately sabotaging Shinzen’s reputation in the community. While Shinzen helped found MAPLE and ordained most of the lay ordainees in the community (and was thus kind of a spiritual father to them), all formal affiliation between Shinzen and MAPLE was subsequently severed.
I was at this meeting, and remember it quite different. Soryu came and said that there were 3 things Shinzen recommended, and that it was great to finally have someone who actually gave concrete recommendations. He told us what they were verbatim, and then asked asked us to carefully consider them over the next little bit and see if we should implement them. We then carefully considered them and did implement some changes, although not everything Shinzen had asked for.
Kyle’s whole narrative here just seems to be wrong to me.
ETA: Also a concrete fact here re “spiritual severing”. While we can dispute the things that were said here until I check the recording, I do know for a fact that Shinzen was invited to run the next lay ordination, and he was the one who declined.
Soryu’s asking us if we should take on those “recommendations” from Shinzen were, by my understanding at least, largely rhetorical. With Soryu’s statements and tone within that talk it was clear he did not favor abiding by them at all.
The talk was so inappropriate[1] and derisive of Shinzen that the executive director and head of Care (the two other branches of the triumvirate you spoke of in another comment) immediately held a meeting with Soryu the next morning pressing him to begin receiving “help,” of the professional kind. Soryu didn’t concede to this.
I heard and was a part of multiple conversations with residents the next day who were either worried about Soryu’s mental health or were questioning if Soryu was truly as trustworthy as he claimed to be. One resident told me “well, I guess that’s the end of that long friendship (of Soryu’s and Shinzen’s).”
I named two of Shinzen’s demands, I can’t remember the third. Could you please share it? And what are the changes that you speak of that were implemented following this talk?
Also, I believe you phrasing them as “recommendations” from Shinzen as misleading. I understood them to be red-lines for Shinzen’s support. That Shinzen could no longer support MAPLE if these “recommendations” were not met.
If I’m wrong, one thing that could help clarify this would be finding and publishing the recording of the session (as all lectures at MAPLE are recorded). If that’s too public, we could also pick a few other people like Kaj Sotala to listen and give their takes.
I use the term inappropriate because hosting a community wide flash-impromptu meeting to berate someone with big standing in the community (helped found MAPLE, hosted multiple retreats there, ordained many MAPLE residents), without them there to represent themselves was not a healthy medium for resolving the situation
Soryu’s asking us if we should take on those “recommendations” from Shinzen were, by my understanding at least, largely rhetorical. With Soryu’s statements and tone within that talk it was clear he did not favor abiding by them at all.
If it was rhetorical (which I did not take it as), it was not treated that way within the community. The 3 recommendations were repeated many times, brought up and discussed in multiple group discussions, by multiple different people in leadership, discussed in all group meetings as well as leadership meetings, with, and without Soryu present.
Most of the changes I saw came around changing our admissions process to better filter people and doing a better job of explaining the rigour of the training.
Soryu had just had his close friend call him, screaming, and any attempt by him to talk or ask clarifying questions was met with him being screamed over. My sense of the situation was someone who had just dealt with that, nevertheless trying to take the feedback seriously.
It’s understandable that you disagree with Kyle’s assessment of MAPLE and of this meeting; that’s why you’re still there and he’s not. As an interested bystander, let me suggest reserving at least one of these comment threads for identifying and resolving factual disputes.
There’s a factual dispute here that could be resolved by making the full meeting recording available to a trusted third party. Would you respond to that suggestion?
This is the kind of objection I’m excited to see, and actually made me update down (tentatively, a bit) on the OP. Releasing that recording, or allowing trusted third parties to listen to it and attest to its contents would make me update more, and the more specific claims where you can say “no, indeed, Kyle is exaggerating” the more seriously I’ll take your objection in general.
(I have my own perspective already regardless of the OP.)
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
For example, Scientology used to advertise itself as a way to prevent a nuclear war, or at least to survive if it happens anyway.
So if you believed that the nuclear war was a serious risk, then by the same logic it made perfect sense to volunteer for sleep deprivation and all kinds of abuse, even if from outside view the entire organization had 0 impact on the probability of nuclear war.
Mostly good point. Maybe a crux here is that Maple’s claims re risk are uh sort of plausible ish. (Sort of, I think Soryu’s understanding is kind of technically inept and whatever his spiritual interpretation is seems obviously wrong, but still.)
A big part of what I’m getting at is like, “in context re ai risk etc, and both their posture wrt it and their their adjacency, we should be taking them more seriously, rather than rounding them to modal cult.”
Or like. I think it’s entirely reasonable to ignore them, my claim is more like, if we’re going to engage, the uh naive anti cult model seems unfair/inappropriate.
I read you as saying something like “they are plausibly genuinely motivated by trying to reduce AI risk, so we shouldn’t necessarily assume that their cultish-looking behaviors come from the same generator as modal cult behaviors”.
My disagreement with that is that learning about AI risk is known to often disrupt various fundamental faculties, such as:
and that being in this kind of an emotional state will make you more inclined to join an organization that (intentionally or not) uses cult tactics to maintain and reinforce your state of distress. Or, if you are a certain kind of person, getting your faculties disrupted in this way may help convince you that the ends justify the means and that it’s fine to create an organization running on cult tactics, as long as you can tell yourself that it will help reduce AI risk in the end.
I think there’s also some reasonable grounds to believe that often this disruption is not really about AI risk. Rather it’s tapping into some pre-existing psychological distress and dismantling existing defenses against it and/or magnifying it until the person can no longer think clearly.
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
So, to me, “these people genuinely believe in AI risk and take it super seriously” is not a strong reason to think “so we should discount the probability of their concerning behaviors being cult-like in a bad way”. Rather, it raises the probability that they are gravitating toward a cult-like attractor, because a mind super seriously believing that X is the most overriding priority in the world is exactly what generates many of the nasty kinds of cults.
“It’s fine for all members of this group to become arbitrarily traumatized and hurt if that furthers X” is just the kind of thing that follows from that emotional and cognitive state. That naturally leads to the leaders ruthlessly wielding that power—believing that they have a moral obligation to wield it, even. And now they’ve created a power dynamic where the members have an incentive to try to overthrow or manipulate the leader to protect themselves, so the leader has to entrench their position and make sure the members really stay in line… and then there’s an ongoing power dynamic that’s going to corrupt the extent to which the group ever even was optimizing for X.
Here someone might raise the objection of “but if X really was the most important thing in the world, wouldn’t it then be ethically justified”… and I don’t think so. The kind of group where everyone knows they can be sacrificed for the cause and where the leadership doesn’t have any ethical injunctions is very likely to create a sick system, intentionally or not. Now you’re in constant distress about the approaching end of the world and about possibly being discarded by your leaders and peers. Not a good environment for thinking clearly, and the underlying psychological distress dynamic is not really optimizing for X anyway, even if may produce lots of actions that superficially look like furthering X.
Obviously, this doesn’t mean that AI risk arguments would be wrong. Nor does it mean that AI risk isn’t an extremely important thing to focus on. But it does mean that believing in AI risk is correlated with going crazy and sinking into dysfunction. So we should strive to lower that correlation by making sure that people working on AI risk have environments that are otherwise actively the opposite of the cult attractors. And make sure leaders in the movement hold on to ethical injunctions where some things are forbidden even if they were to save the world. (Of course, what exactly those injunctions should be is subject to reasonable disagreement… but leaders should demonstrate at least having some.)
It also doesn’t automatically mean that MAPLE is falling victim to this! Or even that if it was, that it would be all bad. An organization can go in this direction and still have value, or not go totally bad. And as has been mentioned, e.g. militaries do seem to often (but certainly not always) avoid many of these failure modes, despite considering all members in principle expendable. (I think a healthy military’s emotional core is coming from a different place than the cult-like distress.)
But I think it does significantly raise rather than lower the odds that this is happening.
Mostly good takes.
Feels maybe pedantic on my part to keep responding? In any case:
“Maple is worried about ai risk and trying to do something about it” isn’t necessarily the whole argument, and I’m not even sure I fully support whatever the steelman would be. But here’s some attempt at such a steelman:
“Maple’s account of the central upstream causes of ai risk (let’s say, perverse market incentives + rivalrous dynamics betw state actors, though maybe this is giving them too much credit) actually seems plausible, and sort of in a way that even Eliezer would agree with, and they are… trying to address those causes. We should be addressing them in these terms, and pointing out if they are object level confused about the causality, or their theory of change is implausible, or they are ineffective, and separately if they are meanwhile harmful. Saying they fit the mold of a classic high-control group and leaving members traumatized doesn’t really get at their cruxes, even if it’s certainly a bad look.”
(This is a better way of expressing it than I had last year or when I was writing my top level comment, actually.)
I think both cult-like/high control stuff and the models of AI (& other) risks at play are relevant if we want to understanding what’s going on.
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
The only lever we have here seems to be decreasing the inflow of more people and somehow getting to people already there, so that they are better-positioned to leave.
If this succeeds, it probably ends with Soryu gritting his teeth and sticking to his guns, concluding that the world is even worse than he thought, selling MAPLE, and departing to a hermit cabin (maybe with a few last followers) to telepathically talk to cyborgregores.
(This may sound sarcastic but I do think this is the best plausible outcome.)
What makes you think this?
It seems evident from the way he handles any sort of meaningful criticism, etc, even from MAPLE’s “spiritual father”:
And on the outside view:
I was at this meeting, and remember it quite different. Soryu came and said that there were 3 things Shinzen recommended, and that it was great to finally have someone who actually gave concrete recommendations. He told us what they were verbatim, and then asked asked us to carefully consider them over the next little bit and see if we should implement them. We then carefully considered them and did implement some changes, although not everything Shinzen had asked for.
Kyle’s whole narrative here just seems to be wrong to me.
ETA: Also a concrete fact here re “spiritual severing”. While we can dispute the things that were said here until I check the recording, I do know for a fact that Shinzen was invited to run the next lay ordination, and he was the one who declined.
Soryu’s asking us if we should take on those “recommendations” from Shinzen were, by my understanding at least, largely rhetorical. With Soryu’s statements and tone within that talk it was clear he did not favor abiding by them at all.
The talk was so inappropriate[1] and derisive of Shinzen that the executive director and head of Care (the two other branches of the triumvirate you spoke of in another comment) immediately held a meeting with Soryu the next morning pressing him to begin receiving “help,” of the professional kind. Soryu didn’t concede to this.
I heard and was a part of multiple conversations with residents the next day who were either worried about Soryu’s mental health or were questioning if Soryu was truly as trustworthy as he claimed to be. One resident told me “well, I guess that’s the end of that long friendship (of Soryu’s and Shinzen’s).”
I named two of Shinzen’s demands, I can’t remember the third. Could you please share it? And what are the changes that you speak of that were implemented following this talk?
Also, I believe you phrasing them as “recommendations” from Shinzen as misleading. I understood them to be red-lines for Shinzen’s support. That Shinzen could no longer support MAPLE if these “recommendations” were not met.
If I’m wrong, one thing that could help clarify this would be finding and publishing the recording of the session (as all lectures at MAPLE are recorded). If that’s too public, we could also pick a few other people like Kaj Sotala to listen and give their takes.
I use the term inappropriate because hosting a community wide flash-impromptu meeting to berate someone with big standing in the community (helped found MAPLE, hosted multiple retreats there, ordained many MAPLE residents), without them there to represent themselves was not a healthy medium for resolving the situation
If it was rhetorical (which I did not take it as), it was not treated that way within the community. The 3 recommendations were repeated many times, brought up and discussed in multiple group discussions, by multiple different people in leadership, discussed in all group meetings as well as leadership meetings, with, and without Soryu present.
Most of the changes I saw came around changing our admissions process to better filter people and doing a better job of explaining the rigour of the training.
Soryu had just had his close friend call him, screaming, and any attempt by him to talk or ask clarifying questions was met with him being screamed over. My sense of the situation was someone who had just dealt with that, nevertheless trying to take the feedback seriously.
Trinley,
It’s understandable that you disagree with Kyle’s assessment of MAPLE and of this meeting; that’s why you’re still there and he’s not. As an interested bystander, let me suggest reserving at least one of these comment threads for identifying and resolving factual disputes.
There’s a factual dispute here that could be resolved by making the full meeting recording available to a trusted third party. Would you respond to that suggestion?
Hey friend,
I don’t currently have the authority to make this happen unfortunately.
Thank you for responding to this; I appreciate it.
I’m pretty surprised by the disagree vote. Does someone think I DO have the authority to make this happen?
This is the kind of objection I’m excited to see, and actually made me update down (tentatively, a bit) on the OP. Releasing that recording, or allowing trusted third parties to listen to it and attest to its contents would make me update more, and the more specific claims where you can say “no, indeed, Kyle is exaggerating” the more seriously I’ll take your objection in general.
(I have my own perspective already regardless of the OP.)
For example, Scientology used to advertise itself as a way to prevent a nuclear war, or at least to survive if it happens anyway.
So if you believed that the nuclear war was a serious risk, then by the same logic it made perfect sense to volunteer for sleep deprivation and all kinds of abuse, even if from outside view the entire organization had 0 impact on the probability of nuclear war.