I read you as saying something like “they are plausibly genuinely motivated by trying to reduce AI risk, so we shouldn’t necessarily assume that their cultish-looking behaviors come from the same generator as modal cult behaviors”.
People’s belief that they should have rest, free time, some money/time/energy to spend on objects of their choosing, abundant sleep, etc. [...]
People’s in-practice ability to “hang out”—to enjoy their friends, or the beach, in a “just being in the moment” kind of way. [...]
People’s understanding of whether commonsense morality holds, and of whether they can expect other folks in this space to also believe that commonsense morality holds. [...]
People’s in-practice tendency to have serious hobbies and to take a deep interest in how the world works. [...]
People’s ability to link in with ordinary institutions and take them seriously (e.g. to continue learning from their day job and caring about their colleagues’ progress and problems; to continue enjoying the dance club they used to dance at; to continue to take an interest in their significant other’s life and work; to continue learning from their PhD program; etc.) [...]
People’s understanding of what’s worth caring about, or what’s worth fighting for [...]
People’s understanding of when to use their own judgment and when to defer to others.
and that being in this kind of an emotional state will make you more inclined to join an organization that (intentionally or not) uses cult tactics to maintain and reinforce your state of distress. Or, if you are a certain kind of person, getting your faculties disrupted in this way may help convince you that the ends justify the means and that it’s fine to create an organization running on cult tactics, as long as you can tell yourself that it will help reduce AI risk in the end.
I think there’s also some reasonable grounds to believe that often this disruption is not really about AI risk. Rather it’s tapping into some pre-existing psychological distress and dismantling existing defenses against it and/or magnifying it until the person can no longer think clearly.
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
So, to me, “these people genuinely believe in AI risk and take it super seriously” is not a strong reason to think “so we should discount the probability of their concerning behaviors being cult-like in a bad way”. Rather, it raises the probability that they are gravitating toward a cult-like attractor, because a mind super seriously believing that X is the most overriding priority in the world is exactly what generates many of the nasty kinds of cults.
“It’s fine for all members of this group to become arbitrarily traumatized and hurt if that furthers X” is just the kind of thing that follows from that emotional and cognitive state. That naturally leads to the leaders ruthlessly wielding that power—believing that they have a moral obligation to wield it, even. And now they’ve created a power dynamic where the members have an incentive to try to overthrow or manipulate the leader to protect themselves, so the leader has to entrench their position and make sure the members really stay in line… and then there’s an ongoing power dynamic that’s going to corrupt the extent to which the group ever even was optimizing for X.
Here someone might raise the objection of “but if X really was the most important thing in the world, wouldn’t it then be ethically justified”… and I don’t think so. The kind of group where everyone knows they can be sacrificed for the cause and where the leadership doesn’t have any ethical injunctions is very likely to create a sick system, intentionally or not. Now you’re in constant distress about the approaching end of the world and about possibly being discarded by your leaders and peers. Not a good environment for thinking clearly, and the underlying psychological distress dynamic is not really optimizing for X anyway, even if may produce lots of actions that superficially look like furthering X.
Obviously, this doesn’t mean that AI risk arguments would be wrong. Nor does it mean that AI risk isn’t an extremely important thing to focus on. But it does mean that believing in AI risk is correlated with going crazy and sinking into dysfunction. So we should strive to lower that correlation by making sure that people working on AI risk have environments that are otherwise actively the opposite of the cult attractors. And make sure leaders in the movement hold on to ethical injunctions where some things are forbidden even if they were to save the world. (Of course, what exactly those injunctions should be is subject to reasonable disagreement… but leaders should demonstrate at least having some.)
It also doesn’t automatically mean that MAPLE is falling victim to this! Or even that if it was, that it would be all bad. An organization can go in this direction and still have value, or not go totally bad. And as has been mentioned, e.g. militaries do seem to often (but certainly not always) avoid many of these failure modes, despite considering all members in principle expendable. (I think a healthy military’s emotional core is coming from a different place than the cult-like distress.)
But I think it does significantly raise rather than lower the odds that this is happening.
Feels maybe pedantic on my part to keep responding? In any case:
“Maple is worried about ai risk and trying to do something about it” isn’t necessarily the whole argument, and I’m not even sure I fully support whatever the steelman would be. But here’s some attempt at such a steelman:
“Maple’s account of the central upstream causes of ai risk (let’s say, perverse market incentives + rivalrous dynamics betw state actors, though maybe this is giving them too much credit) actually seems plausible, and sort of in a way that even Eliezer would agree with, and they are… trying to address those causes. We should be addressing them in these terms, and pointing out if they are object level confused about the causality, or their theory of change is implausible, or they are ineffective, and separately if they are meanwhile harmful. Saying they fit the mold of a classic high-control group and leaving members traumatized doesn’t really get at their cruxes, even if it’s certainly a bad look.”
(This is a better way of expressing it than I had last year or when I was writing my top level comment, actually.)
I think both cult-like/high control stuff and the models of AI (& other) risks at play are relevant if we want to understanding what’s going on.
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
The only lever we have here seems to be decreasing the inflow of more people and somehow getting to people already there, so that they are better-positioned to leave.
If this succeeds, it probably ends with Soryu gritting his teeth and sticking to his guns, concluding that the world is even worse than he thought, selling MAPLE, and departing to a hermit cabin (maybe with a few last followers) to telepathically talk to cyborgregores.
(This may sound sarcastic but I do think this is the best plausible outcome.)
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
For example, Scientology used to advertise itself as a way to prevent a nuclear war, or at least to survive if it happens anyway.
So if you believed that the nuclear war was a serious risk, then by the same logic it made perfect sense to volunteer for sleep deprivation and all kinds of abuse, even if from outside view the entire organization had 0 impact on the probability of nuclear war.
I read you as saying something like “they are plausibly genuinely motivated by trying to reduce AI risk, so we shouldn’t necessarily assume that their cultish-looking behaviors come from the same generator as modal cult behaviors”.
My disagreement with that is that learning about AI risk is known to often disrupt various fundamental faculties, such as:
and that being in this kind of an emotional state will make you more inclined to join an organization that (intentionally or not) uses cult tactics to maintain and reinforce your state of distress. Or, if you are a certain kind of person, getting your faculties disrupted in this way may help convince you that the ends justify the means and that it’s fine to create an organization running on cult tactics, as long as you can tell yourself that it will help reduce AI risk in the end.
I think there’s also some reasonable grounds to believe that often this disruption is not really about AI risk. Rather it’s tapping into some pre-existing psychological distress and dismantling existing defenses against it and/or magnifying it until the person can no longer think clearly.
If this take is correct, then I think it would be very surprising if AI risk happened to be the only intellectual argument that managed to ever have that effect. Rather, there are lots of arguments of the form “X is the most important thing in the world and why you should sacrifice everything on the altar of X”, with various cults being created around different versions of it.
So, to me, “these people genuinely believe in AI risk and take it super seriously” is not a strong reason to think “so we should discount the probability of their concerning behaviors being cult-like in a bad way”. Rather, it raises the probability that they are gravitating toward a cult-like attractor, because a mind super seriously believing that X is the most overriding priority in the world is exactly what generates many of the nasty kinds of cults.
“It’s fine for all members of this group to become arbitrarily traumatized and hurt if that furthers X” is just the kind of thing that follows from that emotional and cognitive state. That naturally leads to the leaders ruthlessly wielding that power—believing that they have a moral obligation to wield it, even. And now they’ve created a power dynamic where the members have an incentive to try to overthrow or manipulate the leader to protect themselves, so the leader has to entrench their position and make sure the members really stay in line… and then there’s an ongoing power dynamic that’s going to corrupt the extent to which the group ever even was optimizing for X.
Here someone might raise the objection of “but if X really was the most important thing in the world, wouldn’t it then be ethically justified”… and I don’t think so. The kind of group where everyone knows they can be sacrificed for the cause and where the leadership doesn’t have any ethical injunctions is very likely to create a sick system, intentionally or not. Now you’re in constant distress about the approaching end of the world and about possibly being discarded by your leaders and peers. Not a good environment for thinking clearly, and the underlying psychological distress dynamic is not really optimizing for X anyway, even if may produce lots of actions that superficially look like furthering X.
Obviously, this doesn’t mean that AI risk arguments would be wrong. Nor does it mean that AI risk isn’t an extremely important thing to focus on. But it does mean that believing in AI risk is correlated with going crazy and sinking into dysfunction. So we should strive to lower that correlation by making sure that people working on AI risk have environments that are otherwise actively the opposite of the cult attractors. And make sure leaders in the movement hold on to ethical injunctions where some things are forbidden even if they were to save the world. (Of course, what exactly those injunctions should be is subject to reasonable disagreement… but leaders should demonstrate at least having some.)
It also doesn’t automatically mean that MAPLE is falling victim to this! Or even that if it was, that it would be all bad. An organization can go in this direction and still have value, or not go totally bad. And as has been mentioned, e.g. militaries do seem to often (but certainly not always) avoid many of these failure modes, despite considering all members in principle expendable. (I think a healthy military’s emotional core is coming from a different place than the cult-like distress.)
But I think it does significantly raise rather than lower the odds that this is happening.
Mostly good takes.
Feels maybe pedantic on my part to keep responding? In any case:
“Maple is worried about ai risk and trying to do something about it” isn’t necessarily the whole argument, and I’m not even sure I fully support whatever the steelman would be. But here’s some attempt at such a steelman:
“Maple’s account of the central upstream causes of ai risk (let’s say, perverse market incentives + rivalrous dynamics betw state actors, though maybe this is giving them too much credit) actually seems plausible, and sort of in a way that even Eliezer would agree with, and they are… trying to address those causes. We should be addressing them in these terms, and pointing out if they are object level confused about the causality, or their theory of change is implausible, or they are ineffective, and separately if they are meanwhile harmful. Saying they fit the mold of a classic high-control group and leaving members traumatized doesn’t really get at their cruxes, even if it’s certainly a bad look.”
(This is a better way of expressing it than I had last year or when I was writing my top level comment, actually.)
I think both cult-like/high control stuff and the models of AI (& other) risks at play are relevant if we want to understanding what’s going on.
I’m not hopeful about pointing to the lack of instrumentality to their supposed goals in the way the org is now as a means to making them update because it/Soryu seems completely oblivious to this lack of instrumentality.
The only lever we have here seems to be decreasing the inflow of more people and somehow getting to people already there, so that they are better-positioned to leave.
If this succeeds, it probably ends with Soryu gritting his teeth and sticking to his guns, concluding that the world is even worse than he thought, selling MAPLE, and departing to a hermit cabin (maybe with a few last followers) to telepathically talk to cyborgregores.
(This may sound sarcastic but I do think this is the best plausible outcome.)
For example, Scientology used to advertise itself as a way to prevent a nuclear war, or at least to survive if it happens anyway.
So if you believed that the nuclear war was a serious risk, then by the same logic it made perfect sense to volunteer for sleep deprivation and all kinds of abuse, even if from outside view the entire organization had 0 impact on the probability of nuclear war.