I am interested if you do think there are many people whose CEV would be either zero-value or net negative? My first thought is that people who are deeply psychologically impaired or broken might be like this (e.g. sociopaths—perhaps SBF), but I am not sure.
Edit: I no longer think that SBF is an especially central/useful pointer here.
@Zach Stein-Perlman Where’s your thumbs-down coming from? SBF is the psychopath that I personally have met a few times, that people know of, and who got a lot of power, so it seems like a useful example to mention. Also don’t forget his personal diary entries read like this:
In a lot of ways I don’t really have a soul. This is a lot more obvious in some contexts than others. But in the end there’s a pretty decent argument that my empathy is fake, my feelings are fake, my facial reactions are fake. I don’t feel happiness.
Reading your first comment above, I reacted as follows:
”What is going on, surely Putin is on the sociopathy/psychopathy spectrum too and not as endearing as SBF with effective altruism as an autistic special interest, so how is this even a question/comparison?”
(With SBF’s CEV I’d admittedly be quite concerned about the greed/recklessness inherent to his philosophy of utilitarianism and risk-taking, and I actually think that sort of thing could backfire uniquely-badly with AIs optimizing everything. But other than that, personality-wise, I just want to flag that while it’s probably always net bad to have people high on the sociopathy/psychopathy spectrum in leadership positions, there’s still a huge difference between the ones where your main concern is only large-scale fraud, vs the ones where the concern is unnecessary wars, genocide, and torture camps.)
I wonder, have many people on here not read the many examples of how harmful certain types of personality can be, for instance in this post and this one?
I wonder, have many people on here not read the many examples of how harmful certain types of personality can be, for instance in this post and this one?
I have strong-downvoted both of those posts! I think they do a mixture of the “taking propaganda at face value” thing, seem to largely be in some kind of negative affect spiral around imagining arbitrarily bad qualities they can assign to people, seem to take an enormously naive view of the psychopathology literature, seem to consistently avoid dealing with the moral reflection dynamics I outline in this post, and seem very rooted in negative utilitarianism, which I disagree with[1].
Overall, I am pretty sad about that whole set of stuff. I somewhat wish I could talk more to people about it, but the negative utilitarianism-leaning stuff usually makes good-faith discourse a lot harder (since the bottom-line for my interlocutors seems clearly written).
Like man, those posts are so frustrating to read. They just… assert things. No epistemic statuses, no caveats, just stuff like this:
The most ardent fanatics are utterly convinced they have found the one infallible authority in possession of ultimate truth and righteousness; they are textbook dogmatists (Rokeach, 1960). For religious fundamentalists, this is usually a holy book containing the divine revelation of God and his prophets. For Nazis, it was Hitler’s Führerprinzip (Leader Principle), codified by Rudolf Hess’s declaration that “the Führer is always right”. Similarly, many communist revolutionaries essentially placed absolute faith in foundational texts like Marx’s Das Kapital, or in the Party itself (Montefiore, 2007). “Angkar is an organization that cannot make mistakes” was a key slogan of the Khmer Rouge.[4]
For the fanatic, any doubt or deviation from these dogmas is not only wrong but evil, culminating in a total “soldier mindset” which defends the pre-existing ideology at all costs (Galef, 2021). This necessitates abandoning even the most basic form of empiricism by “rejecting the evidence of one’s own eyes and ears”, to paraphrase Orwell.[5] The fanatic is thus essentially incorrigible and has no epistemic or moral uncertainty, even in the face of widespread opposition (Gollwitzer et al., 2022).[6]
It honestly reads to me kind of like true crime podcasts? They love always describing everyone they cover with these kinds of extreme characterizations which basically never hold up when you look into them. My honest best guess is these posts serve a kind of similar role within our community.
I am confident the psychological characterization in paragraphs like the above is wrong. “Defends pre-existing ideology at all costs”. Come on, no, of course not “all costs”. Indeed a huge fraction of people caught up in these ideologies end up deconverting or drastically mellowing their views when their social context changes. These seem hilariously strawmanny absolutes that get ascribed to people here.
The fanatic is thus essentially incorrigible and has no epistemic or moral uncertainty, even in the face of widespread opposition
This is exactly the kind of thing I mean when I say people seem really got by propaganda, and when people engage in the fundamental attribution error. Like, I don’t know how I would differentiate paragraphs like the above from ravings of religious people talking about demons or devils.
It seems like in these posts all negative attributes must be assigned to these people, and all their cognitive leanings must be absolute and unwielding. They must “defend the pre-existing ideology at all costs”, they are “utterly convinced”, they are “textbook dogmatists”, they place “absolute faith” in the holy texts. Come on, this is not how people work.
Thanks for sharing that take, which I find largely quite bizarre and surprising. I continue to think those posts are super valuable.
I can understand finding the negative(-leaning) utilitarianism codedness of the writings annoying, but I don’t see why you think it makes good-faith discussion a lot harder. From my perspective, a lot of writings on LW are “yay, hurrah team human”-coded in a way that annoys the crap out of me and makes me want to punch things, but it’s not like that means I can’t get valuable things out of the writings or have to treat the posters here as necessarily adversarial.
I am confident the psychological characterization in paragraphs like the above is wrong. “Defends pre-existing ideology at all costs”. Come on, no, of course not “all costs”. Indeed a huge fraction of people caught up in these ideologies end up deconverting or drastically mellowing their views when their social context changes.
The subject of the sentence you put in quotation marks was “The fanatic”—as in, “the archetypal example of the fanatic.” This sounds to me like more of a writing style issue. Like, it wouldn’t even occur to me to assume that the post is saying that every believer of a harmful ideology is like that. More that there’s a fanaticism attractor and that people at the center of it really do approach that described extreme—which I think is true? I guess that’s the thing you’re contesting, but I feel like history contains some examples of atrocities that are hard to explain without choosing at least one of the following: either some people can get incredibly fanatical, or some people are sadistic/evil and may use fanaticism/ideology as a cover. Either way, at least one of the posts’ messages must be true?
There are some far-future relevance speculations where my model of what is likely to happen with AI is different from David’s, so I think it’s just much more likely that AIs will take over and it won’t matter what the humans who built it were like. But that doesn’t really invalidate too much—I mean part of the posts are also about what sort of qualities we wouldn’t want to see in AIs that we build. On your point about how AI-aided moral reflection and overcoming resource scarcity would reduce fanaticism or other bad consequences from “bad values,” I think that sort of point is underappreciated in some EA circles, but it’s not like it’s obvious that this is what’s going to happen. I think the posts we’re discussing engage well with reasons why reflection might not solve all the problems.
The subject of the sentence you put in quotation marks was “The fanatic”—as in, “the archetypal example of the fanatic.” This sounds to me like more of a writing style issue. Like, it wouldn’t even occur to me to assume that the post is saying that every believer of a harmful ideology is like that. More that there’s a fanaticism attractor and that people at the center of it really do approach that described extreme—which I think is true? I guess that’s the thing you’re contesting, but I feel like history contains some examples of atrocities that are hard to explain without choosing at least one of the following: either some people can get incredibly fanatical, or some people are sadistic/evil and may use fanaticism/ideology as a cover. Either way, at least one of the posts’ messages must be true?
The rest of the post then tries to argue that these kinds of mental traits are relatively widespread, and argues from that incorrigibility and absolute fanaticism.
I agree an “archetype” definition could be fine. But it’s IMO clearly not what’s going on. The post makes no attempt at clarifying how far of an outlier all of the above descriptions are, and later includes sections like:
In the US, around 20% of American adults (roughly 50 million) agree that “God has called Christians to exercise dominion over all areas of American society” (2023 PRRI/Brookings survey, p.4). Similarly, nearly a quarter of US adults (Pew Research Center, 2022) say the Bible should have “a great deal of influence” on US laws. Extrapolating data from a2008-2009 Pew survey (p.47) of 19 African countries, we estimate that roughly 15% of Africa’s 700 million Christians (roughly 100 million) believe that violence against civilians in defense of Christianity can often or sometimes be justified. Christians in Europe and Latin America may plausibly be less fanatical on average. Still, perhaps 200-250 million Christians worldwide (8-10%) could reasonably be classified as ideological fanatics.
This is obviously absurd! 200-250 million Christians absolutely do not fit the description that I quoted in my comment above, which is the only definition they give for what they mean by “ideological fanatics”:
For the fanatic, any doubt or deviation from these dogmas is not only wrong but evil, culminating in a total “soldier mindset” which defends the pre-existing ideology at all costs (Galef, 2021). This necessitates abandoning even the most basic form of empiricism by “rejecting the evidence of one’s own eyes and ears”, to paraphrase Orwell.[5] The fanatic is thus essentially incorrigible and has no epistemic or moral uncertainty, even in the face of widespread opposition (Gollwitzer et al., 2022).[6]
This is obviously not how this works. This is such a blatant example of category gerrymandering and the noncentral fallacy that I think my upset is very justified.
But ideological fanatics take such traits to extremes.
[...]
For the fanatic, any doubt or deviation from these dogmas is not only wrong but evil, culminating in a total “soldier mindset” which defends the pre-existing ideology at all costs (Galef, 2021). This necessitates abandoning even the most basic form of empiricism by “rejecting the evidence of one’s own eyes and ears”, to paraphrase Orwell.[5] The fanatic is thus essentially incorrigible and has no epistemic or moral uncertainty, even in the face of widespread opposition (Gollwitzer et al., 2022).[6]
This is the definition of ideological fanatic as far as I can tell! You are telling me 200-250 million Christians worldwide fit this description?
This is the definition of ideological fanatic as far as I can tell! You are telling me 200-250 million Christians worldwide fit this description?
I agree that’s too high, not on that strict description of fanaticism. But David also writes “For brevity, we focus here on support for ideological violence as the best proxy for ideological fanaticism.” So you may be right that there’s a bit of motte-and-bailey going on with who gets counted as “fanatical”. But I think the post is clear about what it is or isn’t saying.
And just to be clear, I think the true numbers would probably be somewhat shocking even for the strict/extreme definition. Like maybe a quarter of those 200-250 million, in my estimate. Africa still has witch burnings, some places in the US are very religious and almost every family there has this one (extended-)family member who is really fanatical, and I was mostly thinking about US big cities just now but in rural populations it’s probably even more pronounced.
But I think the post is clear about what it is or isn’t saying.
I mean, I do think the post is clear, and the post is saying “tens of millions of people in the world consider any doubt or deviation from these dogmas as not only wrong but evil, culminating in a total “soldier mindset” which defends the pre-existing ideology at all costs. This necessitates abandoning even the most basic form of empiricism by “rejecting the evidence of one’s own eyes and ears”, to paraphrase Orwell. These people are thus essentially incorrigible and have no epistemic or moral uncertainty, even in the face of widespread opposition”
I agree the post is clear on that, but it also extremely unlikely to me.
And you seem to agree! The right number here is not 50-75 million people! Are you telling me that 50-75 million people have “abandoned even the most basic form of empiricism”, “defending their pre-existing ideology at all costs”?
This just doesn’t match any historical conflicts or ideological social movements. Yes, group dynamics can drive people into doing weird and aggressive things, but these are descriptions of what individual people would do, and that a substantial fraction of them would be incapable of reform. In contrast to that, in the moment those group dynamics end, the vast majority of people caught up in these things turn out to be normal and well-adjusted, with the above being a terrible description.
almost every family there has this one family member who is really fanatical
I don’t think I’ve ever met someone who fits the description above. One of the key lines is that “they are incorrigible and have no epistemic or moral uncertainty, even in the face of widespread opposition”. But that doesn’t match what usually goes on here at all. The vast majority of people behaving fanatically would stop doing so when facing widespread opposition, and this is certainly true of the family members I’ve seen here. People are religious fundamentalists because the people around them are. If the people around them stop and start pushing back, a very very small fraction of people would end up insisting for the rest of their lives on their previous beliefs.
The language generally does sound a little strong, but I’d guess it to be directionally correct and that your points wouldn’t significantly change the post’s conclusions (though I admit I’ve only skimmed the post). Like, it’s true that a lot of people will change their minds if the social context changes, but if the ideology manages to maintain a stable-enough social context or one that shifts adaptively enough, then those people’s attitudes can stay quite resilient.
And even if huge numbers of people did change their minds, it’s possible for some not to, e.g. because their psychology for one reason or another ends up leaving them no line of retreat, so that anything ends up being less painful than changing one’s mind. The post also notes that it may be enough for a pretty small number of people to be fanatics, if those people end up in control of a state.
Generally I think that Duncan’s heuristic of betting on existence is a pretty good one that’s generally correct, and also applies for the case of “extreme fanatics do exist”.
I don’t know how I would differentiate paragraphs like the above from ravings of religious people talking about demons or devils.
It seems like in these posts all negative attributes must be assigned to these people, and all their cognitive leanings must be absolute and unwielding.
This seems like a strawman? The post was assigning some negative attributes to these people, not all of them. For one, extreme tribalism implies a loyalty to your own tribe, which is generally seen as a positive virtue. (Stereotypical demons don’t even have that quality.)
How to differentiate this from talk about demons or devils—well, most obviously, demons and devils are supernatural and incompatible with any naturalistic understanding of the world. Some variables within the brain getting stuck in an extreme setting is not. (I do find it a plausible claim that e.g. a superintelligence capable of arbitrarily manipulating such a person’s environment could always find some way of getting the person to change their mind, but I think the post is most reasonably read as “incapable of changing their minds for most practical purposes”.)
The language generally does sound a little strong, but I’d guess it to be directionally correct and that your points wouldn’t significantly change the post’s conclusions (though I admit I’ve only skimmed the post).
[...]
Generally I think that Duncan’s heuristic of betting on existence is a pretty good one that’s generally correct, and also applies for the case of “extreme fanatics do exist”.
I am not arguing against “there are literally no people who could meaningfully be described as having these attributes”, but the post goes on to classify over 500 million people worldwide as having these attributes!
My guess is you would agree with me here if you read the article beyond skimming.
In conclusion, accounting for potential overlap between categories, perhaps 500 million to 1 billion people, roughly 6-12% of the world population, may plausibly be classified as ideological fanatics
Again, their only definition of an “ideological fanatic” is that extremely extremely strong summary I linked above. There is no section of the post that’s like “of course, the vast vast majority of ideological fanatics do not think anything like this and are not well-described by this, and are largely behaving this way due to social momentum, and are maybe mildly on a spectrum in this direction”. It just creates an extremely intense boogeyman of “ideological fanatics” then classifies ~10% of the world population as matching that description.
their only definition of an “ideological fanatic” is that extremely extremely strong summary I linked above. There is no section of the post that’s like “of course, the vast vast majority of ideological fanatics do not think anything like this and are not well-described by this, and are largely behaving this way due to social momentum, and are maybe mildly on a spectrum in this direction”
I think these excerpts are saying something like that?
One overarching characteristic of the fanatical worldview is black-and-white thinking (good vs. evil, us vs. them) with no room for nuance. Let’s not make the same mistake. Like most phenomena, ideological fanaticism exists on a continuum. Furthest from fanaticism are those enlightened few who, following reason and evidence, act with benevolence towards all. A vast middle ground is occupied by religious traditionalists, hyper-partisan activists, conspiracy theorists, and many others. Indeed, a mild form of ideological fanaticism is arguably human nature: we are all somewhat prone to overconfidence, motivated reasoning, and tribalistic in-group favoritism and outgroup discrimination (e.g., Kunda, 1990; Diehl, 1990; Hewstone et al., 2002). [...]
Ideological fanaticism is not just a single sliding scale. Rather, it is multidimensional, that is, people can exhibit different levels of each fanatical triad component. The most dangerous form of ideological fanaticism requires elevated levels of all three characteristics. A hypothetical ‘Bayesian Nazi’, for instance, would lack absolute certainty and thus remain open to changing his mind. Similarly, without Manichean hatred, there is no motivation for mass harm, and without a willingness to use violence, even the most hateful beliefs remain inert.
Nor are fanatical movements monolithic.[13] While their leaders often were malignant narcissists, their followers are frequently ordinary people desperately seeking meaning and certainty in a chaotic, disappointing world (Hoffer, 1951; Kruglanski et al., 2014; Tietjen, 2023). Not all are true believers, either: some merely conform to group pressure, others are cynical opportunists, and many fall somewhere in between.[14] Many fanatics are capable of eventual reform, so we should not demonize them as irredeemably evil.
I agree these talk about potential scales, but it seems to me that they describe “ideological fanatics” as a pretty extreme point on that scale, and then classify at least hundreds of millions of people as falling on that side of the scale?
Not all are true believers, either: some merely conform to group pressure, others are cynical opportunists, and many fall somewhere in between.[14] Many fanatics are capable of eventual reform, so we should not demonize them as irredeemably evil.
These are not the right quantitative adjectives here. The correct ones to use would be:
Practically no people we classify as “ideological fanatics” are true believers
Virtually all (of the 500M+) fanatics are capable of eventual reform
Like, I don’t see how you could use the original quantifiers (“not all” and “many”), and after multiplying them through with the quantitative estimates not arrive at numbers at least 3 OOMs too high for these traits.
but it seems to me that they describe “ideological fanatics” as a pretty extreme point on that scale
I agree that the original text is ambiguous in this regard and that there are reasonable grounds for your reading. Personally I interpret sentences like
ideological fanaticism exists on a continuum
The most dangerous form of ideological fanaticism requires elevated levels of all three characteristics
their followers are frequently ordinary people
to mean something like “we are giving descriptions for the most extreme version of ideological fanaticism as that’s the easiest to gesture toward, but will also include people with less extreme versions when trying to estimate the sizes of the movements”.
I think it’s also relevant that the section on dogmatic certainty that you quoted beings with
The most ardent fanatics
implying that not all fanatics are this extreme; and of course the section on dogmatic certainty was one of the three sub-dimensions for ideological fanaticism rather than the overall definition.
But I think “which one of these readings is more correct” gets pretty subjective and impossible to resolve, so for me the more important test is something like… “if the authors could be read as making either a strong claim or a weaker one, how much do their conclusions depend on the stronger claim?”.
And it seems to me that on an interpretation where they only mean something like “virtually all fanatics are capable of eventual reform in principle, but in practice may be stuck in an environment where that is very unlikely”… then the various dangers that they outline, like “Ideological fanaticism increases the risk of war and conflict” or “Fanatical retributivism may lead to astronomical suffering”, still sound plausible.
Of course, it’s still fair to criticize the authors for e.g. being unclear about this or for implying a stronger claim when a weaker claim would suffice, but I wouldn’t strong-downvote them for that.
With regard to the bit about the 200-250 million Christians, that section does contain this paragraph
How many ideological fanatics are out there? Formulating a precise estimate is nearly impossible, as fanaticism exists on a multidimensional continuum with no clear demarcations, and because good data is sparse. Therefore, the numbers below are merely rough approximations based on limited research. For brevity, we focus here on support for ideological violence as the best proxy for ideological fanaticism. Endorsing ideological violence usually presupposes dogmatism and tribalistic hatred, since one needs to confidently believe the hated target group is deserving of punishment in order to justify violence. Another limitation is that we mostly rely on survey data[45], not actual behavior; this may overestimate fanaticism (if claimed support for violence is mere “cheap talk”) or underestimate it (“social desirability bias”).
I think a reasonable reading of this section/paragraph is also something like “for purposes of this section, we are defining an ideological fanatic as having some combination of these three traits that’s high enough for them to endorse ideological violence” [not implying that they would necessarily all be maxed out on the “dogmatic certainty” dimension].
Here we could apply a similar test of… “if we interpret the 200-250 million Christians narrowly as only being people who endorse ideological violence, rather than assuming that they’re necessarily incapable of changing their minds, does this still e.g. increase the risk of war and conflict?”. I think the answer is pretty clearly yes.
I have little understanding of Putin’s personality. This is why I thought SBF was a better example, because we have more detailed understanding of him—for instance, we have his diary entries! He says it’s likely that all his emotions and empathy are fake! I understand that Putin has probably committed heinous acts, but I am not as aware of evidence he is biologically as impaired emotionally/psychologically as SBF is.
there’s still a huge difference between the ones where your main concern is only large-scale fraud, vs the ones where the concern is unnecessary wars, genocide, and torture camps
Mm, I am not convinced that if SBF wasn’t made the leader of Russia he would not do much more evil than Putin. My impression is that SBF is morally shameless, more competent, and quite unusually ideologically committed to “the ends justify the means”.
Actually, I chatted with an LLM for a bit, and I changed my mind, I no longer think SBF is an especially good/central example of a psychopath. (link to my chat with chatgpt)
“He says it’s likely that all his emotions and empathy are fake!”
I don’t get why you think that’s such a big deal. A lot of people are like that. My guess is something like 5%. A lot of people who are like that don’t admit it. Surely dictators are like 50% likely to be like that just on priors, and then you can add KGB history for Putin.
I feel like you’re overupdating based on SBF admitting something, while not inferring things about Putin (or other dictators) based on past behavior and based on the demands of their role (and getting there).
I mean I understand not knowing much about Putin specifically; if I’m honest, I also don’t know much/couldn’t give you detailed examples, but I’m actually somewhat familiar with KGB history due to an interest in Cold War spy stories, and it’s been said that Putin was an exemplary KGB specimen or whatever, so, we can probably infer with 99.9% confidence that he thinks “ends justify the means” too, because imagine being in the KGB and voicing deontological objections to your superiors, do you think you’re going to rise up through the ranks?
BTW my model of that sort of personality is that when someone says the things that SBF wrote about himself, it’s still compatible with them having genuine feelings of fondness (though somewhat faint rather than all-consuming) for a person (or animal) or two in their lives. And maybe that’s why habryka thinks these personality traits are overpainted/demonized. I even agree that some people might be too categorically negative about the idea that some people on the sociopathy/psychopathy spectrum may actually be alright at least if you’re able to contain some of their bad patterns (like lying). But for the most part, I’d say it still makes for bad leadership and stewardship of others when someone is like that even in the more benign expressions, and we haven’t even gotten into the topic of extreme sadism and tails generally yet, which by the looks of it (my comment here being disagree-voted and the lack of logic in “it’s a spectrum” arguments also pointed out by Steven Byrnes, and surrounding discussion there generally) some people here seem to be in denial about. I don’t understand what’s going on.
Edit: I looked into figures a bit and I think it’s more like 3% for all population, but 5% for men specifically feels like the right estimate to me. And this is assuming “blunted emotionality” rather than “literally has no emotions ever”.
and we haven’t even gotten into the topic of extreme sadism and tails generally yet, which by the looks of it (my comment here being disagree-voted and the lack of logic in “it’s a spectrum” arguments also pointed out by Steven Byrnes, and surrounding discussion there generally) some people here seem to be in denial about. I don’t understand what’s going on.
I disagree-voted it, mostly because my strong guess (based on having done that for a bunch of other crimes like this) is that the actual drivers of the crime you are talking here about won’t actually be well-characterized as the kind of sadism you are talking about. It would require digging into the details, and it didn’t seem worth it to me to do that, so just a disagreement-vote seemed most appropriate.
If you end up looking into it (e.g., you could talk to Claude starting with a prompt and our recent comments here) and change your mind (or not), please let me know. I suggest doing so on a day where you’re not necessarily planning to get a lot of work done, because reading about this stuff really weighs you down. Unfortunately the sadism component is on-the-nose.
I agree with you that it’s often the case that the media paints people as evil where other stuff is going on rather than just “evil personality full stop” (like the intense hatred towards mothers who harm their children when they suffer from extreme postnatal depression, or have mental problems that generate Munchausen by proxy expressions). But sometimes people really like torturing others for fun and there’s ample documentation of that sort of personality not just in the sextortion cases I alluded to, but throughout history when you read about places that used torture (not even just the victims saying that the torturers seemed to enjoy it, sometimes the torturers write about it themselves).
I wonder if maybe there’s a selection effect where the media kind of stops reporting on things that get too shocking, meaning where extreme sadism is involved, so if you just go by shocking media examples, it’s possible to miss the tails. But it’s different with history where historians often go to great lengths highlighting how bad the atrocities were in some times and places.
I think SBF is an example that would be hard to debate in a neutral way, due to the beef many people will have with him, and the stronger feelings people have as someone who was/is close to this community. I share some distaste for trying to use him as an example in this context.
Some people’s minds are plausibly shaped such that they would destroy the future this way — but my guess is this requires fanatical dedication to a belief system or vision, of the kind that isn’t compatible with actively being in power. People in power are often corrupt, but their highly competitive positions can’t afford much brokenness in the minds that occupy them. Those minds have to be largely intact to do the job, which screens off many of the worst outcomes.
And then certainly it doesn’t seem that hard to imagine people who CEV is zero-value, or in a cosmic sense close to it. It seems like there will be at least some people who end up not caring for the cosmos, who just want to live out their live in peace, and then die, in the way they were expecting to for most of their life. Hopefully such a person would end up giving over the cosmos so someone who does care, but if they don’t then that would qualify.
I am interested if you do think there are many people whose CEV would be either zero-value or net negative? My first thought is that people who are deeply psychologically impaired or broken might be like this (
e.g. sociopaths—perhaps SBF), but I am not sure.Edit: I no longer think that SBF is an especially central/useful pointer here.
@Zach Stein-Perlman Where’s your thumbs-down coming from? SBF is the psychopath that I personally have met a few times, that people know of, and who got a lot of power, so it seems like a useful example to mention. Also don’t forget his personal diary entries read like this:
Reading your first comment above, I reacted as follows:
”What is going on, surely Putin is on the sociopathy/psychopathy spectrum too and not as endearing as SBF with effective altruism as an autistic special interest, so how is this even a question/comparison?”
(With SBF’s CEV I’d admittedly be quite concerned about the greed/recklessness inherent to his philosophy of utilitarianism and risk-taking, and I actually think that sort of thing could backfire uniquely-badly with AIs optimizing everything. But other than that, personality-wise, I just want to flag that while it’s probably always net bad to have people high on the sociopathy/psychopathy spectrum in leadership positions, there’s still a huge difference between the ones where your main concern is only large-scale fraud, vs the ones where the concern is unnecessary wars, genocide, and torture camps.)
I wonder, have many people on here not read the many examples of how harmful certain types of personality can be, for instance in this post and this one?
I have strong-downvoted both of those posts! I think they do a mixture of the “taking propaganda at face value” thing, seem to largely be in some kind of negative affect spiral around imagining arbitrarily bad qualities they can assign to people, seem to take an enormously naive view of the psychopathology literature, seem to consistently avoid dealing with the moral reflection dynamics I outline in this post, and seem very rooted in negative utilitarianism, which I disagree with[1].
Overall, I am pretty sad about that whole set of stuff. I somewhat wish I could talk more to people about it, but the negative utilitarianism-leaning stuff usually makes good-faith discourse a lot harder (since the bottom-line for my interlocutors seems clearly written).
Like man, those posts are so frustrating to read. They just… assert things. No epistemic statuses, no caveats, just stuff like this:
It honestly reads to me kind of like true crime podcasts? They love always describing everyone they cover with these kinds of extreme characterizations which basically never hold up when you look into them. My honest best guess is these posts serve a kind of similar role within our community.
I am confident the psychological characterization in paragraphs like the above is wrong. “Defends pre-existing ideology at all costs”. Come on, no, of course not “all costs”. Indeed a huge fraction of people caught up in these ideologies end up deconverting or drastically mellowing their views when their social context changes. These seem hilariously strawmanny absolutes that get ascribed to people here.
This is exactly the kind of thing I mean when I say people seem really got by propaganda, and when people engage in the fundamental attribution error. Like, I don’t know how I would differentiate paragraphs like the above from ravings of religious people talking about demons or devils.
It seems like in these posts all negative attributes must be assigned to these people, and all their cognitive leanings must be absolute and unwielding. They must “defend the pre-existing ideology at all costs”, they are “utterly convinced”, they are “textbook dogmatists”, they place “absolute faith” in the holy texts. Come on, this is not how people work.
and furthermore I think is evidence that something pretty deep is going wrong in how they are thinking about things
Thanks for sharing that take, which I find largely quite bizarre and surprising. I continue to think those posts are super valuable.
I can understand finding the negative(-leaning) utilitarianism codedness of the writings annoying, but I don’t see why you think it makes good-faith discussion a lot harder. From my perspective, a lot of writings on LW are “yay, hurrah team human”-coded in a way that annoys the crap out of me and makes me want to punch things, but it’s not like that means I can’t get valuable things out of the writings or have to treat the posters here as necessarily adversarial.
The subject of the sentence you put in quotation marks was “The fanatic”—as in, “the archetypal example of the fanatic.” This sounds to me like more of a writing style issue. Like, it wouldn’t even occur to me to assume that the post is saying that every believer of a harmful ideology is like that. More that there’s a fanaticism attractor and that people at the center of it really do approach that described extreme—which I think is true? I guess that’s the thing you’re contesting, but I feel like history contains some examples of atrocities that are hard to explain without choosing at least one of the following: either some people can get incredibly fanatical, or some people are sadistic/evil and may use fanaticism/ideology as a cover. Either way, at least one of the posts’ messages must be true?
There are some far-future relevance speculations where my model of what is likely to happen with AI is different from David’s, so I think it’s just much more likely that AIs will take over and it won’t matter what the humans who built it were like. But that doesn’t really invalidate too much—I mean part of the posts are also about what sort of qualities we wouldn’t want to see in AIs that we build. On your point about how AI-aided moral reflection and overcoming resource scarcity would reduce fanaticism or other bad consequences from “bad values,” I think that sort of point is underappreciated in some EA circles, but it’s not like it’s obvious that this is what’s going to happen. I think the posts we’re discussing engage well with reasons why reflection might not solve all the problems.
The rest of the post then tries to argue that these kinds of mental traits are relatively widespread, and argues from that incorrigibility and absolute fanaticism.
I agree an “archetype” definition could be fine. But it’s IMO clearly not what’s going on. The post makes no attempt at clarifying how far of an outlier all of the above descriptions are, and later includes sections like:
This is obviously absurd! 200-250 million Christians absolutely do not fit the description that I quoted in my comment above, which is the only definition they give for what they mean by “ideological fanatics”:
This is obviously not how this works. This is such a blatant example of category gerrymandering and the noncentral fallacy that I think my upset is very justified.
This is the definition of ideological fanatic as far as I can tell! You are telling me 200-250 million Christians worldwide fit this description?
I agree that’s too high, not on that strict description of fanaticism. But David also writes “For brevity, we focus here on support for ideological violence as the best proxy for ideological fanaticism.” So you may be right that there’s a bit of motte-and-bailey going on with who gets counted as “fanatical”. But I think the post is clear about what it is or isn’t saying.
And just to be clear, I think the true numbers would probably be somewhat shocking even for the strict/extreme definition. Like maybe a quarter of those 200-250 million, in my estimate. Africa still has witch burnings, some places in the US are very religious and almost every family there has this one (extended-)family member who is really fanatical, and I was mostly thinking about US big cities just now but in rural populations it’s probably even more pronounced.
I mean, I do think the post is clear, and the post is saying “tens of millions of people in the world consider any doubt or deviation from these dogmas as not only wrong but evil, culminating in a total “soldier mindset” which defends the pre-existing ideology at all costs. This necessitates abandoning even the most basic form of empiricism by “rejecting the evidence of one’s own eyes and ears”, to paraphrase Orwell. These people are thus essentially incorrigible and have no epistemic or moral uncertainty, even in the face of widespread opposition”
I agree the post is clear on that, but it also extremely unlikely to me.
And you seem to agree! The right number here is not 50-75 million people! Are you telling me that 50-75 million people have “abandoned even the most basic form of empiricism”, “defending their pre-existing ideology at all costs”?
This just doesn’t match any historical conflicts or ideological social movements. Yes, group dynamics can drive people into doing weird and aggressive things, but these are descriptions of what individual people would do, and that a substantial fraction of them would be incapable of reform. In contrast to that, in the moment those group dynamics end, the vast majority of people caught up in these things turn out to be normal and well-adjusted, with the above being a terrible description.
I don’t think I’ve ever met someone who fits the description above. One of the key lines is that “they are incorrigible and have no epistemic or moral uncertainty, even in the face of widespread opposition”. But that doesn’t match what usually goes on here at all. The vast majority of people behaving fanatically would stop doing so when facing widespread opposition, and this is certainly true of the family members I’ve seen here. People are religious fundamentalists because the people around them are. If the people around them stop and start pushing back, a very very small fraction of people would end up insisting for the rest of their lives on their previous beliefs.
I have met Christians like that and I don’t see why those numbers would be too high. We live in extreme filter bubbles.
The language generally does sound a little strong, but I’d guess it to be directionally correct and that your points wouldn’t significantly change the post’s conclusions (though I admit I’ve only skimmed the post). Like, it’s true that a lot of people will change their minds if the social context changes, but if the ideology manages to maintain a stable-enough social context or one that shifts adaptively enough, then those people’s attitudes can stay quite resilient.
And even if huge numbers of people did change their minds, it’s possible for some not to, e.g. because their psychology for one reason or another ends up leaving them no line of retreat, so that anything ends up being less painful than changing one’s mind. The post also notes that it may be enough for a pretty small number of people to be fanatics, if those people end up in control of a state.
Generally I think that Duncan’s heuristic of betting on existence is a pretty good one that’s generally correct, and also applies for the case of “extreme fanatics do exist”.
This seems like a strawman? The post was assigning some negative attributes to these people, not all of them. For one, extreme tribalism implies a loyalty to your own tribe, which is generally seen as a positive virtue. (Stereotypical demons don’t even have that quality.)
How to differentiate this from talk about demons or devils—well, most obviously, demons and devils are supernatural and incompatible with any naturalistic understanding of the world. Some variables within the brain getting stuck in an extreme setting is not. (I do find it a plausible claim that e.g. a superintelligence capable of arbitrarily manipulating such a person’s environment could always find some way of getting the person to change their mind, but I think the post is most reasonably read as “incapable of changing their minds for most practical purposes”.)
[...]
I am not arguing against “there are literally no people who could meaningfully be described as having these attributes”, but the post goes on to classify over 500 million people worldwide as having these attributes!
My guess is you would agree with me here if you read the article beyond skimming.
Again, their only definition of an “ideological fanatic” is that extremely extremely strong summary I linked above. There is no section of the post that’s like “of course, the vast vast majority of ideological fanatics do not think anything like this and are not well-described by this, and are largely behaving this way due to social momentum, and are maybe mildly on a spectrum in this direction”. It just creates an extremely intense boogeyman of “ideological fanatics” then classifies ~10% of the world population as matching that description.
I think these excerpts are saying something like that?
I agree these talk about potential scales, but it seems to me that they describe “ideological fanatics” as a pretty extreme point on that scale, and then classify at least hundreds of millions of people as falling on that side of the scale?
These are not the right quantitative adjectives here. The correct ones to use would be:
Practically no people we classify as “ideological fanatics” are true believers
Virtually all (of the 500M+) fanatics are capable of eventual reform
Like, I don’t see how you could use the original quantifiers (“not all” and “many”), and after multiplying them through with the quantitative estimates not arrive at numbers at least 3 OOMs too high for these traits.
I agree that the original text is ambiguous in this regard and that there are reasonable grounds for your reading. Personally I interpret sentences like
to mean something like “we are giving descriptions for the most extreme version of ideological fanaticism as that’s the easiest to gesture toward, but will also include people with less extreme versions when trying to estimate the sizes of the movements”.
I think it’s also relevant that the section on dogmatic certainty that you quoted beings with
implying that not all fanatics are this extreme; and of course the section on dogmatic certainty was one of the three sub-dimensions for ideological fanaticism rather than the overall definition.
But I think “which one of these readings is more correct” gets pretty subjective and impossible to resolve, so for me the more important test is something like… “if the authors could be read as making either a strong claim or a weaker one, how much do their conclusions depend on the stronger claim?”.
And it seems to me that on an interpretation where they only mean something like “virtually all fanatics are capable of eventual reform in principle, but in practice may be stuck in an environment where that is very unlikely”… then the various dangers that they outline, like “Ideological fanaticism increases the risk of war and conflict” or “Fanatical retributivism may lead to astronomical suffering”, still sound plausible.
Of course, it’s still fair to criticize the authors for e.g. being unclear about this or for implying a stronger claim when a weaker claim would suffice, but I wouldn’t strong-downvote them for that.
With regard to the bit about the 200-250 million Christians, that section does contain this paragraph
I think a reasonable reading of this section/paragraph is also something like “for purposes of this section, we are defining an ideological fanatic as having some combination of these three traits that’s high enough for them to endorse ideological violence” [not implying that they would necessarily all be maxed out on the “dogmatic certainty” dimension].
Here we could apply a similar test of… “if we interpret the 200-250 million Christians narrowly as only being people who endorse ideological violence, rather than assuming that they’re necessarily incapable of changing their minds, does this still e.g. increase the risk of war and conflict?”. I think the answer is pretty clearly yes.
I have little understanding of Putin’s personality. This is why I thought SBF was a better example, because we have more detailed understanding of him—for instance, we have his diary entries! He says it’s likely that all his emotions and empathy are fake! I understand that Putin has probably committed heinous acts, but I am not as aware of evidence he is biologically as impaired emotionally/psychologically as SBF is.
Mm, I am not convinced that if SBF wasn’t made the leader of Russia he would not do much more evil than Putin. My impression is that SBF is morally shameless, more competent, and quite unusually ideologically committed to “the ends justify the means”.
Actually, I chatted with an LLM for a bit, and I changed my mind, I no longer think SBF is an especially good/central example of a psychopath. (link to my chat with chatgpt)
“He says it’s likely that all his emotions and empathy are fake!”
I don’t get why you think that’s such a big deal. A lot of people are like that. My guess is something like 5%. A lot of people who are like that don’t admit it. Surely dictators are like 50% likely to be like that just on priors, and then you can add KGB history for Putin.
I feel like you’re overupdating based on SBF admitting something, while not inferring things about Putin (or other dictators) based on past behavior and based on the demands of their role (and getting there).
I mean I understand not knowing much about Putin specifically; if I’m honest, I also don’t know much/couldn’t give you detailed examples, but I’m actually somewhat familiar with KGB history due to an interest in Cold War spy stories, and it’s been said that Putin was an exemplary KGB specimen or whatever, so, we can probably infer with 99.9% confidence that he thinks “ends justify the means” too, because imagine being in the KGB and voicing deontological objections to your superiors, do you think you’re going to rise up through the ranks?
BTW my model of that sort of personality is that when someone says the things that SBF wrote about himself, it’s still compatible with them having genuine feelings of fondness (though somewhat faint rather than all-consuming) for a person (or animal) or two in their lives. And maybe that’s why habryka thinks these personality traits are overpainted/demonized. I even agree that some people might be too categorically negative about the idea that some people on the sociopathy/psychopathy spectrum may actually be alright at least if you’re able to contain some of their bad patterns (like lying). But for the most part, I’d say it still makes for bad leadership and stewardship of others when someone is like that even in the more benign expressions, and we haven’t even gotten into the topic of extreme sadism and tails generally yet, which by the looks of it (my comment here being disagree-voted and the lack of logic in “it’s a spectrum” arguments also pointed out by Steven Byrnes, and surrounding discussion there generally) some people here seem to be in denial about. I don’t understand what’s going on.
Edit: I looked into figures a bit and I think it’s more like 3% for all population, but 5% for men specifically feels like the right estimate to me. And this is assuming “blunted emotionality” rather than “literally has no emotions ever”.
I disagree-voted it, mostly because my strong guess (based on having done that for a bunch of other crimes like this) is that the actual drivers of the crime you are talking here about won’t actually be well-characterized as the kind of sadism you are talking about. It would require digging into the details, and it didn’t seem worth it to me to do that, so just a disagreement-vote seemed most appropriate.
If you end up looking into it (e.g., you could talk to Claude starting with a prompt and our recent comments here) and change your mind (or not), please let me know. I suggest doing so on a day where you’re not necessarily planning to get a lot of work done, because reading about this stuff really weighs you down. Unfortunately the sadism component is on-the-nose.
I agree with you that it’s often the case that the media paints people as evil where other stuff is going on rather than just “evil personality full stop” (like the intense hatred towards mothers who harm their children when they suffer from extreme postnatal depression, or have mental problems that generate Munchausen by proxy expressions). But sometimes people really like torturing others for fun and there’s ample documentation of that sort of personality not just in the sextortion cases I alluded to, but throughout history when you read about places that used torture (not even just the victims saying that the torturers seemed to enjoy it, sometimes the torturers write about it themselves).
I wonder if maybe there’s a selection effect where the media kind of stops reporting on things that get too shocking, meaning where extreme sadism is involved, so if you just go by shocking media examples, it’s possible to miss the tails. But it’s different with history where historians often go to great lengths highlighting how bad the atrocities were in some times and places.
I think SBF is an example that would be hard to debate in a neutral way, due to the beef many people will have with him, and the stronger feelings people have as someone who was/is close to this community. I share some distaste for trying to use him as an example in this context.
Ok, fair enough, I can see that it would overall be easier for the discourse to stick to examples that are less local.
I talk a bit about this in this section:
And then certainly it doesn’t seem that hard to imagine people who CEV is zero-value, or in a cosmic sense close to it. It seems like there will be at least some people who end up not caring for the cosmos, who just want to live out their live in peace, and then die, in the way they were expecting to for most of their life. Hopefully such a person would end up giving over the cosmos so someone who does care, but if they don’t then that would qualify.