I flat out cannot understand why someone would think advanced AIs will remain relatively unpersuasive, other than to think that AI will not get much more capable than it already is, and that’s if we fully restrict the AI to using known standard persuasion techniques and rule out any wizardry.
I can speak for myself on this, maybe it extrapolates to others. I spent a whole year and a half in the “super persuasion makes no sense as a thing that can ever exist unless you mean hypnosis and drugs” position. Then a friend pointed out, blackmail, bribery, deception, and yes hypnotism and drugs all count. And I went, “Oh, I was thinking about just talking talking. Like normal talking but magically I agree with the AI. Wow, I had it totally wrong.” Felt really stupid for a while, got over it. The word “super persuasion” is accurate but tremendously self-defeating because it doesn’t register as including to normies all the things it includes to techies. I think normie, I got fooled by the word. I don’t think I’m alone.
I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.
Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.
But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.
There is something else happening here with broader implications. A large part of the Anglosphere cannot talk rationally about Intelligence because doing so is racist, fascist, and right wing.
It steps out of politics and into AI because if you believe there is no difference in mental capabilities between someone with Down syndrome and John Von Neumann, that the difference in their outcomes is only because of the systems within which they function oppressing the person with Downs, then you obviously cannot believe in ASI as being smarter than you yourself.
There is a powerful set of disincentives here that oppose rational thought on this topic. And they’re not just on one side of the aisle.
Americans are generally terrible at decoupling, but I have found my best success by decoupling the idea of intelligence and model capabilities. By talking about capabilities, and never using the word intelligence, I can sometimes get people to avoid slipping into a political mindset and stay with me in the technical realm. Ymmv.
I spent a whole year and a half in the “super persuasion makes no sense as a thing that can ever exist unless you mean hypnosis and drugs” position. Then a friend pointed out, blackmail, bribery, deception, and yes hypnotism and drugs all count. And I went, “Oh, I was thinking about just talking talking. Like normal talking but magically I agree with the AI. Wow, I had it totally wrong.” Felt really stupid for a while, got over it. The word “super persuasion” is accurate but tremendously self-defeating because it doesn’t register as including to normies all the things it includes to techies. I think normie, I got fooled by the word. I don’t think I’m alone.
As a believer in superpersuasion, that’s not what I think. I think there are very effective salespeople, politicians, cult leaders, etc. who have the “normal talking but magically I agree” effect some of the time, maybe any one of them doesn’t work on most people but most people are susceptible to someone, and if you just take the convex hull of human persuasion ability you get something superhuman that can do the “normal talking but magically I agree” thing most of the time.
Like, given the existence of very unusually effective human persuaders, I just don’t see how someone could reasonably believe that superpersuasion-by-talking isn’t possible, unless they’re doing the motte-and-bailey that Zvi calls out by thinking that it has to mean ‘being able to persuade literally anyone of literally anything, no exceptions’ (I have seen exactly this a few times).
(Scott Alexander had a good post making this point, with Hitler(?), Joseph Smith, and Muhammad as examples, that I can’t find.)
The taboo on ‘intelligence’ is not that people are just outraged by any attempts to talk about it; rather, there’s a general assumption that someone stating that one group of people is more intelligent than the other also believes that the former group is more deserving of moral consideration and political representation.
If I wanted to talk about differences in intelligence between races, I’d qualify it with the statement that all people deserve equal moral consideration and (upon reaching adulthood) representation. I also don’t want to talk about it, because I think all the evidence on it sucks and the people who talk about it rarely consider environmental contamination or socio-cultural factors, but I wouldn’t be offended if you asked me in that manner.
If someone disagrees with or omits that statement, then I’m going to assume they’ve been using motivated reasoning to affirm their previous biases, and they should be prepared to defend their sources beforehand. I am one of the people who make that assumption.
models also out persuade national championships debaters and professional canvassers, although the models regress when their ‘quantity of facts presented’ is limited to the human norms.
I can speak for myself on this, maybe it extrapolates to others. I spent a whole year and a half in the “super persuasion makes no sense as a thing that can ever exist unless you mean hypnosis and drugs” position. Then a friend pointed out, blackmail, bribery, deception, and yes hypnotism and drugs all count. And I went, “Oh, I was thinking about just talking talking. Like normal talking but magically I agree with the AI. Wow, I had it totally wrong.” Felt really stupid for a while, got over it. The word “super persuasion” is accurate but tremendously self-defeating because it doesn’t register as including to normies all the things it includes to techies. I think normie, I got fooled by the word. I don’t think I’m alone.
There is something else happening here with broader implications. A large part of the Anglosphere cannot talk rationally about Intelligence because doing so is racist, fascist, and right wing.
It steps out of politics and into AI because if you believe there is no difference in mental capabilities between someone with Down syndrome and John Von Neumann, that the difference in their outcomes is only because of the systems within which they function oppressing the person with Downs, then you obviously cannot believe in ASI as being smarter than you yourself.
There is a powerful set of disincentives here that oppose rational thought on this topic. And they’re not just on one side of the aisle.
Americans are generally terrible at decoupling, but I have found my best success by decoupling the idea of intelligence and model capabilities. By talking about capabilities, and never using the word intelligence, I can sometimes get people to avoid slipping into a political mindset and stay with me in the technical realm. Ymmv.
As a believer in superpersuasion, that’s not what I think. I think there are very effective salespeople, politicians, cult leaders, etc. who have the “normal talking but magically I agree” effect some of the time, maybe any one of them doesn’t work on most people but most people are susceptible to someone, and if you just take the convex hull of human persuasion ability you get something superhuman that can do the “normal talking but magically I agree” thing most of the time.
Like, given the existence of very unusually effective human persuaders, I just don’t see how someone could reasonably believe that superpersuasion-by-talking isn’t possible, unless they’re doing the motte-and-bailey that Zvi calls out by thinking that it has to mean ‘being able to persuade literally anyone of literally anything, no exceptions’ (I have seen exactly this a few times).
(Scott Alexander had a good post making this point, with Hitler(?), Joseph Smith, and Muhammad as examples, that I can’t find.)
The taboo on ‘intelligence’ is not that people are just outraged by any attempts to talk about it; rather, there’s a general assumption that someone stating that one group of people is more intelligent than the other also believes that the former group is more deserving of moral consideration and political representation.
If I wanted to talk about differences in intelligence between races, I’d qualify it with the statement that all people deserve equal moral consideration and (upon reaching adulthood) representation. I also don’t want to talk about it, because I think all the evidence on it sucks and the people who talk about it rarely consider environmental contamination or socio-cultural factors, but I wouldn’t be offended if you asked me in that manner.
If someone disagrees with or omits that statement, then I’m going to assume they’ve been using motivated reasoning to affirm their previous biases, and they should be prepared to defend their sources beforehand. I am one of the people who make that assumption.
except that we have actual evidence that this is the case …
When Large Language Models are More PersuasiveThan Incentivized Humans, and Why, 2025
https://arxiv.org/abs/2505.09662
LLMs are literally more persuasive than people paid to argue the point. that’s a superhuman ability
Doesn’t say how the human persuaders were selected, and if they weren’t top experts it doesn’t demonstrate superhuman ability.
models also out persuade national championships debaters and professional canvassers, although the models regress when their ‘quantity of facts presented’ is limited to the human norms.
It’s covered in the first dispatch here.
https://aistop.watch/p/shifting-perspectives