+4. In some regards it’s sad to have to rehash this argument, but I feel that this argument has been going around in the public discourse, and so it’s worthwhile to write up a thorough account of what’s naive about it and how to move past it. My sense is that it has become less prevalent since the essay; perhaps the essay helped.
Many folks have distaste for Eliezer’s style, or for perhaps implying that a weak-man argument is fully representative of positions he disagrees with; I think some of these criticisms are valid but do not mean the essay isn’t (a) pretty right in many places and well-written, nor (b) the best existing rebuttal to this perspective I’m aware of. If anyone has a better one to link to then please do and I’ll reduce my vote for this to +1!
any realistic villain should be constructed so that if a real-world version of the villain could read your dialogue for them, they would nod along and say, “Yes, that is how I would argue that.”
Is The Spokesperson a realistic villain?
Has this argument been going around?
In 2024, I wasn’t able to find anyone making this argument. My sense is that it was not at all prevalent, and continues to be not at all prevalent. By analogy, Bernie Bankman is OpenAI (or other AI lab) and The Spokesperson is OpenAI’s representatives. As far as I know, OpenAI were not making the argument in 2024 that OpenAI hasn’t killed everyone and therefore they won’t kill everyone in the future.
There could be a race of killer robots in the far future, but I don’t work on not turning AI evil today for the same reason I don’t worry about the problem of overpopulation on the planet Mars.
That was a defensible position in 2015. With the benefit of hindsight it doesn’t seem that “work on not turning AI evil” in 2015 was especially effective at altering our trajectory as a civilization, the main group who tried to do that work was MIRI, and while they argue it was worth doing the work, they admit that it didn’t pan out. Regardless, Andrew Ng is not making The Spokesperson’s argument, he specifically allows that killer robots could exist in the future, despite not existing in 2015. So I remain unaware of anyone making the argument with a straight face.
If you disagree, I encourage you to find an example (or two!) and update me.
Best existing rebuttal?
There are certainly many people who act in various circumstances as if there will never be any surprises, but without actually saying things like “there will never be any surprises”. So maybe we need a rebuttal to the blindness, rather than to the non-existent arguments.
Thinking about the best other rebuttals to such blindness, I think Nassim Taleb covers this well as “tail risk blindness”. Nassim Taleb is not to everyone’s taste, I know, but he’s a good writer on this topic. It may seem silly to talk about AI-caused extinction as a “tail risk” when many of us have a high P(Doom). However, on a day-to-day basis P(Doom) is low—I’m probably not going to go extinct today. This is the same scenario as companies taking financial tail risks—the chance of subprime collapsing today is low, the chance of it collapsing at some point in the next ten years is high.
Or really, anyone who uses the phrase “hypothetical risk” or “hypothetical threat” as a conversation-stopper when talking about ASI extinction, is implicitly invoking the intuitive idea that we should by default be deeply skeptical of things that we have not already seen with our own eyes.
Is The Spokesperson a realistic villain?
Obviously I agree that The Spokesperson is not going to sound realistic and sympathetic when he is arguing for “Ponzi Pyramid Incorporated” led by “Bernie Bankman”. It’s a reductio ad absurdum, showing that this style of argument proves too much. That’s the whole point.
Thank you for the concrete example of Unfalsifiable Stories of Doom from Barnett et al in November 2025 I think there are several important differences between the two arguments. To avoid taking up too much of our time, I’m going to dwell on one in particular.
Dismiss or engage with theoretical arguments?
The Spokesperson in Empiricism! is dismissive of the entire concept of predicting the future using “words words words and thinking”. Barnett et al are not. I think this is clearest in their engagement with IABIED’s claim that AIs steer in alien directions that only mostly coincide with helpfulness. Here’s the claim:
Modern AIs are pretty helpful (or at least not harmful) to most users, most of the time. But as we noted above, a critical question is how to distinguish an AI that deeply wants to be helpful and do the right thing, from an AI with weirder and more complex drives that happen to line up with helpfulness under typical conditions, but which would prefer other conditions and outcomes even more. … This long list of cases look just like what the “alien drives” theory predicts, in sharp contrast with the “it’s easy to make AIs nice” theory that labs are eager to put forward.
Their counterargument is:
Assume that the “alien drives” theory is true. Let’s operate this theory and make predictions.
When Barnett et al operate this theory and make predictions with it, they predict that when we elicit undesired AI behavior, it will mostly be alien undesired behavior.
When Barnett et al observe undesired AI behavior, it appears to them to be mostly humanlike undesired behavior. This includes the specific behaviors cited by IABIED.
This is evidence against the “alien drives” theory.
I think a good counter to this portion of Barnett et al is to disagree with steps 2&3.
2: Alien beings who talk to humans will talk in human language if they can, if they want to persuade, instruct, threaten, etc. So the model doesn’t predict completely alien behavior.
3: Some of the behavior we see is pretty alien. Eg, Spiritual Bliss, adversarial inputs, Waluigi Effect.
Whereas a lecture from The Empiricist about latent variables is not a good counter. Barnett et al agree that there are latent variables like “alien drives” vs “human drives”, and claim that the observed evidence is a better fit for the “human drives” theory.
+4. In some regards it’s sad to have to rehash this argument, but I feel that this argument has been going around in the public discourse, and so it’s worthwhile to write up a thorough account of what’s naive about it and how to move past it. My sense is that it has become less prevalent since the essay; perhaps the essay helped.
Many folks have distaste for Eliezer’s style, or for perhaps implying that a weak-man argument is fully representative of positions he disagrees with; I think some of these criticisms are valid but do not mean the essay isn’t (a) pretty right in many places and well-written, nor (b) the best existing rebuttal to this perspective I’m aware of. If anyone has a better one to link to then please do and I’ll reduce my vote for this to +1!
It is written that
Is The Spokesperson a realistic villain?
Has this argument been going around?
In 2024, I wasn’t able to find anyone making this argument. My sense is that it was not at all prevalent, and continues to be not at all prevalent. By analogy, Bernie Bankman is OpenAI (or other AI lab) and The Spokesperson is OpenAI’s representatives. As far as I know, OpenAI were not making the argument in 2024 that OpenAI hasn’t killed everyone and therefore they won’t kill everyone in the future.
Since 2024, AI has advanced substantially, so I asked Opus 4.5 for examples of people making this argument. It wasn’t aware of any. Its first concrete suggestion was Andrew Ng: Fearing a rise of killer robots is like worrying about overpopulation on Mars from 2015.
That was a defensible position in 2015. With the benefit of hindsight it doesn’t seem that “work on not turning AI evil” in 2015 was especially effective at altering our trajectory as a civilization, the main group who tried to do that work was MIRI, and while they argue it was worth doing the work, they admit that it didn’t pan out. Regardless, Andrew Ng is not making The Spokesperson’s argument, he specifically allows that killer robots could exist in the future, despite not existing in 2015. So I remain unaware of anyone making the argument with a straight face.
If you disagree, I encourage you to find an example (or two!) and update me.
Best existing rebuttal?
There are certainly many people who act in various circumstances as if there will never be any surprises, but without actually saying things like “there will never be any surprises”. So maybe we need a rebuttal to the blindness, rather than to the non-existent arguments.
Thinking about the best other rebuttals to such blindness, I think Nassim Taleb covers this well as “tail risk blindness”. Nassim Taleb is not to everyone’s taste, I know, but he’s a good writer on this topic. It may seem silly to talk about AI-caused extinction as a “tail risk” when many of us have a high P(Doom). However, on a day-to-day basis P(Doom) is low—I’m probably not going to go extinct today. This is the same scenario as companies taking financial tail risks—the chance of subprime collapsing today is low, the chance of it collapsing at some point in the next ten years is high.
Or in other words:
I feel like I see it pretty often. Check out “Unfalsifiable stories of doom”, for example.
Or really, anyone who uses the phrase “hypothetical risk” or “hypothetical threat” as a conversation-stopper when talking about ASI extinction, is implicitly invoking the intuitive idea that we should by default be deeply skeptical of things that we have not already seen with our own eyes.
Obviously I agree that The Spokesperson is not going to sound realistic and sympathetic when he is arguing for “Ponzi Pyramid Incorporated” led by “Bernie Bankman”. It’s a reductio ad absurdum, showing that this style of argument proves too much. That’s the whole point.
Thank you for the concrete example of Unfalsifiable Stories of Doom from Barnett et al in November 2025 I think there are several important differences between the two arguments. To avoid taking up too much of our time, I’m going to dwell on one in particular.
Dismiss or engage with theoretical arguments?
The Spokesperson in Empiricism! is dismissive of the entire concept of predicting the future using “words words words and thinking”. Barnett et al are not. I think this is clearest in their engagement with IABIED’s claim that AIs steer in alien directions that only mostly coincide with helpfulness. Here’s the claim:
Their counterargument is:
Assume that the “alien drives” theory is true. Let’s operate this theory and make predictions.
When Barnett et al operate this theory and make predictions with it, they predict that when we elicit undesired AI behavior, it will mostly be alien undesired behavior.
When Barnett et al observe undesired AI behavior, it appears to them to be mostly humanlike undesired behavior. This includes the specific behaviors cited by IABIED.
This is evidence against the “alien drives” theory.
I think a good counter to this portion of Barnett et al is to disagree with steps 2&3.
2: Alien beings who talk to humans will talk in human language if they can, if they want to persuade, instruct, threaten, etc. So the model doesn’t predict completely alien behavior.
3: Some of the behavior we see is pretty alien. Eg, Spiritual Bliss, adversarial inputs, Waluigi Effect.
Whereas a lecture from The Empiricist about latent variables is not a good counter. Barnett et al agree that there are latent variables like “alien drives” vs “human drives”, and claim that the observed evidence is a better fit for the “human drives” theory.