The intended audience for this piece deserves to be stated more explicitly. For whom does the (let’s say universal) need to be the hero lead to adoption of successionist ideology? The small minority of people who both work in AI and accept there is associated x-risk have cognitive dissonance to resolve. What about average people? Not spelling this out risks leaving the impression that susceptibility to the meme is a more widespread problem than it really is.
YonatanK
Karma: 108
Many practice and endorse ethical heuristics against the censure of speech on any topic, especially any salient and politically relevant topic, lest such censure mess with our love of truth, or our ability to locate good policy options via the free and full exchange of ideas, or our freedom/autonomy/self-respect broadly.
I don’t think this is actually true.
Even among rationalists I believe there are red lines for ideas that cannot be raised without censure and disgust. I won’t attempt to draw them. The fact that among rationalists these lines lie other than where many people would draw them, including on the topic of racial difference, is not accepted as evidence of a commitment to open-mindedness that overrides other ethical commitments but just as a lack of commitment to those specific principles, with the commitment to open-mindedness as thin cover. Tetlock’s ideas around sacred values, which can’t be easily traded off, may be useful here. It’s not that those willing to discuss racial differences don’t have sacred values, it’s just that non-racism isn’t one of them.
Regarding the clash between the prudence heuristic of “don’t do something that has a 10% chance of killing all people” and other heuristics such as “don’t impede progress,” we have to consider the credibility problem in the assertion of risk by experts, when many of the same experts continue to work on A(G)I (and are making fortunes doing so). The statements about the risk say one thing but the actions say another, so we can’t conclude that anyone is actually trading off, in a real sense, against the prudence heuristic. This relates to my previous comment: “don’t kill all humans” seems like a sacred value, and so statements suggesting one is making the trade-off are not credible. From this “revealed belief” perspective, a statement “I believe there is a 10% chance that AI will kill all people” by an AI expert still working toward AGI is a false statement, and the only way to increase belief in the prediction is for AI experts to stop working on AI (at which point stopping the suicidal hold-outs becomes much easier). Conversely, amplifying the predictions of risk by leaders in the AI industry is a great way to confound the advocacy of the conscientious objectors.
It’s true that Acemoglu generally avoids dealing with extreme AI capabilities, and maybe should be more explicit about what is in and out of scope when he talks about AI. But the criticism I would lay at his feet is that it seems like a lot of what he does is explain to people who’ve been “dumbed” by learning economics how economics gets it all wrong, but without acknowledging that the intuitive, never-took-econ perspective doesn’t need his corrections. Sort of a “man on the inside,” when it’s not clear whether it’s worth the effort. A better example I’d suggest to represent your argument is Korinek and Suh’s “Scenarios for Transition to AGI,” which (as the title says) considers AGI, arrives at scenarios where, for instance, wages collapse, but completely ignores how this might upset economic models’ unstated assumptions like wages needing to remain above subsistence level to avoid complete social breakdown.
Nassim Nicholas Taleb, who loves taking down economists, is always worth a read as it is more generalizable than just AI stuff. The generalization is “human behavior is (correctly) tuned to avoiding being wiped out by power asymmetries and the unexpected, not maximizing expected returns under friendly conditions where a nation-state is there to save you from devastating losses.”
Thanks, Seth. What troubles me at the meta-level is the assumption of exclusive privilege implied by rationalist/utilitarian arguments, that of getting to make choices between extreme outcomes. It’s not just “I, as a rationalist, have considered the trade-offs between X and Y and, if forced to, will choose X.” It’s “I, a rationalist, believe that rationalism is superior to heuristic-based and otherwise inconsistent reasoning, and therefore assume the responsibility of making choices on behalf of those inferior reasoners.” There’s not much further to go to get to “I will conceal the ‘mild s-risk’ of the deaths of billions from them to get them to ally with me to avoid the x-risks that I am concerned about (but to which they, in their imperfect reasoning, are relatively indifferent).”
Thanks for this.
A minor comment and a major one:
-
The nits: the section on the the Israeli military’s use of AI against Hamas could use some tightening to avoid getting bogged down in the particularities of the Palestine situation. The line “some of the surveillance tactics Israeli settlers tested in Palestine” (my emphasis) to me suggests the interpretation that all Israelis are “settlers,” which is not the conventional use of that term. The conventional use of settlers applied only to those Israelis living over the Green Line, and particularly those doing so with the ideological intent of expanding Israel’s de facto borders. Similarly but separately, the discussion about Microsoft’s response to me seemed to take as facts what I believe to still only be allegations.
-
The major comment: I feel you could go farther to connect the dots between the “enshittification” of Anthropic and the issues you raise about the potential of AI to help enshittify democratic regimes. The idea that there are “exogenously” good and bad guys, with the former being trustworthy to develop A(G)I and the latter being the ones “we” want to stop from winning the race, is really central to AI discourse. You’ve pointed out the pattern in which participating in the race turns the “good” guys into bad guys (or at least untrustworthy ones).
-
I think this is the right response to the piece, but begs a more explicit challenge of the conclusion that underdog bias is maladaptive (@Garrett Baker offers both pre-modern tribal life and modern international relations as spheres in which this behavior is sensible).
One ought to be careful of the “anti-bias bias” leading one to accept evolutionary explanations for biases but then makes up reasons why they’re maladaptive to fit the (speculative) narrative that the world can be perfected by increasing the prevalence of objectively true beliefs.
Thank you. I certainly don’t want wrong interpretations to be distracting, so I’ve added to the transition from the introduction of “The Affair” to the discussion of the public discourse on Trumpism, to make the analogy more clear.
If I may, if it’s still not clear I’d ask for an attempt at an interpretation that doesn’t immediately contradict the whole argument, and if you think you’ve found it, paraphrase it and suggest how it could be more clearly presented. If you then want to rebut the argument, you’ll have something coherent to work against.
I have just written a full post inspired by this comment.
But being equally against both requires a positive program to prevent Option 1 other than the default of halting technological development that can lead to it (and thus taking Option 2, or a delay in immortality because human research is slower)! Conversely, without committing to finding such a program, pursuing the avoidance of Option 2 is an implicit acceptance of Option 1. Are you committing to this search? And if it fails, which option will you choose?
Well, it doesn’t sound like I misunderstood you so far, but just so I’m clear, are you not also saying that people ought to favor being annihilated by a small number of people controlling an aligned (to them) AGI that also grants them immortality over dying naturally with no immortality-granting AGI ever being developed? Perhaps even that this is an obviously correct position?
Can you speak to the difficulties of addressing risks from development in the national defense sector, which tends to be secret and therefore exposes us to the streetlight problem?
For this to be true, parliamentarians would have to be like ducklings who are impressed by whoever gets to them first or perversely run away from protecting The Greater Good merely because it is The Greater Good. That level of cynicism goes far beyond what falls out of the standard Interest Group model (e.g. Mancur Olsen’s). By that model, given that ControlAI represents a committed interest group, there is no reason to believe they can’t win.
First let me say that with respect to the world of alignment research, or the AI world in general, I am nothing. I don’t have a job in those areas, I am physically remote from where the action is. My contribution consists of posts and comments here.
This assertion deserves a lot of attention IMO, worthy of a post on its own, something along the lines of Why Rationalists Aren’t Winners (meant not to mock, but to put it in terms of what rationalism is supposed to do). The gist is that morality is useful for mass coordination to solve collective action problems. When you participate in deliberation about what is good for the group, help arrive at shared answers to the question “how ought we to behave?” and then commit to following those answers, that is power and effectiveness. Overcoming biases that help with coordination so you can, what, win at poker, is not winning. Nassim Nicholas Taleb covers this quite well.
Thanks for working through your thinking. And thanks for bringing my/our attention to “The Peace War,” I was not aware of it until now. My only caveat is that one must discount the verisimilitude of science fiction because it demands conflict to be interesting to read. It creates oppressive conditions for the protagonists to overcome, when rational antagonists would eschew those oppressive conditions so that there’s no need to protect themselves from plucky protagonists.
The same kind of reasoning applies to the bringing about of AI overlords if you don’t have to. @Mars_Will_Be_Ours covers this well in their comment.
The egoist/nihilist categories aren’t mutually exclusive. “For the environment” is not nihilistic nor non-egoist when the environment is the provider of everything you need to live a good, free, peaceful albeit finite life.
In that case, AI risk becomes similar to aging risk – it will kill me and my friends and relatives. The only difference is the value of future generations.
The casualness with which you throw out this comment seems to validate my assertion that “AI risk” and “risk of a misaligned AI destroying humanity” have become nearly conflated because of what, from the outside, appears like an incidental idiosyncrasy, longtermism, that initially attracted people to the study of AI alignment.
Part of the asymmetry that I’m trying to get acknowledgement of is subjective (or, if you prefer, due to differing utility functions). For most people “aging risk” is not even a thing but “I, my friends, and relatives all being killed” very much is. This is not a philosophical argument, it’s a fact about fundamental values. And fundamental differences in values, especially between large majorities and empowered minorities, are a very big deal.
I’m struggling to find the meat in this post. The idea that winning a fight for control can actually mean losing, because one’s leadership proves worse for the group than if one’s rival had won strikes me as one of the most basic properties of politics. The fact that the questions “Who would be better for national security”? vs “who will ensure I, and not my neighbor, will get more of the pie?” are quite distinct is something anyone who has ever voted in a national election ought to have considered. You state that “most power contests are not like this” (i.e. about shared outcomes) but that’s just plainly wrong, it should be obvious to anyone existing in a human group that “what’s good for the group” (including who should get what, to incentivize defense of, or other productive contributions to, the group) is usually the crux, otherwise there would be no point in political debate. So what am I missing?
Ironically, you then blithely state that AI risk is a special case where power politics ARE purely about “us” all being in the same boat, completely ignoring the concern that some accelerationists really might eventually try to run away with the whole game (I have been beating the drum about asymmetric AI risk for some time, so this is personally frustrating). Even if these concerns are secondary to wholly shared risk, it seems weird to (incorrectly) describe “most power politics” as being about purely asymmetric outcomes and then not account for them at all in your treatment of AI risk.
The way you stated this makes it seem like your conclusion for the reason why the Democrats lost (and by extension, what they need to do to avoid losing in the future) is obviously correct. But the Median Voter Theorem you invoked is a conditional statement, and I don’t think it’s at all obvious that its conditions held for the 2024 US presidential election.
What I find lacking is any depth to the retrospection. Hanania’s willingness to update his position clears a rather low bar. To go further one has to search for systematic causes for the error. For example, being wrong with the markets seems like a good opportunity to examine the hidden incentives that cause the market to get it wrong, not shrug and say “who could have known?”
I feel the question misstates the natsec framing by jumping to the later stages of AGI and ASI. This is important because it leads to a misunderstanding of the rhetoric that convinces normal non-futurists, who aren’t spending their days thinking about superintelligence.
The American natsec framing is about an effort to preserve the status quo in which the US is the hegemon. It is a conservative appeal with global reach, which works because Pax Americana has been relatively peaceful and prosperous. Anything that threatens American dominance, including giving ground in the AI race, appears dangerously destabilizing. Any risks from AI acceleration are literally after-thoughts (a problem for tomorrow, not today).Absurd as it is, the Trumpist effort to burn the American-led system of global cooperation to the ground is still branded as a conservative return to an imagined glorious past.
The challenge in defeating this conservative natsec framing lies in communicating that radical change is all that is on the menu, but with some options far worse than others. I, for one, currently believe the fatal effect pre-AGI AI will have on democracy and other liberal values, regardless of who wields it, to be a promising rhetorical avenue that should be amplified.
Richard, reading this piece with consideration of other pieces you’ve written/delivered about Effective Altruism, such as your Lessons from my time in Effective Altruism and your recent provocative talk at EA Global Boston lead me to wonder what it is (if anything) that leads you to self-identify as an Effective Altruist? There may not be any explicit EA shibboleth, but it seems to me to nevertheless entail a set of particular methods, and once you have moved beyond enough of them it may not make any sense to call oneself an Effective Altruist.
My mental model of EA has it intentionally operating at the margins, in the same way that arbitrageurs do, maximizing returns (specifically social ones) to a small number of actors willing to act not just in contrary fashion to but largely in isolation from the larger population. Once we recognize the wisdom of faith it seems to me we are moving back into the normie fold, and in that integration or synthesis the EA exclusivity habit may be a hindrance.
Any updates to this 2025 position for 2026?