For this to be true, parliamentarians would have to be like ducklings who are impressed by whoever gets to them first or perversely run away from protecting The Greater Good merely because it is The Greater Good. That level of cynicism goes far beyond what falls out of the standard Interest Group model (e.g. Mancur Olsen’s). By that model, given that ControlAI represents a committed interest group, there is no reason to believe they can’t win.
YonatanK
Karma: 93
First let me say that with respect to the world of alignment research, or the AI world in general, I am nothing. I don’t have a job in those areas, I am physically remote from where the action is. My contribution consists of posts and comments here.
This assertion deserves a lot of attention IMO, worthy of a post on its own, something along the lines of Why Rationalists Aren’t Winners (meant not to mock, but to put it in terms of what rationalism is supposed to do). The gist is that morality is useful for mass coordination to solve collective action problems. When you participate in deliberation about what is good for the group, help arrive at shared answers to the question “how ought we to behave?” and then commit to following those answers, that is power and effectiveness. Overcoming biases that help with coordination so you can, what, win at poker, is not winning. Nassim Nicholas Taleb covers this quite well.
Thanks for working through your thinking. And thanks for bringing my/our attention to “The Peace War,” I was not aware of it until now. My only caveat is that one must discount the verisimilitude of science fiction because it demands conflict to be interesting to read. It creates oppressive conditions for the protagonists to overcome, when rational antagonists would eschew those oppressive conditions so that there’s no need to protect themselves from plucky protagonists.
The same kind of reasoning applies to the bringing about of AI overlords if you don’t have to. @Mars_Will_Be_Ours covers this well in their comment.
The egoist/nihilist categories aren’t mutually exclusive. “For the environment” is not nihilistic nor non-egoist when the environment is the provider of everything you need to live a good, free, peaceful albeit finite life.
In that case, AI risk becomes similar to aging risk – it will kill me and my friends and relatives. The only difference is the value of future generations.
The casualness with which you throw out this comment seems to validate my assertion that “AI risk” and “risk of a misaligned AI destroying humanity” have become nearly conflated because of what, from the outside, appears like an incidental idiosyncrasy, longtermism, that initially attracted people to the study of AI alignment.
Part of the asymmetry that I’m trying to get acknowledgement of is subjective (or, if you prefer, due to differing utility functions). For most people “aging risk” is not even a thing but “I, my friends, and relatives all being killed” very much is. This is not a philosophical argument, it’s a fact about fundamental values. And fundamental differences in values, especially between large majorities and empowered minorities, are a very big deal.
$500 bounty for engagement on asymmetric AI risk
I’m struggling to find the meat in this post. The idea that winning a fight for control can actually mean losing, because one’s leadership proves worse for the group than if one’s rival had won strikes me as one of the most basic properties of politics. The fact that the questions “Who would be better for national security”? vs “who will ensure I, and not my neighbor, will get more of the pie?” are quite distinct is something anyone who has ever voted in a national election ought to have considered. You state that “most power contests are not like this” (i.e. about shared outcomes) but that’s just plainly wrong, it should be obvious to anyone existing in a human group that “what’s good for the group” (including who should get what, to incentivize defense of, or other productive contributions to, the group) is usually the crux, otherwise there would be no point in political debate. So what am I missing?
Ironically, you then blithely state that AI risk is a special case where power politics ARE purely about “us” all being in the same boat, completely ignoring the concern that some accelerationists really might eventually try to run away with the whole game (I have been beating the drum about asymmetric AI risk for some time, so this is personally frustrating). Even if these concerns are secondary to wholly shared risk, it seems weird to (incorrectly) describe “most power politics” as being about purely asymmetric outcomes and then not account for them at all in your treatment of AI risk.
The way you stated this makes it seem like your conclusion for the reason why the Democrats lost (and by extension, what they need to do to avoid losing in the future) is obviously correct. But the Median Voter Theorem you invoked is a conditional statement, and I don’t think it’s at all obvious that its conditions held for the 2024 US presidential election.
What I find lacking is any depth to the retrospection. Hanania’s willingness to update his position clears a rather low bar. To go further one has to search for systematic causes for the error. For example, being wrong with the markets seems like a good opportunity to examine the hidden incentives that cause the market to get it wrong, not shrug and say “who could have known?”
I feel the question misstates the natsec framing by jumping to the later stages of AGI and ASI. This is important because it leads to a misunderstanding of the rhetoric that convinces normal non-futurists, who aren’t spending their days thinking about superintelligence.
The American natsec framing is about an effort to preserve the status quo in which the US is the hegemon. It is a conservative appeal with global reach, which works because Pax Americana has been relatively peaceful and prosperous. Anything that threatens American dominance, including giving ground in the AI race, appears dangerously destabilizing. Any risks from AI acceleration are literally after-thoughts (a problem for tomorrow, not today).Absurd as it is, the Trumpist effort to burn the American-led system of global cooperation to the ground is still branded as a conservative return to an imagined glorious past.
The challenge in defeating this conservative natsec framing lies in communicating that radical change is all that is on the menu, but with some options far worse than others. I, for one, currently believe the fatal effect pre-AGI AI will have on democracy and other liberal values, regardless of who wields it, to be a promising rhetorical avenue that should be amplified.
Richard, reading this piece with consideration of other pieces you’ve written/delivered about Effective Altruism, such as your Lessons from my time in Effective Altruism and your recent provocative talk at EA Global Boston lead me to wonder what it is (if anything) that leads you to self-identify as an Effective Altruist? There may not be any explicit EA shibboleth, but it seems to me to nevertheless entail a set of particular methods, and once you have moved beyond enough of them it may not make any sense to call oneself an Effective Altruist.
My mental model of EA has it intentionally operating at the margins, in the same way that arbitrageurs do, maximizing returns (specifically social ones) to a small number of actors willing to act not just in contrary fashion to but largely in isolation from the larger population. Once we recognize the wisdom of faith it seems to me we are moving back into the normie fold, and in that integration or synthesis the EA exclusivity habit may be a hindrance.
A relevant, very recent opinion piece that has been syndicated around the country, explaining the universal value of faith:
https://www.latimes.com/opinion/story/2025-03-29/it-is-not-faith-that-divides-us
There’s a gap in the Three Mile Island/Chernobyl/Fukushima analogy, because those disasters were all in the peaceful uses of nuclear power. I’m not saying that they didn’t also impact the nuclear arms race, only that, for completeness, the arms race dynamics have to be considered as well.
YonatanK’s Shortform
There are at least 3 levers of social influence, and I suspect that we undervalue the 3rd one for getting anything done, especially when it comes to AI safety. They are 1) government policy 2) the actions of firms 3) low-coordination behaviors of individuals influenced by norms. There is a subclass of #3 that is having its day in the sun, the behaviors of employees of the US federal government, a.k.a. the “Deep State.” If their behaviors didn’t matter there wouldn’t be a perceived need by the Trump administration to purge them en masse (and replace with loyalists, or not). But if government employees’ low-coordination individual choices matter, then so can the choices of members of the general population.
States and firms are modern instruments, whereas (at least if you trust some of the accounts from Dawn of Everything) for about 100,000 years the more organic form of coordination was all humans had, and it worked surprisingly well (for example, people could and did travel long distances and count on receiving shelter from strangers).
As already stated, we rely on norm-observance in government employees performing their duties, and in everyone else to more or less comply with laws in functioning welfare states, but traditional norm enforcement is weakened by liberal laissez fair values. But if one believes (a la Suleyman’s The Coming Wave) that AI undermines the liberal welfare state, which is likely to be captured by powerful AI firms, then one shouldn’t discount norm-enforced resistance emerging to fill the void for an increasingly disenfranchised population.It is therefore a mistake to treat the race dynamics of AI development between firms and nation-states as an inevitable force pointing in only one direction. Given a critical mass of people recognizing that AI is bad for them, low-coordination resistance is possible, despite the absence of democratic policy-making.
On the flip side, this also suggests a tipping point where AI economic disruption becomes extremely violent, between powerful government-capturing firms wishing to maintain control and general populations resisting. Thus we should consider the existence of a hidden race between would-be powerful government-capturing firms and a would-be resistant population.
That summary doesn’t sound to me to be in the neighborhood of the intended argument. I would be grateful if you pointed to passages that suggest that reading so that I can correct them (DM me if that’s preferable).
Where I see a big disconnect is your conclusion that “AI will have an incentive to do X.” The incentives that the essay discusses are human incentives, not those of a hypothetical artificial agent.
Populectomy.ai
The subject of your post is the recurring patterns of avoidance you’re observing, without mentioning the impact on those more receptive and willing to engage. Nevertheless, I figure you’d still appreciate examples of the latter:
A link to the GD website was sent to me by a relatively distance acquaintance. This is a person of not-insignificant seniority at one of the FAANGs, whose current job is now hitched to AI. They have no specific idea about my own thoughts about AI risk, so my inference is that they send it to anyone they deem sufficiently wonky. The tone of the message in which they sent the link was “holy smokes, we are totally screwed,” suggesting strong buy-in to the paper’s argument and that it had an emotional impact.
There are two kinds of beliefs, those that can be affirmed individually (true independently of what others do) and those that depend on others acting as if they believe the same thing. They are, in other words, agreements. One should be careful not to conflate the two.
What you describe as “neutrality” to me seems to be a particular way of framing institutional forbearance and similar terms of cooperation in the face of the possibility of unrestrained competition and mutual destruction. When agreements collapse, it is not because these terms were unworkable (except for in the trivial sense that, well, they weren’t invulnerable to gaming and do on) but because cooperation between humans can always break down.
@AnthonyC I may be mistaken, but I took @M. Y. Zuo to be offering a reductio ad absurdum response to your comment about not being indifferent between the two ways of dying. The ‘which is a worse way to die’ debate doesn’t respond to what I wrote. I said
With respect to the survival prospects for the average human, this [whether or not the dying occurs by AGI] seems to me to be a minor detail.
I did not say that no one should care about the difference.
But the two risks are not in competition, they are complementary. If your concern about misalignment is based on caring about the continuation of the human species, and you don’t actually care how many humans other humans would kill in a successful alignment(-as-defined-here) scenario, a credible humans-kill-most-humans risk is still really helpful to your cause, because you can ally yourself with the many rational humans who don’t want to be killed either way to prevent both outcomes by killing AI in its cradle.
You have a later response to some clarifying comments from me, so this may be moot, but I want to call out that my emphasis is on the behavior of human agents who are empowered by automation that may fall well short of AGI. A “pivotal act” is a very germane idea, but rather than the pivotal act of the first AGI eliminating would-be AGI competitors, this act is carried out by humans taking out their human rivals.
It is pivotal because once the target population size has been achieved, competition ends, and further development of the AI technology can be halted as unnecessarily risky.
Can you speak to the difficulties of addressing risks from development in the national defense sector, which tends to be secret and therefore exposes us to the streetlight problem?