Someone who is interested in learning and doing good.
My Twitter: https://twitter.com/MatthewJBar
My Substack: https://matthewbarnett.substack.com/
Matthew Barnett
Karma: 10,730
My views on AI have indeed changed over time, on a variety of empirical and normative questions, but I think you’re inferring larger changes than are warranted from that comment in isolation.
Here’s a comment from 2023 where I said:
The term “AI takeover” is ambiguous. It conjures an image of a violent AI revolution, but the literal meaning of the term also applies to benign scenarios in which AIs get legal rights and get hired to run our society fair and square. A peaceful AI takeover would be good, IMO.
In fact, I still largely agree with the comment you quoted. The described scenario remains my best guess for how things could go wrong with AI. However, I chose my words poorly in that comment. Specifically, I was not clear enough about what I meant by “disempowerment.”
I should have distinguished between two different types of human disempowerment. The first type is violent disempowerment, where AIs take power by force. I consider this morally bad. The second type is peaceful or voluntary disempowerment, where humans willingly transfer power to AIs through legal and economic processes. I think this second type will likely be morally good, or at least morally neutral.
My moral objection to “AI takeover”, both now and back then, applies primarily to scenarios where AIs suddenly seize power through unlawful or violent means, against the wishes of human society. I have, and had, far fewer objections to scenarios where AIs gradually gain power by obtaining legal rights and engaging in voluntary trade and cooperation with humans.
The second type of scenario is what I hope I am working to enable, not the first. My reasoning for accelerating AI development is straightforward: accelerating AI will produce medical breakthroughs that could save billions of lives. It will also accelerate dramatic economic and technological progress that will improve quality of life for people everywhere. These benefits justify pushing forward with AI development.
I do not think violent disempowerment scenarios are impossible, just unlikely. And I think that pausing AI development would not meaningfully reduce the probability of such scenarios occurring. Even if pausing AI did reduce this risk, I think the probability of violent disempowerment is low enough that accepting this risk is justified by the billions of lives that faster AI development could save.
She asks why the book doesn’t spend more time explaining why an intelligence explosion is likely to occur. The answer is the book is explicitly arguing a conditional, what happens if it does occur, and acknowledges that it may or may not occur, or occur on any given time frame.
Is it your claim here that the book is arguing the conditional: “If there’s an intelligence explosion, then everyone dies?” If so, then it seems completely valid to counterargue: “Well, an intelligence explosion is unlikely to occur, so who cares?”
I expect there are no claims to the effect that there will be only one chance to correctly align the first AGI.
For the purpose of my argument, there is no essential distinction between ‘the first AGI’ and ‘the first ASI’. My main point is to dispute the idea that there will be a special ‘it’ at all, which we need to align on our first and only try. I am rejecting the scenario where a single AI system suddenly takes over the world. Instead, I expect AI systems will continuously and gradually assume more control over the world over time. In my view, there will not be one decisive system, but rather a continuous process of AIs assuming greater control over time.
To understand the distinction I am making, consider the analogy of genetically engineering humans. By assumption, if the tech continues improving, there will eventually be a point where genetically engineered humans will be superhuman in all relevant respects compared to ordinary biological humans. They will be smarter, stronger, healthier, and more capable in every measurable way. Nonetheless, there is no special point at which we develop ‘the superhuman’. There is no singular ‘it’ to build, which then proceeds to take over the world in one swift action. Instead, genetically engineered humans would simply progressively get smarter, more capable, and more powerful across time as the technology improves. At each stage of technological innovation, these enhanced humans would gradually take over more responsibilities, command greater power in corporations and governments, and accumulate a greater share of global wealth. The transition would be continuous rather than discontinuous.
Yes, at some point such enhanced humans will possess the raw capability to take control over the world through force. They could theoretically coordinate to launch a sudden coup against existing institutions and seize power all at once. But the default scenario seems more likely: a continuous transition from ordinary human control over the world to superhuman genetically engineered control over the world. They would gradually occupy positions of power through normal economic and political processes rather than through sudden conquest.
You’re saying that slow growth on multiple systems means we can get one of them right, by course correcting.
That’s not what I’m saying. My argument was not about multiple simultaneously existing systems growing slowly together. It was instead about how I dispute the idea of a unique or special point in time when we build “it” (i.e., the AI system that takes over the world), the value of course correction, and the role of continuous iteration.
As the review makes very clear, the argument isn’t about AGI, it’s about ASI. And yes, they argue that you would in fact only get one chance to align the system that takes over.
I’m aware; I was expressing my disagreement with their argument. My comment was not premised on whether we were talking about “the first AGI” or “the first ASI”. I was making a more fundamental point.
In particular: I am precisely disputing the idea that there will be “only one chance to align the system that takes over”. In my view, the future course of AI development will not be well described as having a single “system that takes over”. Instead, I anticipate waves of AI deployment that gradually, and continuously assume more control.
I fundamentally dispute the entire framing of thinking about “the system” that we need to align on our “first try”. I think AI development is an ongoing process in which we can course correct. I am disputing that there is an important, unique point when we will build “it” (i.e. the ASI).
I would strongly disagree with the notion that FOOM is “a key plank” in the story for why AI is dangerous. Indeed, one of the most useful things that I, personally, got from the book, was seeing how it is *not* load bearing for the core arguments.
I think the primary reason why the foom hypothesis seems load-bearing for AI doom is that without a rapid AI and local takeoff, we won’t simply get “only one chance to correctly align the first AGI [ETA: or the first ASI]”.
If foom occurs, there will be a point where a company develops an AGI that quickly transitions from being just an experimental project to something capable of taking over the entire world. This presents a clear case for caution: if the AI project you’re working on will undergo explosive recursive self-improvement, then any alignment mistakes you build into it will become locked in forever. You cannot fix them after deployment because the AI will already have become too powerful to stop or modify.
However, without foom, we are more likely to see a gradual and diffuse transition from human control over the world to AI control over the world, without any single AI system playing a critical role in the transition by itself. The fact that the transition is not sudden is crucial because it means that no single AI release needs to be perfectly aligned before deployment. We can release imperfect systems, observe their failures, and fix problems in subsequent versions. Our experience with LLMs demonstrates this pattern, where we could fix errors after deployment, making sure future model releases don’t have the same problems (as illustrated by Sydney Bing, among other examples).
A gradual takeoff allows for iterative improvement through trial and error, and that’s simply really important. Without foom, there is no single critical moment where we must achieve near-perfect alignment without any opportunity to learn from real-world deployment. There won’t be a single, important moment where we abruptly transition from working on “aligning systems incapable of taking over the world” to “aligning systems capable of taking over the world”. Instead, systems will simply gradually and continuously get more powerful, with no bright lines.
Without foom, we can learn from experience and course-correct in response to real-world observations. My view is that this fundamental process of iteration, experimentation, and course correction in response to observed failures makes the problem of AI risk dramatically more tractable than it would be if foom were likely.
Roko says it’s impossible, I say it’s possible and likely.
I’m not sure Roko is arguing that it’s impossible for capitalist structures and reforms to make a lot of people worse off. That seems like a strawman to me. The usual argument here is that such reforms are typically net-positive: they create a lot more winners than losers. Your story here emphasizes the losers, but if the reforms were indeed net-positive, we could just as easily emphasize the winners who outnumber the losers.
In general, literally any policy that harms people in some way will look bad if you focus solely on the negatives, and ignore the positives.
I recognize that. But it seems kind of lame to respond to a critique of an analogy by simply falling back on another, separate analogy. (Though I’m not totally sure if that’s your intention here.)
I’m arguing that we won’t be fine. History doesn’t help with that, it’s littered with examples of societies that thought they would be fine. An example I always mention is enclosures in England, where the elite deliberately impoverished most of the country to enrich themselves.
Is the idea here that England didn’t do “fine” after enclosures? But in the century following the most aggressive legislative pushes towards enclosure (roughly 1760-1830), England led the industrial revolution, with large, durable increases in standards of living for the first time in world history—for all social classes, not just the elite. Enclosure likely played a major role in the increase in agricultural productivity in England, which created unprecedented food abundance in England.
It’s true that not everyone benefitted from these reforms, inequality increased, and a lot of people became worse off from enclosure (especially in the short-term, during the so-called Engels’ pause), but on the whole, I don’t see how your example demonstrates your point. If anything your example proves the opposite.
It would be helpful if you could clearly specify the “basic argumentation mistakes” you see in the original article. The parent comment mentioned two main points: (1) the claim that I’m being misleading by listing costs of an LVT without comparing them to benefits, and (2) the claim that an LVT would likely replace existing taxes rather than add to them.
If I’m wrong on point (2), that would likely stem from complex empirical issues, not from a basic argumentation mistake. So I’ll focus on point (1) here.
Regarding (1), my article explicitly stated that its purpose was not to offer a balanced evaluation, but to highlight potential costs of an LVT that I believe are often overlooked or poorly understood. This is unlike the analogy given, where someone reviews a car only by noting its price while ignoring its features. The disanalogy is that, in the case of a car review, the price is already transparent and easily verifiable. However, with an LVT, the costs are often unclear, downplayed, or poorly communicated in public discourse, at least in my experience on twitter.
By pointing out these underdiscussed costs, I’m aiming to provide readers with information they may not have encountered, helping them make a more informed overall judgment. Moreover, I explicitly and prominently linked to a positive case for an LVT and encouraged readers to compare both perspectives to reach a final conclusion.
A better analogy would be a Reddit post warning that although a car is advertised at $30,000, the true cost is closer to $60,000 after hidden fees are included. That post would still add value, even if it doesn’t review the car in full, because it would provide readers valuable information they might not be familiar with. Likewise, my article aimed to contribute by highlighting costs of an LVT that might otherwise go unnoticed.
Doesn’t the revealed preference argument also imply people don’t care much about dying from aging? (This is invested in even less than catastrophic risk mitigation and people don’t take interventions that would prolong their lives considerably.) I agree revealed preferences imply people care little about the long run future of humanity, but they do imply caring much more about children living full lives than old people avoiding aging.
I agree that the amount of funding explicitly designated for anti-aging research is very low, which suggests society doesn’t prioritize curing aging as a social goal. However, I think your overall conclusion is significantly overstated. A very large fraction of conventional medical research specifically targets health and lifespan improvements for older people, even though it isn’t labeled explicitly as “anti-aging.”
Biologically, aging isn’t a single condition but rather the cumulative result of multiple factors and accumulated damage over time. For example, anti-smoking campaigns were essentially efforts to slow aging by reducing damage to smokers’ bodies—particularly their lungs—even though these campaigns were presented primarily as life-saving measures rather than “anti-aging” initiatives. Similarly, society invests a substantial amount of time and resources in mitigating biological damage caused by air pollution and obesity.
Considering this broader understanding of aging, it seems exaggerated to claim that people aren’t very concerned about deaths from old age. I think public concern depends heavily on how the issue is framed. My prediction is that if effective anti-aging therapies became available and proven successful, most people would eagerly purchase them for high sums, and there would be widespread political support to subsidize those technologies.
Right now explicit support for anti-aging research is indeed politically very limited, but that’s partly because robust anti-aging technologies haven’t been clearly demonstrated yet. Medical technologies that have proven effective at slowing aging (even if not labeled as such) have generally been marketed as conventional medical technologies and typically enjoy widespread political support and funding.
I agree that delaying a pure existential risk that has no potential upside—such as postponing the impact of an asteroid that would otherwise destroy complex life on Earth—would be beneficial. However, the risk posed by AI is fundamentally different from something like an asteroid strike because AI is not just a potential threat: it also carries immense upside potential to improve and save lives. Specifically, advanced AI could dramatically accelerate the pace of scientific and technological progress, including breakthroughs in medicine. I expect this kind of progress would likely extend human lifespans and greatly enhance our quality of life.
Therefore, if we delay the development of AI, we are likely also delaying these life-extending medical advances. As a result, people who are currently alive might die of aging-related causes before these benefits become available. This is a real and immediate issue that affects those we care about today. For instance, if you have elderly relatives whom you love and want to see live longer, healthier lives, then—assuming all else is equal—it makes sense to want rapid medical progress to occur sooner rather than later.
This is not to say that we should accelerate AI recklessly and do it even if that would dramatically increase existential risk. I am just responding to your objection, which was premised on the idea that delaying AI could be worth it even if delaying AI doesn’t reduce x-risk at all.
It sounds like you’re talking about multi-decade pauses and imagining that people agree such a pause would only slightly reduce existential risk. But, I think a well timed safety motivated 5 year pause/slowdown (or shorter) is doable and could easily cut risk by a huge amount.
I suspect our core disagreement here primarily stems from differing factual assumptions. Specifically, I doubt that delaying AI development—even if timed well and if the delay were long in duration—would meaningfully reduce existential risk beyond a tiny amount. However, I acknowledge I haven’t said much to justify this claim here. Given this differing factual assumption, pausing AI development seems somewhat difficult to justify from a common-sense moral perspective, and very difficult to justify from a worldview that puts primary importance on people who currently exist.
My guess is that the “common sense” values tradeoff is more like 0.1% than 1% because of people caring more about kids and humanity having a future than defeating aging.
I suspect the common-sense view is closer to 1% than 0.1%, though this partly depends on how we define “common sense” in this context. Personally, I tend to look to revealed preferences as indicators of what people genuinely value. Consider how much individuals typically spend on healthcare and how much society invests in medical research relative to explicit existential risk mitigation efforts. There’s an enormous gap, suggesting society greatly values immediate survival and the well-being of currently living people, and places relatively lower emphasis on abstract, long-term considerations about species survival as a concern separate from presently existing individuals.
Politically, existential risk receives negligible attention compared to conventional concerns impacting currently-existing people. If society placed as much importance on the distant future as you’re suggesting, the US government would likely have much lower debt, and national savings rates would probably be higher. Moreover, if individuals deeply valued the flourishing of humanity independently of the flourishing of current individuals, we probably wouldn’t observe such sharp declines in birth rates globally.
None of these pieces of evidence alone are foolproof indicators that society doesn’t care that much about existential risk, but combined, they paint a picture of our society that’s significantly more short-term focused, and substantially more person-affecting than you’re suggesting here.
I care deeply about many, many people besides just myself (in fact I care about basically everyone on Earth), and it’s simply not realistic to expect that I can convince all of them to sign up for cryonics. That limitation alone makes it clear that focusing solely on cryonics is inadequate if I want to save their lives. I’d much rather support both the acceleration of general technological progress through AI, and cryonics in particular, rather than placing all hope in just one of those approaches.
Furthermore, curing aging would be far superior to merely making cryonics work. The process of aging—growing old, getting sick, and dying—is deeply unpleasant and degrading, even if one assumes a future where cryonic preservation and revival succeed. Avoiding that suffering entirely is vastly more desirable than having to endure it in the first place. Merely signing everyone up for cryonics would be insufficient to address this suffering, whereas I think AI could accelerate medicine and other technologies to greatly enhance human well-being.
The value difference commenters keep pointing out needs to be far bigger than they represent it to be, in order for it to justify increasing existential risk in exchange for some other gain.
I disagree with this assertion. Aging poses a direct, large-scale threat to the lives of billions of people in the coming decades. It doesn’t seem unreasonable to me to suggest that literally saving billions of lives is worth pursuing even if doing so increases existential risk by a tiny amount [ETA: though to be clear, I agree it would appear much more unreasonable if the reduction in existential risk were expected to be very large]. Loosely speaking, this idea only seems unreasonable to those who believe that existential risk is overwhelmingly more important than every other concern by many OOMs—so much so that it renders all other priorities essentially irrelevant. But that’s a fairly unusual and arguably extreme worldview, not an obvious truth.
I am essentially a preference utilitarian and an illusionist regarding consciousness. This combination of views leads me to conclude that future AIs will very likely have moral value if they develop into complex agents capable of long-term planning, and are embedded within the real world. I think such AIs would have value even if their preferences look bizarre or meaningless to humans, as what matters to me is not the content of their preferences but rather the complexity and nature of their minds.
When deciding whether to attribute moral patienthood to something, my focus lies primarily on observable traits, cognitive sophistication, and most importantly, the presence of clear open-ended goal-directed behavior, rather than on speculative or less observable notions of AI welfare, about which I am more skeptical. As a rough approximation, my moral theory aligns fairly well with what is implicitly proposed by modern economists, who talk about revealed preferences and consumer welfare.
Like most preference utilitarians, I believe that value is ultimately subjective: loosely speaking, nothing has inherent value except insofar as it reflects a state of affairs that aligns with someone’s preferences. As a consequence, I am comfortable, at least in principle, with a wide variety of possible value systems and future outcomes. This means that I think a universe made of only paperclips could have value, but only if that’s what preference-having beings wanted the universe to be made out of.
To be clear, I also think existing people have value too, so this isn’t an argument for blind successionism. Also, it would be dishonest not to admit that I am also selfish to a significant degree (along with almost everyone else on Earth). What I have just described simply reflects my broad moral intuitions about what has value in our world from an impartial point of view, not a prescription that we should tile the universe with paperclips. Since humans and animals are currently the main preference-having beings in the world, at the moment I care most about fulfilling what they want the world to be like.
Note that unborn people are merely potential, as their existence depends on our choices. Future generations aren’t guaranteed—we decide whether or not they will exist, particularly those who might be born decades or centuries from now. This makes their moral status far less clear than someone who already exists or who is certain to exist at some point regardless of our choices.
Additionally, if we decide to account for the value of future beings, we might consider both potential human people and future AI entities capable of having moral value. From a utilitarian perspective, both human and AI welfare presumably matters. This makes the ethical calculus more complicated, as the dilemma isn’t merely about whether we risk losing all future generations, but rather whether we risk shifting posterity from humans to AIs.
Personally, I’m largely comfortable evaluating our actions primarily—though not entirely—based on their impact on current human lives, or at least people (and animals) who will exist in the near-term. I value our present generation. I want us to keep living and to thrive. It would be a tragedy if we either went extinct or died from aging. However, to the extent that I care about distant future generations, my concern is substrate-impartial, and I don’t particularly favor humans over AIs.
For me, I’m at ~10% P(doom). Whether I’d accept a proposed slowdown depends on how much I expect it decrease this number.[2]
How do you model this situation? (also curious on your numbers)
I put the probability that AI will directly cause humanity to go extinct within the next 30 years at roughly 4%. By contrast, over the next 10,000 years, my p(doom) is substantially higher, as humanity could vanish for many different possible reasons, and forecasting that far ahead is almost impossible. I think a pause in AI development matters most for reducing the near-term, direct AI-specific risk, since the far-future threats are broader, more systemic, harder to influence, and only incidentally involve AI as a byproduct of the fact that AIs will be deeply embedded in our world.
I’m very skeptical that a one-year pause would meaningfully reduce this 4% risk. This skepticism arises partly because I doubt much productive safety research would actually happen during such a pause. In my view, effective safety research depends heavily on an active feedback loop between technological development and broader real-world applications and integration, and pausing the technology would essentially interrupt this feedback loop. This intuition is also informed by my personal assessment of the contributions LW-style theoretical research has made toward making existing AI systems safe—which, as far as I can tell, has been almost negligible (though I’m not implying that all safety research is similarly ineffective or useless).
I’m also concerned about the type of governmental structures and centralization of power required to enforce such a pause. I think pausing AI would seriously risk creating a much less free and dynamic world. Even if we slightly reduce existential risks by establishing an international AI pause committee, we should still be concerned about the type of world we’re creating through such a course of action. Some AI pause proposals seem far too authoritarian or even totalitarian to me, providing another independent reason why I oppose pausing AI.
Additionally, I think that when AI is developed, it won’t merely accelerate life-extension technologies and save old people’s lives; it will likely also make our lives vastly richer and more interesting. I’m excited about that future, and I want the 8 billion humans alive today to have the opportunity to experience it. This consideration adds another important dimension beyond merely counting potential lives lost, again nudging me towards supporting acceleration.
Overall, the arguments in favor of pausing AI seem surprisingly weak to me, considering the huge potential upsides from AI development, my moral assessment of the costs and benefits, my low estimation of the direct risk from misaligned AI over the next 30 years, and my skepticism about how much pausing AI would genuinely reduce AI risks.
“But with AI risk, the stakes put most of us on the same side: we all benefit from a great future, and we all benefit from not being dead.”
I appreciate this thoughtful perspective, and I think it makes sense, in some respects, to say we’re all on the same “side”. Most people presumably want a good future and want to avoid catastrophe, even if we have different ideas on how to get there.
That said, as someone who falls on the accelerationist side of things, I’ve come to realize that my disagreements with others often come down to values and not just facts. For example, a common disagreement revolves around the question: How bad would it be if by slowing down AI, we delay life-saving medical technologies that otherwise would have saved our aging parents (along with billions of other people) from death? Our answer to this question isn’t just empirical: it also reflects our moral priorities. Even if we agreed on all the factual predictions, how we weigh this kind of moral loss would still greatly affect our policy views.
Another recurring question is how to evaluate the loss incurred by the risk of unaligned AI: how bad would it be exactly if AI was not aligned with humans? Would such an outcome just be a bad outcome for us, like how aging and disease are bad to the people who experience it, or would it represent a much deeper and more tragic loss of cosmic significance, comparable to the universe never being colonized?
For both of these questions, I tend to fall on the side that makes acceleration look like the more rational choice, which can help explain my self-identification in that direction.
So while factual disagreements do matter, I think it’s important to recognize that value differences can run just as deep. And those disagreements can unfortunately put us on fundamentally different sides, despite surface-level agreement on abstract goals like “not wanting everyone to die”.
Previous discussion of this post can be found here.
I’d consider it bad if AIs take actions that result in a large fraction of humans becoming completely destitute and dying as a result.
But I think such an outcome would be bad whether it’s caused by a human or an AI. The more important question, I think, is whether such an outcome is likely to occur if we grant AIs legal rights. The answer to this, I think, is no. I anticipate that AGI-driven automation will create so much economic abundance in the future that it will likely be very easy to provide for the material needs of all biological humans.
Generally I think biological humans will receive income through charitable donations, government welfare programs, in-kind support from family members, interest, dividends, by selling their assets, or by working human-specific service jobs where consumers intrinsically prefer hiring human labor (e.g., maybe childcare). Given vast prosperity, these income sources seem sufficient to provide most humans with an adequate, if not incredibly high, standard of living.