Nick Bostrom

Karma: 632

Nick Bostrom 11 Jun 2026 4:32 UTC
15 points
6
on: Thoughts on Claude Fable’s silent safeguards
Maybe they should have called it “Claude Mythos-early-limited”. Then people might be excited that they got access to this limited version already rather than having to wait even longer before getting any access to Mythos at all.

Nick Bostrom 18 Feb 2026 13:17 UTC
3 points
0
in reply to: charding’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
The go/no-go model is not meant to show that a P(doom) of up to 97% is “acceptable” (or at least it would risk being highly misleading to say that). The model is only meant to show that up to that level of risk, launching superintelligence increases life expectancy under the given assumptions. That model ignores many important factors (such as distributional considerations and diminishing marginal utility in QALYs), which is why a series of more complicated models are introduced that take into account some of these other factors. (Even the most elaborate of the models introduced is still only very schematic and leaves out much that is relevant, as all formal models of this sort do. “For these and other reasons, the preceding analysis—although it highlights several relevant considerations and tradeoffs—does not on its own imply support for any particular policy prescriptions.”).
By the way, there may also be reasons to regard implementing a lottery that would involve going out and killing some random subset of the human population differently from allowing technological progress to continue—even if we were to stipulate that the two cases were exactly parallel with respect to some set of consequentialist outcome metrics. (Also, while in your example, using randomization would equalize people’s chances or ex ante expected lifespans, it would lead to radically uneven ex post outcomes. Some people with egalitarian intuitions care about inequality of outcomes, not only inequality of chances or opportunities—especially in cases where the inequality of outcomes is not connected to personal motivations, efforts, or choices.)

Nick Bostrom 17 Feb 2026 13:45 UTC
1 point
0
in reply to: kman’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
The ’1400′ number was indeed picked somewhat arbitrarily and merely to illustrate the point that our life expectancy goes up even for quite levels of AI risk. I thought this point was worth making because I’ve seen the opposite claim made or insinuated by some anti-AI advocates.
Later models in the paper take into account more factors—in particular, temporal discounting and diminishing marginal utility in QALYs. In these more full-fledged models, it is not the case that we get “launch immediately” for all non-extreme settings of the parameters even if we postulate that post-superintelligence lifespans could be a billion years.
(In general, as I said in the preamble, I recommend not focussing overly on the exact numbers that pop out of these simple models. The formal models are mostly intended to serve as illustrations of various possible reasoning assumptions and the general patterns they imply.)

Nick Bostrom 15 Feb 2026 11:21 UTC
1 point
0
in reply to: DaystarEld’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
(I added a few remarks on this in my reply to quetzal_rainbow, although—sorry—nothing numerical.)

Nick Bostrom 15 Feb 2026 11:08 UTC
12 points
2
in reply to: quetzal_rainbow’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Making progress in cryonics or, better still, in longevity medicine, would be good ways to lengthen optimal [for existing people, from a mundane perspective] AI timelines.
With cryonics, the social obstacles seem formidable. My impression is that there’s at least a somewhat decent chance that current technology is already sufficient for successful preservation, and has been so for decades; yet uptake remains negligible. The discontinuity that suspension involves—in social participation etc. - also reduces its attractiveness.
With longevity medicine, uptake would be easier (although it would still be challenging to achieve a globally significant impact if treatments are expensive). So far, technical progress has been slow. Maybe there’s some hope that AI progress could accelerate this, even short of full general superintelligence—an additional reason for why, if there is to be a pause, it may be best for that pause to occur as late as possible.
There is also accumulating risks that our civilization gets destroyed before superintelligence. This includes xrisks. But also significant from a person-affecting perspective are scenarios of collapse or derailment that fall short of extinction or permanent loss of potential. For example, a big nuclear war or a bad engineered pandemic or a global breakdown of order could be devastating from a person-affecting perspective even if in the long-run humanity recovered.
If somehow we could be become both individually and collectively safe, then I think the person-affecting perspective would favor a much slower and more risk-averse pace of AI development.
What links here?
- Nick Bostrom's comment on Optimal Timing for Superintelligence: Mundane Considerations for Existing People by Nick Bostrom (15 Feb 2026 11:21 UTC; 1 point)

Nick Bostrom 14 Feb 2026 20:12 UTC
18 points
4
in reply to: habryka’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
In other papers (e.g. Existential Risks (2001), Astronomical Waste (2003), and Existential Risk Prevention as a Global Priority (2013)) I focus mostly on what follows from a mundane impersonal perspective. Since that perspective is even further out of step with how humanity makes governance decisions, is it your opinion that those paper should likewise be castigated? (Some people who hate longtermism have done so, quite vehemently.) But my view is that there can be value in working out what follows from various possible theoretical positions, especially ones that have a distinguished pedigree and are taken seriously in the intellectual tradition. Certainly this is a very standard thing to do in academic philosophy, and I think it’s usually a healthy practice.

Nick Bostrom 14 Feb 2026 16:08 UTC
3 points
0
in reply to: Lukas_Gloor’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
On the substance-level, btw, one “arcane consideration” that I would put a lot of weight on, even on person-affecting views, is stuff like what Bostrom talks about in the Cosmic Host paper. … Even if we can’t align AIs to human values, I feel like we at least have a duty to make them good at the building of peaceful coalitions/it being an okay thing to add to the cosmic host.
Yes, although, as that paper discusses, speed may also be important insofar as it reduces the risk of us failing to add anything at all, since that’s also something the cosmic host may care about—the risk that we fail ever to produce superintelligence. (My views about those things are quite tentative, and they fall squarely into the ‘arcane’. I agree on their importance.)

Nick Bostrom 14 Feb 2026 13:58 UTC
16 points
1
in reply to: habryka’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
The average planning horizon for climate change regulation is many decades in the future. Nuclear waste management policies are expected to contain waste for hundreds of years. … Maybe you have some other evidence that convinced you that countries and policy-makers operate on person-affecting views?
Of course they don’t consistently operate on any specific moral view. But I would claim that they are less badly approximated by ‘benefit currently existing citizens’ than ‘neutrally benefit all possible future people (or citizens) that might be brought into existence over future eons’. Much less is spent on things like nuclear waste management and preventing climate change than on providing amenities for the current population. In fact, they may be spending a net negative amount of resources on trying to benefit future generations, since they are often saddling future generations with vast debt burdens in order to fund current consumption. (FHI—particularly Toby Ord—was involved in some efforts to try to infuse a little bit more consideration of future generations in UK policymaking, but I think only very limited inroads where made on that front.)

Nick Bostrom 14 Feb 2026 3:24 UTC
17 points
1
in reply to: habryka’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Like, I think a persuasive or reasonable paper would have put its central load-bearing assumptions up front.
More up front than in the title?
it obviously shouldn’t be the basis of societal decision-making, and luckily also isn’t”.
Societal decision-making typically uses a far narrower basis than the general person-affecting stance that this paper analyzes. For example, not only do voters and governments usually not place much weight on not-yet-conceived people that might come into existence in future millennia, but they care relatively little about what happens to currently existing persons in other countries.

Nick Bostrom 13 Feb 2026 17:22 UTC
56 points
7
in reply to: Jan_Kulveit’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Yes the post explicitly considers things only from a mundane person-affecting stance, and I would not argue that this is the correct stance or the one I would all-things-considered endorse. However, it may be a component in a more plausible complex view, or a caucus in a ‘moral parliament’; so I think it is worth investigating what it implies. If ethics is complicated, we may need a divide and conquer strategy, where we isolate and analyze one element at a time.
Existing ordinary people usually have preferences and values not captured by the QALY calculation, such as wanting their kids having happy lives, or wanting the world not to end, or even wanting to live in a world which they understand.
I agree that people also have such preferences. Regarding people wanting their kids to have happy lives, the post does discuss that (under ‘other-focused prudential concerns’). Ceteris paribus, this pushes towards longer timelines being optimal. People might want their parents to have happy lives, which pushes in the opposite direction.
Preferences for not wanting the world to end: I think this would need to be discussed in the context of an analysis of which timelines are optimal from an xrisk-minimization perspective, which was (lamely) set aside for possible future work.
“live in a world they understand”—If this is something people want, it seems plausible AI could help a lot, by providing better explanations and perhaps cognitive enhancements. In the status quo, I think most technological devices (and to some extent our institutions) are, to most people, black boxes that offer somewhat understandable affordances. And even highly educated people are ignorant about the most fundamental aspects of how the world works, since we e.g. lack a theory of quantum gravity or a proper understanding of simulation theory and whatever other such ultimate parameters define reality.
Using the “undergo risky surgery” analogy, this analysis assumes the patient’s preferences have no place in the analysis, and if and when the surgery should happen should be decided by a utilitarian social planner.
I don’t think so. In the surgery example, the patient presumably gets to make the final decision; yet one may want to analyze the relevant tradeoffs to help inform their choice. In the case of AI, one can analyze the consequences for various interests or values of different choices that could be made—independently of who (or what institution) one thinks ultimately ought to make the decision. In any case, it seems likely that the outcome will be determined by processes that are not morally optimal or maximally legitimate; so the main question of interest may be how, on the margin, various stakeholders might wish to try to nudge things.
The analysis hedges by “not supporting any particular policy prescriptions” but I do expect in practice to be quoted in informal motte-and-bailey way
That is plausible. (This is also the case with much of my other writings, alas—I have no comparative advantage in being an ‘on message’ communicator; and I suspect any tiny specks of value there might be in my work would be washed away if I tried much harder to exercise message discipline.)

Nick Bostrom 14 Sep 2025 23:29 UTC
26 points
12
in reply to: anaguma’s comment on: AnnaSalamon’s Shortform
I’m not recommending it (haven’t done a cost-benefit analysis) but Anna was considering flying less because of the fume risk, and bringing a gas mask in the carry-on seems less burdensome than refraining from flying.

Nick Bostrom 14 Sep 2025 16:59 UTC
21 points
7
in reply to: AnnaSalamon’s comment on: AnnaSalamon’s Shortform
Could bring a gas mask?

Nick Bostrom 5 Sep 2025 2:07 UTC
3 points
2
in reply to: joshc’s comment on: Open Global Investment as a Governance Model for AGI
My biggest problem with this proposal is that it restricts AGI projects to a single entity, which I think is pretty far from the status quo.
It doesn’t, though. The paper talks about OGI-N as well as OGI-1. In the former version, there are multiple AGI-developing entities.

Nick Bostrom 5 Sep 2025 2:02 UTC
7 points
3
in reply to: NickH’s comment on: Open Global Investment as a Governance Model for AGI
I think that’s an overstatement. There is still plenty of demand from international investors for U.S. assets, and e.g. yields on U.S. 30-year treasury bonds are not that high by historical standards (although they’ve been climbing somewhat since their historic lows in 2020.) If there is a reduction in confidence in other sorts of USG commitments, but maintained confidence in it’s basic financial commitments and in U.S. property law, then that might support the paper’s contention that the latter kind form a comparatively more reliable basis for positive-sum deals in relation to AGI-development than other types of agreements.
What links here?
- Open Global Investment: Comparisons and Criticisms by Algon (15 Oct 2025 17:20 UTC; 15 points)

Nick Bostrom 31 Aug 2025 2:05 UTC
14 points
3
in reply to: Haiku’s comment on: Open Global Investment as a Governance Model for AGI
“I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn’t.”
You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier.
Regarding attitudes about AGI, that’s probably a bigger topic for another time. But regarding your and your loved ones’ life expectancy, from a mundane perspective (which leaves out much that is actually very relevant), it would presumably be some small number of decades without AGI—less if the people you love are elderly or seriously ill. Given aligned AGI, it could be extremely long (and immensely better in quality). So even if we assume that AGI would arrive soon unless stopped (e.g. in 5 years) and would result in immediate death if unaligned (which is very far from a given), then it seems like your life expectancy would be vastly longer if AGI developed even if the chance of alignment were quite small.
These beginnings could never have happened through quiet dealings and gently laid plans. They happened because people were honest and loud.
I don’t doubt that loud people sometimes make things happen, though all-too-often the things they make happen turn out to have been for the worse. For my own part, I don’t feel there’s such a deficit of loud people in the world that it is my calling to rush out and join them. This is partly a matter of personality, but I hope there’s a niche from which one can try to contribute in a more detached manner (and that there is value in a “rationalist project” that seeks to protect and facilitate that).

Nick Bostrom 30 Aug 2025 15:01 UTC
16 points
8
in reply to: Haiku’s comment on: Open Global Investment as a Governance Model for AGI
Every known plan for a post-AGI world is one which I do not expect my loved ones to survive.
I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn’t.
Btw, the OGI model is not primarily intended for a post-AGI world, but rather for a near-term or intermediary stage.
However, I agree that if somebody thinks that we should completely stop AGI then the OGI model would presumably not be the way to go. It is presented as an alternative to other governance models for the development of AGI (such as Manhattan project, CERN, Intelsat, etc.). This paper doesn’t address the desirability of developing AGI.
to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve —
Fwiw, I think it’s more likely that AI will be aligned than that it will be shut down.
I am grateful that you have spread awareness of the risk of human extinction from AI. I am genuinely saddened that you seem to be working to bring it about.
One has to take the rough with the smooth… (But really, you seem to be misattributing motive to me here.)
If we here who know the stakes are not united in our call to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve — then what was the rationalist project ever about?
I see it more like a flickering candle straining to create a small patch of visibility in an otherwise rather dark environment. Strong calls for unanimity and falling into line with a political campaign message is a wind that might snuff it out.

Nick Bostrom 29 Aug 2025 1:24 UTC
9 points
1
in reply to: Jack_S’s comment on: Open Global Investment as a Governance Model for AGI
It becomes partly terminological, but I would say that China-OGI-1 would by definition be a situation in which global investors in a Chinese company that develops AGI enjoy reasonably reliable assurances that their property rights would be largely respected. It seems maybe more attractive than the closest alternatives (i.e. a situation in which AGI is developed by a Chinese company and international investors don’t have reasonable assurances that their rights would be protected, or a situation in which AGI is developed by a Chinese Manhattan project)? So the factors you point to don’t affect the desirability of China-OGI-1 but rather the probability of that version of the OGI model becoming instantiated.
Btw, I should maybe also reemphasize that I’m putting forward the model more so that it can be considered alongside other models that have been proposed, rather than as something that I have a strong or definitive commitment to. I could easily imagine coming to favor other approaches, either as a result of arguments or because of changes in the world that alters the practical landscape. I generally have a high level of uncertainty about the politics/governance/macrostrategy of AI (doubly so in the Chinese context, where I have even less understanding), and I tend to think we’ll need to feel and negotiate our way forward one tentative step at a time rather than operate on a fixed plan.
What links here?
- Open Global Investment: Comparisons and Criticisms by Algon (15 Oct 2025 17:20 UTC; 15 points)

Nick Bostrom 29 Aug 2025 0:55 UTC
19 points
3
in reply to: Wei Dai’s comment on: Open Global Investment as a Governance Model for AGI
I always thought it was totally crazy for people to lump Nick Bostrom and Marc Andreessen together into TESCREAL and criticize them in the same breath, but this post plays right into such criticism.
I’m also bald...

Nick Bostrom 28 Aug 2025 18:18 UTC
3 points
−12
in reply to: Wei Dai’s comment on: Open Global Investment as a Governance Model for AGI
This idea/plan seems to legitimize giving founders and early investors of AGI companies extra influence on or ownership of the universe (or just extremely high financial returns, if they were to voluntarily sell some shares to the public as envisioned here), which is hard for me to stomach from a fairness or incentives perspective, given that I think such people made negative contributions to our civilizational trajectory by increasing x-risk.
One question is whether a different standard should be applied in this case than elsewhere in our capitalist economy (where, generally, the link between financial rewards and positive or negative contributions to xrisk reduction is quite tenuous). One could argue that this is the cooperative system we have in place, and that there should be a presumption against retroactively confiscating people who invested their time or money on the basis of the existing rules. (Adjusting levels of moral praise in light of differing estimations of the nature of somebody’s actions or intentions may be a more appropriate place for this type of consideration to feed in. Though it’s perhaps also worth noting that the prevailing cultural norms at the time, and still today, seem to favor contributing to the development more advanced AI technologies.)
Furthermore, it would be consistent with the OGI model for governments (particularly the host government) to take some actions to equalize or otherwise adjust outcomes. For example, many countries, including the U.S., have a progressive taxation system, and one could imaging adding some higher tax brackets beyond those that currently exist—such as an extra +10% marginal tax rate for incomes or capital gains exceeding 1 trillion dollars, or exceeding 1% of GDP, or whatever. (In the extreme, if taxation rates began approaching 100%, this would become confiscatory and would be incompatible with the OGI model; but there is plenty of room below that for society to choose some level of redistribution.)
What links here?
- Open Global Investment: Comparisons and Criticisms by Algon (15 Oct 2025 17:20 UTC; 15 points)

Nick Bostrom 28 Aug 2025 1:47 UTC
12 points
2
in reply to: Mitchell_Porter’s comment on: Open Global Investment as a Governance Model for AGI
the governance philosophy here seems to be “let the companies do as they will and let events unfold as they may”
That is not quite right. The idea is rather that the government does whatever it does by regulating companies, or possibly entering into some soft-nationalization public-private partnership, as opposed to by operating an AGI project on its own (as in the Manhattan model) or by handing it over to an international agency or consortium (as in the CERN and Intelsat models).
There doesn’t seem to be anything here which addresses the situation in which one company tries to take over the world using its AGI, or in which an AGI acting on its own initiative tries to take over the world, etc.
It doesn’t particularly address the situation in which an AGI on its own initiative tries to take over the world. That is a concern common to all of the governance models. In the OGI model, there are two potential veto points: the company itself can choose not to develop or a deploy an AI that it deems too risky, and the host government can prevent the company from developing or deploying an AI that fails to meet some standard that the government stipulates. (In the Manhattan model, there’s only one veto point.)
As for the situation in which one company tries to take over the world using its AGI, the host government may choose to implement safeguards against this (e.g. by closely scrutinizing what AGI corporations are up to). Note that there are analogous concerns in the alternative models, where e.g. a government lab or some other part of a government might try to use AGI for power grabs. (Again, the double veto points in the OGI model might have some advantage here, although the issue is complicated.)