Could bring a gas mask?
Nick Bostrom
Karma: 450
My biggest problem with this proposal is that it restricts AGI projects to a single entity, which I think is pretty far from the status quo.
It doesn’t, though. The paper talks about OGI-N as well as OGI-1. In the former version, there are multiple AGI-developing entities.
I think that’s an overstatement. There is still plenty of demand from international investors for U.S. assets, and e.g. yields on U.S. 30-year treasury bonds are not that high by historical standards (although they’ve been climbing somewhat since their historic lows in 2020.) If there is a reduction in confidence in other sorts of USG commitments, but maintained confidence in it’s basic financial commitments and in U.S. property law, then that might support the paper’s contention that the latter kind form a comparatively more reliable basis for positive-sum deals in relation to AGI-development than other types of agreements.
“I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn’t.”
You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier.
Regarding attitudes about AGI, that’s probably a bigger topic for another time. But regarding your and your loved ones’ life expectancy, from a mundane perspective (which leaves out much that is actually very relevant), it would presumably be some small number of decades without AGI—less if the people you love are elderly or seriously ill. Given aligned AGI, it could be extremely long (and immensely better in quality). So even if we assume that AGI would arrive soon unless stopped (e.g. in 5 years) and would result in immediate death if unaligned (which is very far from a given), then it seems like your life expectancy would be vastly longer if AGI developed even if the chance of alignment were quite small.
These beginnings could never have happened through quiet dealings and gently laid plans. They happened because people were honest and loud.
I don’t doubt that loud people sometimes make things happen, though all-too-often the things they make happen turn out to have been for the worse. For my own part, I don’t feel there’s such a deficit of loud people in the world that it is my calling to rush out and join them. This is partly a matter of personality, but I hope there’s a niche from which one can try to contribute in a more detached manner (and that there is value in a “rationalist project” that seeks to protect and facilitate that).
Every known plan for a post-AGI world is one which I do not expect my loved ones to survive.
I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn’t.
Btw, the OGI model is not primarily intended for a post-AGI world, but rather for a near-term or intermediary stage.
However, I agree that if somebody thinks that we should completely stop AGI then the OGI model would presumably not be the way to go. It is presented as an alternative to other governance models for the development of AGI (such as Manhattan project, CERN, Intelsat, etc.). This paper doesn’t address the desirability of developing AGI.
to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve —
Fwiw, I think it’s more likely that AI will be aligned than that it will be shut down.
I am grateful that you have spread awareness of the risk of human extinction from AI. I am genuinely saddened that you seem to be working to bring it about.
One has to take the rough with the smooth… (But really, you seem to be misattributing motive to me here.)
If we here who know the stakes are not united in our call to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve — then what was the rationalist project ever about?
I see it more like a flickering candle straining to create a small patch of visibility in an otherwise rather dark environment. Strong calls for unanimity and falling into line with a political campaign message is a wind that might snuff it out.
It becomes partly terminological, but I would say that China-OGI-1 would by definition be a situation in which global investors in a Chinese company that develops AGI enjoy reasonably reliable assurances that their property rights would be largely respected. It seems maybe more attractive than the closest alternatives (i.e. a situation in which AGI is developed by a Chinese company and international investors don’t have reasonable assurances that their rights would be protected, or a situation in which AGI is developed by a Chinese Manhattan project)? So the factors you point to don’t affect the desirability of China-OGI-1 but rather the probability of that version of the OGI model becoming instantiated.
Btw, I should maybe also reemphasize that I’m putting forward the model more so that it can be considered alongside other models that have been proposed, rather than as something that I have a strong or definitive commitment to. I could easily imagine coming to favor other approaches, either as a result of arguments or because of changes in the world that alters the practical landscape. I generally have a high level of uncertainty about the politics/governance/macrostrategy of AI (doubly so in the Chinese context, where I have even less understanding), and I tend to think we’ll need to feel and negotiate our way forward one tentative step at a time rather than operate on a fixed plan.
I always thought it was totally crazy for people to lump Nick Bostrom and Marc Andreessen together into TESCREAL and criticize them in the same breath, but this post plays right into such criticism.
I’m also bald...
This idea/plan seems to legitimize giving founders and early investors of AGI companies extra influence on or ownership of the universe (or just extremely high financial returns, if they were to voluntarily sell some shares to the public as envisioned here), which is hard for me to stomach from a fairness or incentives perspective, given that I think such people made negative contributions to our civilizational trajectory by increasing x-risk.
One question is whether a different standard should be applied in this case than elsewhere in our capitalist economy (where, generally, the link between financial rewards and positive or negative contributions to xrisk reduction is quite tenuous). One could argue that this is the cooperative system we have in place, and that there should be a presumption against retroactively confiscating people who invested their time or money on the basis of the existing rules. (Adjusting levels of moral praise in light of differing estimations of the nature of somebody’s actions or intentions may be a more appropriate place for this type of consideration to feed in. Though it’s perhaps also worth noting that the prevailing cultural norms at the time, and still today, seem to favor contributing to the development more advanced AI technologies.)
Furthermore, it would be consistent with the OGI model for governments (particularly the host government) to take some actions to equalize or otherwise adjust outcomes. For example, many countries, including the U.S., have a progressive taxation system, and one could imaging adding some higher tax brackets beyond those that currently exist—such as an extra +10% marginal tax rate for incomes or capital gains exceeding 1 trillion dollars, or exceeding 1% of GDP, or whatever. (In the extreme, if taxation rates began approaching 100%, this would become confiscatory and would be incompatible with the OGI model; but there is plenty of room below that for society to choose some level of redistribution.)
the governance philosophy here seems to be “let the companies do as they will and let events unfold as they may”
That is not quite right. The idea is rather that the government does whatever it does by regulating companies, or possibly entering into some soft-nationalization public-private partnership, as opposed to by operating an AGI project on its own (as in the Manhattan model) or by handing it over to an international agency or consortium (as in the CERN and Intelsat models).
There doesn’t seem to be anything here which addresses the situation in which one company tries to take over the world using its AGI, or in which an AGI acting on its own initiative tries to take over the world, etc.
It doesn’t particularly address the situation in which an AGI on its own initiative tries to take over the world. That is a concern common to all of the governance models. In the OGI model, there are two potential veto points: the company itself can choose not to develop or a deploy an AI that it deems too risky, and the host government can prevent the company from developing or deploying an AI that fails to meet some standard that the government stipulates. (In the Manhattan model, there’s only one veto point.)
As for the situation in which one company tries to take over the world using its AGI, the host government may choose to implement safeguards against this (e.g. by closely scrutinizing what AGI corporations are up to). Note that there are analogous concerns in the alternative models, where e.g. a government lab or some other part of a government might try to use AGI for power grabs. (Again, the double veto points in the OGI model might have some advantage here, although the issue is complicated.)
Open Global Investment as a Governance Model for AGI
There’s also https://whenaiseemsconscious.org/. (Ideally this will be improved over time. Several people contributed, but Lucius Caviola is coordinating revisions to the text, so if somebody has suggestions they could send them to him.)
Could we get info on this by looking at metrics of corruption? I’m not familiar with the field, but I know it’s been busy recently, and maybe there’s some good papers that put the private and public sectors on the same scale. A quick google scholar search mostly just convinced me that I’d be better served asking an expert.
I suspect it would be difficult to get much useful signal on this from the academic literature. This particular issue might instead come down to how much you trust the various specific persons that are the most likely corporate AI leaders versus some impression of how trustworthy, wholesome, and wise the key people inside or controlling a government-run AGI program would be (in the U.S. or China, over the coming years).
Btw, I’m thinking of the OGI model as offering something of a dual veto structure—in order for something to proceed, it would have be favored by both the corporation and the host government (in contrast to an AGI Manhattan project, where it would just need to be favored by the government). So at least the potential may exist for there to be more checks and balances and oversight in the corporate case, especially in the versions that involve some sort of very soft nationalization.
your comparison includes points like “OGI obviates the need for massive government funding” … If governments enact a policy that maintains the ability to buy shares in AI labs, but requires massive government funding and is disagreeable to incumbents, that seems to be part of a different story (and with a different story about how you get trustworthiness, fair distribution, etc.) than the story you’re telling about OGI.
In the OGI model, governments have the option to buy shares but also the option not to. It doesn’t require government funding, but if one thinks that it would be good for governments to spend money some on AGI-related stuff then they could do so in the OGI model just as well as in other models—in some countries, maybe even more easily, since e.g. some pension funds and sovereign wealth funds could more easily be used to buy stocks than to be clawed back and used to fund a Manhattan project. Also, I’m imagining that it would be less disagreeable to incumbents (especially key figures in AI labs and their investors) for governments to invest money in their companies than to have their companies shut down or nationalized or outcompeted by a government-run project.
Tunneling is always a concern in corporate structures, but alternative organizational forms suffer similar problems. Government officials, university department heads, and NGO executives also sometimes misuse the powers of their office to pursue personal or factional interests rather than the official mission of the organization they are supposed to represent. We would need a reason for thinking that this problem is worse in the corporate case in order for it to be a consideration against the OGI model.
As for the suggestion that governments (nationally or internationally) should prohibit profit-generating activities by AI labs that have major negative externalities, this is fully consistent with the OGI model (see section “The other half of the picture”, on p. 4). AGI corporations would be subject to government regulation and oversight, just like other corporations are—and, plausibly, the intensity of government involvement would be much greater in this case, given the potentially transformative impacts of the technology they are developing. It would also consistent with the OGI model for governments to offer contracts or prizes for various prosocial applications of AI.
You are looking at the wreckage of an abandoned book project. We got bogged down & other priorities came up. Instead of writing the book, we decided to just publish a working outline and call it a day.
The result is not particularly optimized for tech executives or policymakers — it’s not really optimized for anybody, unfortunately.
The propositions all *aspire* to being true, although some of may not be particularly relevant or applicable in certain scenarios. Still, there could be value on working out sensible things to say to cover quite a wide range of scenarios, partly because we don’t know which scenario will happen (and there is disagreement over the probabilities), but partly also because this wider structure — including the parts that don’t directly pertain to the scenario that actually plays out — might form a useful intellectual scaffolding, which could slightly constrain and inform people’s thinking of the more modal scenarios.
I think it’s unclear how well reasoning by analogy works in this area. Or rather: I guess it works poorly, but reasoning deductively from first principles (at SL4, or SL15, or whatever) might be equally or even more error-prone. So I’ve got some patience for both approaches, hoping the combo has a better chance of avoiding fatal error than either the softheaded or the hardheaded approach has on its own.
“In IEEE Spectrum’s sad little attempt at Singularity coverage, one bright spot is Paul Wallich’s “Who’s Who In The Singularity”,...”
Brightness here being a relative quality… I am labeled green, meaning “true believer, thinks it will happen within 30 years.” Yet I am quoted (correctly) as saying “I would… assign less than a 50% probability to superintelligence being developed by 2033.” (I also don’t endorse “once the singularity comes near, we will all be kicking ourselves for not having brought it about sooner”, even though they attribute this to me as my “central argument”.)
Reg Oracle AI, I’m not sure how much of a disagreement there exists between Eliezer and me. My position has not been that it is definitely the case that Oracle AI is the way to go. Rather, my position is something like “this seems to have at least something going for it; I have not yet been convinced by the arguments I’ve heard against it; it deserves some further consideration”. (The basic rationale is this: While I agree that a utility function that is maximized by providing maximally correct and informative answers to our questions is clearly unworkable (since this could lead the SI to transform all Earth into more computational hardware so as to better calculate the answer), it might turn out to be substantially easier to specify the needed constraints to avoid such catastrophic side-effects of an Oracle AI than it is to solve the Friendliness problem in its general form—I’m not at all sure it is easier, but I haven’t yet been persuaded it is not.)
Reg disagreement between Robin and Eliezer on singularity: They’ve discussed this many times, both here and on other mailinglists. But the discussion always seems to end prematurely. I think this would make for a great disagreement case study—topic is important, both are disagreement savvy, both know and respect one another, both have some subject matter expertise… I would like them to try once to get to the bottom of the issue, and continue discussion until they either cease disagreeing or at least agree exactly on what they disagree about, and why, and on how each person justifies the persistent disagreement.
Eliezer, the criteria you list may be necessary for the evolution of complex structures. But I think it’s worth highlighting that practically important evolutionary results could come about without the need for new complex structures. For example, suppose we have a population of controlled self-replicating nanobots, built unwisely in such a way that they keep replicating until a separate breaking circuit kicks in and shuts off replication. Now suppose there is a mutation in the code of one nanobot such that its offspring lack a working breaking circuit. Then this mutant nanobot could start an exponential goo. There need only be a one-step selection, but the results could be dramatic. Similarly with Hanson’s colonizers that burn the cosmic commons—they might not gain much in complexity through evolution, but evolutionary selection could ensure that a certain type of colonizer (which was present but very rare at time=0) will eventually dominate at the frontier.
Wei, no I don’t think I considered the possibility of discounting people by their algorithmic complexity.
I can see that in the context of Everett it seems plausible to weigh each observer with a measure proportional to the amplitude squared of the branch of the wave function on which he is living. Moreover, it seems right to use this measure both to calculate the anthropic probability of me finding myself as that observer and the moral importance of that observer’s well-being.
Assigning anthropic probabilities over infinite domains is problematic. I don’t know of a fully satisfactory explanation of how to do this. One natural approach might to explore might be to assign some Turing machine based measure to each of the infinite observers. Perhaps we could assign plausible probabilities by using such an approach (although I’d like to see this worked out in detail before accepting that it would work).
If I understand your suggestion correctly, you propose that the same anthropic probability measure should also be used as a measure of moral importance. But there seems to me to be a problem. Consider a simple classical universe with two very similar observers. On my reckoning they should each get anthropic probability measure 1⁄2 (rejecting SIA, the Self-Indication Assumption). Yet it appears that they should each have a moral weight of 1. Does your proposal require that one accepts the SIA? Or am I misinterpreting you? Or are you trying to explicate not total utilitarianism but average utilitarianism?
I have a paper which explores the problem in a somewhat more general way (but see especially section 6.3).
Infinite Ethics: http://www.nickbostrom.com/ethics/infinite.pdf
Carl, I like your suggestion to establish a prize for avoing mega-disasters and existential risks. (Meanwhile, I’m going to send Petrov a small donation.)
One of the bias issues this raises is the possibility of bias in how we allocate our attention. One could think of an attention allocation as if it involved an implicit belief that “this is worth attending to”. Then we can think of how this kind of implicit belief might be biased. For example, in the ancestral environment nobody was worth attending to because they had prevented millions of deaths by refraining from pressing a button; so maybe we are biased in the direction of allocating too little attention to such acts… Some future post might explore this in more detail.
Eliezer, thanks for your post.
I’m not recommending it (haven’t done a cost-benefit analysis) but Anna was considering flying less because of the fume risk, and bringing a gas mask in the carry-on seems less burdensome than refraining from flying.