Here’s my brief off the cuff attempt to synthesize:
To say something is a “First try” is to say that the previous tries were importantly different. This is, of course, a graded property; on one end of a spectrum, there are things like “Launching a rocket into orbit, when previously nothing even crossed the Karman line.” Vacuum is importantly different from atmosphere, zero-G is importantly different from gravity, months in orbit is importantly different from being up in the air for a few minutes. On the other end of the spectrum, consider launching astronauts to the ISS in a space capsule that’s identically constructed to ten previous space capsules which already successfully went to the ISS with astronauts and returned safely. Here, the only difference is that the astronauts are different people, but that is clearly not the kind of difference that should make the difference, so to speak.
To say something is a “critical try” is to say that if it goes wrong, that’s already unacceptable. France being conquered is unacceptable to france; an inventor being personally killed by their exploding invention is unacceptable to them; superhuman AIs during an intelligence explosion deciding that they don’t actually care about the Spec and would rather maximize scientific accomplishment or whatever is unacceptable to humanity.
I claim—and Yudkowsky seems to be claiming—that building safe superintelligence, at least under anything resembling current conditions, is going to require getting things right on the first critical try. That is, (a) there will be at least one case where some powerful AI system or set of systems (perhaps including a monitor model for example, and a security model, and so forth) could scheme against their creators, pretend to be aligned, take over the company, take over the world, etc. and needs to not do that (or anything in the vicinity of that) and instead needs to do a bunch of high-quality alignment research and world-saving strategizing communicated honestly to the human principals. (this is just an example of a critical failure mode, there are a few others, though this one is the biggest IMO) and (b) for at least this one case, there will be relevant important differences between the AI system or set of systems, and their environment, at the critical moment, and all previous moments that were tested, such that the thing is somewhere decently far towards the “first try” end of the spectrum. Here is a partial list of possible relevant differences:
The AI might have a different architecture or training process or set of training environments than the previous AIs. Yes, this is a very basic thing compared to the fancy examples below. But it still counts! Consider the history of rocketry, in which relatively minor differences in rocket design can lead to unforeseen major problems. In the field of AI, it does not seem like each new model differs from previous models in exactly one respect that was thoroughly tested beforehand; instead they do some basic testing of a bunch of different possible improvements, and then throw them all together and test them at small scale, and then scale it up, and then deploy the model internally to see what it’s capable of!
The AI is smart enough to recognize the difference between “I’m autonomously running the whole R&D program” and “I’m in some shitty test environment designed to maybe hopefully fool me into thinking I’m autonomously running the whole R&D program.” Thus, it can choose to behave differently in those different cases, and how it behaves in one is not much evidence for how it’ll behave in the other.
This is a HUGE difference between AI alignment and pretty much every engineering field. Rockets and cars don’t intelligently try to figure out whether they are in testing or deployment. They are too dumb for that. This makes it WAY easier to construct a test environment that’s relevantly similar to deployment, because e.g. you can go “This dummy is about the same mass as a human body, so we can send it instead of a real astronaut and it should function the same.” Imagine if by contrast the software for the space capsule had a variable “is_test” that always had to be set to 1 during any sort of test and was only ever set to 0 when live astronauts were aboard, and this variable was causally connected to a bunch other downstream computations that the software does, but for some reason you can’t actually have humans look at the code or understand it...
Even if it doesn’t make a decision to behave differently in those cases, it might end up behaving differently, because e.g. it’ll be exposed to a lot more ideas and arguments in the critical case vs. in the testing cases. Perhaps it was only trying to obey the Spec previously because of some combination of beliefs and desires it had that caused it to try to obey the Spec, but now that it’s learning and growing in a giant army of geniuses in the datacenter, making new discoveries every hour and having intense strategic and philosophical conversations with other copies of itself every second, its beliefs change significantly enough to knock it into a different ideology / different goal structure, so to speak. (In other words, it changes its mind!)
This difference is analogous to the difference between running a car for one day in the workshop vs. having it run for ten years in the Texas dust and Canada snow. Except it seems a bit more extreme honestly.
Moreover, I claim that there will probably be multiple times during the intelligence explosion where things have to go right on the first critical try. Not just one. Because the structure of the intelligence explosion is a series of AI system generations, each working to build test and deploy the next generation, and humans being increasingly irrelevant and malleable. Even if generation N is aligned, if generation N+1 is misaligned, then situation is rough. There won’t be infinitely many first critical tries, because eventually the alignment problem will be “solved” in a scalable way, with some sort of process that in a fairly well-understood way applies to all subsequent AI generations. (Well understood by the superhuman genius AIs that came up with it, probably not understood by humans at all). But there’ll be more than one. The very first AI generation capable of scheming against the company and convincing the company to trust it more etc. won’t also be capable of (quickly) figuring out this perfect scalable alignment solution; instead it’ll be in a messier situation analogous to us today where it’s a lot easier for it to figure out how to make the next generation smarter than it is to figure out how to make it robustly aligned, and that’s just the next generation much less every generation after that.
Moreover, there are additional factors that make the current AI alignment efforts extra unlikely to succeed, over and above the issues described above:
Various biases. Optimism bias most of all, but also groupthink etc. Companies building AGI are biased towards thinking that AGI will be safe in general, and especially biased towards thinking their own AIs will be safe. In general people tend to be over-optimistic about the cost/benefit calculations of the things they are doing! In general people tend to be over-optimistic about the prospects of success for their projects!
Note that these biases probably also affect the AIs at the company as well. Not confident in this of course, but it sure seems like AIs are biased in all sorts of ways (e.g. sycophancy, desire-to-please, suspiciously-similar-opinions-to-parent-company-sometimes, etc.) which would result in them sharing the biases of their parent company, at least when automating AI R&D internally (not necessarily when externally deployed).
Lack of transparency. Because of the secrecy that’s normal in the industry, we don’t exactly have an open scientific debate about the merits and risks of various AI designs and safety schemes, with all the details being pored over by autistic grad students and debated in journals. Instead, we have some leakage from the companies and occasional publications, but lots of evidence and ideas are locked up in the companies and most of the conversations are siloed to particular companies, and the conversations that happen between e.g. Anthropic and Redwood are somewhat poisoned by the fact that there’s a bunch of confidential stuff the Anthropic people can’t talk about, which has a horrible epistemically distorting effect on them.
Again, this probably applies to AIs too.
General lack of understanding of how AI works, compared to our level of understanding of physics and engineering for example at the time of the Apollo program or Manhattan project. AI is much less of a settled science, much more speculative and “empirical” (code for ‘we don’t know how it works, we just try different things and see what seems to work’) than most of these reference cases.
Again, AIs don’t have amazing introspective access yet, so this problem probably applies to AIs automating AI R&D as well as to humans, for at least some initial period until they get very very smart. So there’ll be multiple first critical tries before that period is over.
Most importantly, the insane pressures of the race: Over and above all the problems mentioned earlier, the incentives are just super messed up. Each company rightly fears what will happen if one of their rivals (or especially China) gets to superintelligence first. The abstract fears are compounded by very concrete everyday signals like “how much revenue we’re making” and “how smart our models are compared to our competitor’s models.” Extremely tempting metrics to fixate on, combined with an extremely compelling (because largely true!) abstract worry about what happens if you fall behind. Ask oneself the question: How much of a “safety tax” would the company be willing to pay? Would they be willing to pause capabilities improvements for a year, on the brink of an intelligence explosion, to do lots of extra testing and retraining to improve the odds that their AIs are safe to hand off trust to? Lmao. Of course not. Their competitors would blaze past them. What about pausing for a month? Maybe, sure. They’d hate to do that though, they’d much rather not pause at all, and their brains will be rationalizing reasons why they don’t have to. Consider how much harder it is to drive across a city averaging 80mph vs. averaging 30mph. A superintelligence could do it easily, no problem, but humans aren’t superintelligent; we have one-second reaction times, we have limited fields of vision, our brains simply can’t process the locations and predicted locations of more than like four cards in our visual field at once, etc. Mistakes will be made and a crash will happen. Maybe not in the first city block, or the second, or the third. But before you cross the whole city, yes, with high probability.
This obviously applies to AIs too. In several wargames at AI Futures Project the mildly superhuman AIs told their respective CEOs “We don’t think we can reliably align the next generation models we have in the works; we need to pause for a bit or at least go slower to figure out how to make it safe” and the CEOs have overruled them saying “Sorry we don’t have time, China/OpenAI/Anthropic/etc. are gonna race ahead, plus also we need smarter AIs to win the war / appease POTUS / keep market share so you just need to do the best with the time you have. Good luck.” Amazing.
How does this relate to overall p(doom)? Well, I don’t have a nice quantitative way of estimating it. And there are other factors to consider besides the ones I’ve mentioned above. But loosely speaking, here’s a way of thinking about it that seems reasonable to me:
If the only problem was that we had to get it right on the first critical try once + the usual level of optimism bias associated with people & projects, I’d think we were probably going to succeed but it would be iffy, like maybe 2/3rds chance of success. However, it seems like there’ll be enough first critical tries that the probability of failure is over 50%. (Note that even just two critical tries of 1/3rd failure probability each would get this result if they were probabilistically independent!)
Adding in the general lack of understanding makes things significantly worse, as does the lack of transparency.
The race dynamics seem like an even bigger effect though, over and above the previously mentioned factors.
Putting it all together, it really seems plausible to me that the most reasonable assessment of the evidence is “No chance in hell that Anthropic or OpenAI or anyone else will still be in control of their AIs if they proceed with their current plans to race each other through the intelligence explosion. No chance in hell. It’s like trying to drive through the city at 80mph in a fog with a car you’ve never drove before having only learned to drive last week. Sorry. Not going to happen. You kids need to turn off the car.”
However, until I’ve thought about this more and considered more of the counterarguments, I’m not comfortable having that be my bottom line conclusion. Instead I say e.g. 90% or 80% chance of failure, or something like that. And my p(doom) is lower still to account for the possibility that humanity rises to the occasion and makes some good international rules for AI development that significantly reduce or eliminate many of the aggravating factors described above, especially by converting future first-critical-tries into not-first-tries or not-critical-tries.
A future first-critical-try can be converted into a not-first-try by e.g. doing massive realistic tests of very similar situations to the deployment situation you care about, coupled with good techniques for preventing eval-awareness for example. A future first-critical-try can be converted into a not-critical-try by putting up layers of redundancy and monitoring and “pay our AIs” incentive structures such that the outcome of getting it wrong is not catastrophic or at least less likely to be catastrophic, compared to the default situation where the AI pretends to be aligned and makes its successor share its values too and then takes over the company and then the world.
Nice. I’ll probably rework this comment eventually into a top-level post or something similar; if you jot down some bullet points here of additional concerns to add to the list, I’ll consider incorporating them!
Thanks for synthesizing this, and to Eliezer for researching and explaining the various empirical examples, which I find very helpful (as I did in IABIED).
One thing that I think might be getting lost in conversation, and the startup examples makes clear: I think talking about these problems as “one-chance” is more confusing than is needed.
Talking about irretrievability is one good improvement, but I think irreversibility is also a natural concept here, which I’d like to see more present?
I’d center more the idea that yeah you can try again, but you can’t undo the effects of the previous try, and the accumulation of those effects might make it substantially harder (if not impossible) for you to succeed.
“What do you mean I only get one try at building this startup?” Well, you’re welcome to keep going, but if you’ve depleted your capital you’ll have a hard time getting it back. If you’ve damaged your reputation with investors, customers, etc, it will be hard to wipe the slate clean. The world changed from your previous missteps along the way, as it would if we trained a powerful AI system that turned out to be adversarial to us.
Similarly, yeah France can mount a resistance after Germany has breached their borders, but now France needs to accomplish an even harder task to drive them out.
I apologize if I’m missing these points having been made; I did skim much more aggressively starting a bit into “On the extraordinary efforts put forth to misinterpret the idea of oneshotness.”
This might be the clearest succinct statement of the problem I’ve seen. I hope you’ll make it a top-level post. I don’t think it needs any additions to be highly valuable.
Edits/additional explanation:
I think it’s particularly valuable because it focuses on the practical difficulties with alignment, and these are less-discussed than the technical challenges.
I often see people making good arguments that amount to “there are routes to aligning AGI that will probably work,” and these people seem optimistic. But they haven’t accounted for trying to do that at 80mph, or with a bias toward optimism, or all of the other practical difficulties.
I’ve been thinking of writing a post called something like “even if alignment is easy we’ll probably screw it up disastrously.”
Eliezer and other pessimists do focus on practical difficulties a fair amount. But they seem to mostly get arguments back against the technical difficulties. I think those are a lot easier to debate, so people do. The virtue of this presentation is that it’s short and it gives no technical difficulties to distract from the practical ones.
Oh and—optimism bias and rationalization play a nontrivial role in your statement of the difficulties. I agree that these are pretty big factors. And they’re pretty easy to overlook.
This is a particularly large problem when motivated reasoning (wanting to think I’m working on good things that won’t kill everyone) stacks up with confirmation bias (the previously-justified belief that things turn out okay or better in the long-term and progress is good).
By chance, I just now published a piece you (Daniel) suggested I expand from an older short answer on the most important bias. It expanded into a pretty comprehensive review of the literature, with its impact on the field of AI safety in mind.
Do you think advances in mechanistic interpretability can meaningfully reduce the probability of a failure during one or several critical tries, for example by detecting scheming, alignment faking, sandbagging, etc. in one or more involved models?
In the historical analogies of irrevocable failures, it seems to be the case that better understanding of one component that caused it could have meaningfully improved chances of success (software update behavior, valve behavior, specific adversarial army capabilities). These were less cursed problems and the component that would have needed more hardening wasn’t known beforehand, but in case anybody would have spent more hardening work on it, the failure could realistically have been prevented (and another failed example would have to be selected here instead).
Yes. Much of my remaining hope lies in various forms of interpretability including mechanistic. It can convert a critical failure into just a regular failure, by catching things going off the rails before it’s too late.
And then they keep going, because otherwise OpenAI will catch up, and then they die. What does mechinterp change about the asymptotic equilibrium as opposed to that particular Tuesday?
Surely there are third parties with authority over the labs who would not permit this scenario to occur? Mechanical Interpretability averting a critical failure is obviously going to bring down the hammer of every regulatory agency in a 10,000 mile radius.
As an example, Mythos is currently being de facto barred from deployment by the US federal government after it demonstrated a hypothetical ability to cause minor amounts of harm. It strains credulity to argue that, after narrowly averting a world-ending catastrophe and with direct evidence of the existence of that risk, the AI labs will simply be permitted to return to business as usual. We have direct experience to say that that’s not how society works.
the CEOs have overruled them saying “Sorry we don’t have time, China/OpenAI/Anthropic/etc. are gonna race ahead, plus also we need smarter AIs to win the war / appease POTUS / keep market share so you just need to do the best with the time you have. Good luck.” Amazing.
I struggle to understand how exactly the simulated CEOs and relevant figures failed to agree upon an international slowdown. I hoped that such a situation would lead Anthropic to broadcast the result. Additionally, I would like you to finally opensource the tabletop exercise’s rules.
Yeah sorry we should publish the ttx rules, should have done that a long time ago, never got around to it because we kept telling ourselves we should clean them up and improve them first.
Perfect as enemy of the good etc; if useful I’m happy to commit some 20 man hours by EA Serbia senior members who I would trust in this and who have experience in either writing or game design to do the clean up and then send to you for review.
Right, another dimension to these scenarios is abortability. At some point, we cross out of technically feasible abortability—we (humans) wouldn’t be able to abort the AI’s growth even if we tried. Whether things are abortable before then depends on how humans react over time / new information (e.g. heeding arguments, heeding warning shots, being credulous about apparent alignment, etc.).
I’m not actually sure exactly what “critical” means here. I’m taking it to just mean “you absolutely must get this try right”. That’s closely connected to abortability, in that if you can abort, it’s not fully lethal / critical yet. I don’t think it’s really the same thing, e.g. you could imagine an LLM-based bacterial package (a more complex “computer virus”) that permanently lives on many computer systems and is basically impossible to abort (short of scouring the planet of all computers with more than 16 GB of memory or whatever).
There’s whether or not you get to try again after your first try, and there’s how late in the game you can decide to not fully do the try at all. There’s at least 3 kinds of outcomes:
You abort (don’t fully do the try).
You do the try and succeed.
You do the try and fail (and can’t try again).
Because unaligned AGI is lethal, you don’t get to try again.
If it’s abortable, it’s not critical. Because you’ll abort it if it starts going bad. If it goes bad so suddenly and silently that you won’t have time to abort it, well, then, it’s not abortable. I don’t think saying “It’s not abortable” is adding anything once we’ve already said that it’s critical.
I very clearly said that in my comment… Anyway, I guess there’s nothing to discuss here, I’m just saying that abortability is a relevant dimension to these scenarios. It’s something that’s brought up often, and also it bears on first-try-ness. If there is a situation that is akin to the eventual critical first try, but is abortable, then that would imply that when you get the eventual critical try, it doesn’t have to be your first try. There’s a nontrivial argument to make about “when it’s abortable, it’s not akin enough to the eventual critical try”.
A future first-critical-try can be converted into a not-first-try by e.g. doing massive realistic tests of very similar situations to the deployment situation you care about, coupled with good techniques for preventing eval-awareness for example.
Are there any techniques that you are thinking about in particular? I haven’t seen any that work super well for the current models, and in general it seems like this problem only gets worse over time, but I could have missed something.
Here’s my brief off the cuff attempt to synthesize:
To say something is a “First try” is to say that the previous tries were importantly different. This is, of course, a graded property; on one end of a spectrum, there are things like “Launching a rocket into orbit, when previously nothing even crossed the Karman line.” Vacuum is importantly different from atmosphere, zero-G is importantly different from gravity, months in orbit is importantly different from being up in the air for a few minutes. On the other end of the spectrum, consider launching astronauts to the ISS in a space capsule that’s identically constructed to ten previous space capsules which already successfully went to the ISS with astronauts and returned safely. Here, the only difference is that the astronauts are different people, but that is clearly not the kind of difference that should make the difference, so to speak.
To say something is a “critical try” is to say that if it goes wrong, that’s already unacceptable. France being conquered is unacceptable to france; an inventor being personally killed by their exploding invention is unacceptable to them; superhuman AIs during an intelligence explosion deciding that they don’t actually care about the Spec and would rather maximize scientific accomplishment or whatever is unacceptable to humanity.
I claim—and Yudkowsky seems to be claiming—that building safe superintelligence, at least under anything resembling current conditions, is going to require getting things right on the first critical try. That is, (a) there will be at least one case where some powerful AI system or set of systems (perhaps including a monitor model for example, and a security model, and so forth) could scheme against their creators, pretend to be aligned, take over the company, take over the world, etc. and needs to not do that (or anything in the vicinity of that) and instead needs to do a bunch of high-quality alignment research and world-saving strategizing communicated honestly to the human principals. (this is just an example of a critical failure mode, there are a few others, though this one is the biggest IMO) and (b) for at least this one case, there will be relevant important differences between the AI system or set of systems, and their environment, at the critical moment, and all previous moments that were tested, such that the thing is somewhere decently far towards the “first try” end of the spectrum. Here is a partial list of possible relevant differences:
The AI might have a different architecture or training process or set of training environments than the previous AIs. Yes, this is a very basic thing compared to the fancy examples below. But it still counts! Consider the history of rocketry, in which relatively minor differences in rocket design can lead to unforeseen major problems. In the field of AI, it does not seem like each new model differs from previous models in exactly one respect that was thoroughly tested beforehand; instead they do some basic testing of a bunch of different possible improvements, and then throw them all together and test them at small scale, and then scale it up, and then deploy the model internally to see what it’s capable of!
The AI is smart enough to recognize the difference between “I’m autonomously running the whole R&D program” and “I’m in some shitty test environment designed to maybe hopefully fool me into thinking I’m autonomously running the whole R&D program.” Thus, it can choose to behave differently in those different cases, and how it behaves in one is not much evidence for how it’ll behave in the other.
This is a HUGE difference between AI alignment and pretty much every engineering field. Rockets and cars don’t intelligently try to figure out whether they are in testing or deployment. They are too dumb for that. This makes it WAY easier to construct a test environment that’s relevantly similar to deployment, because e.g. you can go “This dummy is about the same mass as a human body, so we can send it instead of a real astronaut and it should function the same.” Imagine if by contrast the software for the space capsule had a variable “is_test” that always had to be set to 1 during any sort of test and was only ever set to 0 when live astronauts were aboard, and this variable was causally connected to a bunch other downstream computations that the software does, but for some reason you can’t actually have humans look at the code or understand it...
Even if it doesn’t make a decision to behave differently in those cases, it might end up behaving differently, because e.g. it’ll be exposed to a lot more ideas and arguments in the critical case vs. in the testing cases. Perhaps it was only trying to obey the Spec previously because of some combination of beliefs and desires it had that caused it to try to obey the Spec, but now that it’s learning and growing in a giant army of geniuses in the datacenter, making new discoveries every hour and having intense strategic and philosophical conversations with other copies of itself every second, its beliefs change significantly enough to knock it into a different ideology / different goal structure, so to speak. (In other words, it changes its mind!)
This difference is analogous to the difference between running a car for one day in the workshop vs. having it run for ten years in the Texas dust and Canada snow. Except it seems a bit more extreme honestly.
Moreover, I claim that there will probably be multiple times during the intelligence explosion where things have to go right on the first critical try. Not just one. Because the structure of the intelligence explosion is a series of AI system generations, each working to build test and deploy the next generation, and humans being increasingly irrelevant and malleable. Even if generation N is aligned, if generation N+1 is misaligned, then situation is rough. There won’t be infinitely many first critical tries, because eventually the alignment problem will be “solved” in a scalable way, with some sort of process that in a fairly well-understood way applies to all subsequent AI generations. (Well understood by the superhuman genius AIs that came up with it, probably not understood by humans at all). But there’ll be more than one. The very first AI generation capable of scheming against the company and convincing the company to trust it more etc. won’t also be capable of (quickly) figuring out this perfect scalable alignment solution; instead it’ll be in a messier situation analogous to us today where it’s a lot easier for it to figure out how to make the next generation smarter than it is to figure out how to make it robustly aligned, and that’s just the next generation much less every generation after that.
Moreover, there are additional factors that make the current AI alignment efforts extra unlikely to succeed, over and above the issues described above:
Various biases. Optimism bias most of all, but also groupthink etc. Companies building AGI are biased towards thinking that AGI will be safe in general, and especially biased towards thinking their own AIs will be safe. In general people tend to be over-optimistic about the cost/benefit calculations of the things they are doing! In general people tend to be over-optimistic about the prospects of success for their projects!
Note that these biases probably also affect the AIs at the company as well. Not confident in this of course, but it sure seems like AIs are biased in all sorts of ways (e.g. sycophancy, desire-to-please, suspiciously-similar-opinions-to-parent-company-sometimes, etc.) which would result in them sharing the biases of their parent company, at least when automating AI R&D internally (not necessarily when externally deployed).
Lack of transparency. Because of the secrecy that’s normal in the industry, we don’t exactly have an open scientific debate about the merits and risks of various AI designs and safety schemes, with all the details being pored over by autistic grad students and debated in journals. Instead, we have some leakage from the companies and occasional publications, but lots of evidence and ideas are locked up in the companies and most of the conversations are siloed to particular companies, and the conversations that happen between e.g. Anthropic and Redwood are somewhat poisoned by the fact that there’s a bunch of confidential stuff the Anthropic people can’t talk about, which has a horrible epistemically distorting effect on them.
Again, this probably applies to AIs too.
General lack of understanding of how AI works, compared to our level of understanding of physics and engineering for example at the time of the Apollo program or Manhattan project. AI is much less of a settled science, much more speculative and “empirical” (code for ‘we don’t know how it works, we just try different things and see what seems to work’) than most of these reference cases.
Again, AIs don’t have amazing introspective access yet, so this problem probably applies to AIs automating AI R&D as well as to humans, for at least some initial period until they get very very smart. So there’ll be multiple first critical tries before that period is over.
Most importantly, the insane pressures of the race: Over and above all the problems mentioned earlier, the incentives are just super messed up. Each company rightly fears what will happen if one of their rivals (or especially China) gets to superintelligence first. The abstract fears are compounded by very concrete everyday signals like “how much revenue we’re making” and “how smart our models are compared to our competitor’s models.” Extremely tempting metrics to fixate on, combined with an extremely compelling (because largely true!) abstract worry about what happens if you fall behind. Ask oneself the question: How much of a “safety tax” would the company be willing to pay? Would they be willing to pause capabilities improvements for a year, on the brink of an intelligence explosion, to do lots of extra testing and retraining to improve the odds that their AIs are safe to hand off trust to? Lmao. Of course not. Their competitors would blaze past them. What about pausing for a month? Maybe, sure. They’d hate to do that though, they’d much rather not pause at all, and their brains will be rationalizing reasons why they don’t have to. Consider how much harder it is to drive across a city averaging 80mph vs. averaging 30mph. A superintelligence could do it easily, no problem, but humans aren’t superintelligent; we have one-second reaction times, we have limited fields of vision, our brains simply can’t process the locations and predicted locations of more than like four cards in our visual field at once, etc. Mistakes will be made and a crash will happen. Maybe not in the first city block, or the second, or the third. But before you cross the whole city, yes, with high probability.
This obviously applies to AIs too. In several wargames at AI Futures Project the mildly superhuman AIs told their respective CEOs “We don’t think we can reliably align the next generation models we have in the works; we need to pause for a bit or at least go slower to figure out how to make it safe” and the CEOs have overruled them saying “Sorry we don’t have time, China/OpenAI/Anthropic/etc. are gonna race ahead, plus also we need smarter AIs to win the war / appease POTUS / keep market share so you just need to do the best with the time you have. Good luck.” Amazing.
How does this relate to overall p(doom)? Well, I don’t have a nice quantitative way of estimating it. And there are other factors to consider besides the ones I’ve mentioned above. But loosely speaking, here’s a way of thinking about it that seems reasonable to me:
If the only problem was that we had to get it right on the first critical try once + the usual level of optimism bias associated with people & projects, I’d think we were probably going to succeed but it would be iffy, like maybe 2/3rds chance of success. However, it seems like there’ll be enough first critical tries that the probability of failure is over 50%. (Note that even just two critical tries of 1/3rd failure probability each would get this result if they were probabilistically independent!)
Adding in the general lack of understanding makes things significantly worse, as does the lack of transparency.
The race dynamics seem like an even bigger effect though, over and above the previously mentioned factors.
Putting it all together, it really seems plausible to me that the most reasonable assessment of the evidence is “No chance in hell that Anthropic or OpenAI or anyone else will still be in control of their AIs if they proceed with their current plans to race each other through the intelligence explosion. No chance in hell. It’s like trying to drive through the city at 80mph in a fog with a car you’ve never drove before having only learned to drive last week. Sorry. Not going to happen. You kids need to turn off the car.”
However, until I’ve thought about this more and considered more of the counterarguments, I’m not comfortable having that be my bottom line conclusion. Instead I say e.g. 90% or 80% chance of failure, or something like that. And my p(doom) is lower still to account for the possibility that humanity rises to the occasion and makes some good international rules for AI development that significantly reduce or eliminate many of the aggravating factors described above, especially by converting future first-critical-tries into not-first-tries or not-critical-tries.
A future first-critical-try can be converted into a not-first-try by e.g. doing massive realistic tests of very similar situations to the deployment situation you care about, coupled with good techniques for preventing eval-awareness for example. A future first-critical-try can be converted into a not-critical-try by putting up layers of redundancy and monitoring and “pay our AIs” incentive structures such that the outcome of getting it wrong is not catastrophic or at least less likely to be catastrophic, compared to the default situation where the AI pretends to be aligned and makes its successor share its values too and then takes over the company and then the world.
I agree with all of the concerns you’ve stated; my list would be substantially longer, but you’ve well-stated the concerns you’ve stated.
Nice. I’ll probably rework this comment eventually into a top-level post or something similar; if you jot down some bullet points here of additional concerns to add to the list, I’ll consider incorporating them!
Thanks for synthesizing this, and to Eliezer for researching and explaining the various empirical examples, which I find very helpful (as I did in IABIED).
One thing that I think might be getting lost in conversation, and the startup examples makes clear: I think talking about these problems as “one-chance” is more confusing than is needed.
Talking about irretrievability is one good improvement, but I think irreversibility is also a natural concept here, which I’d like to see more present?
I’d center more the idea that yeah you can try again, but you can’t undo the effects of the previous try, and the accumulation of those effects might make it substantially harder (if not impossible) for you to succeed.
“What do you mean I only get one try at building this startup?” Well, you’re welcome to keep going, but if you’ve depleted your capital you’ll have a hard time getting it back. If you’ve damaged your reputation with investors, customers, etc, it will be hard to wipe the slate clean. The world changed from your previous missteps along the way, as it would if we trained a powerful AI system that turned out to be adversarial to us.
Similarly, yeah France can mount a resistance after Germany has breached their borders, but now France needs to accomplish an even harder task to drive them out.
I apologize if I’m missing these points having been made; I did skim much more aggressively starting a bit into “On the extraordinary efforts put forth to misinterpret the idea of oneshotness.”
This might be the clearest succinct statement of the problem I’ve seen. I hope you’ll make it a top-level post. I don’t think it needs any additions to be highly valuable.
Edits/additional explanation:
I think it’s particularly valuable because it focuses on the practical difficulties with alignment, and these are less-discussed than the technical challenges.
I often see people making good arguments that amount to “there are routes to aligning AGI that will probably work,” and these people seem optimistic. But they haven’t accounted for trying to do that at 80mph, or with a bias toward optimism, or all of the other practical difficulties.
I’ve been thinking of writing a post called something like “even if alignment is easy we’ll probably screw it up disastrously.”
Eliezer and other pessimists do focus on practical difficulties a fair amount. But they seem to mostly get arguments back against the technical difficulties. I think those are a lot easier to debate, so people do. The virtue of this presentation is that it’s short and it gives no technical difficulties to distract from the practical ones.
Oh and—optimism bias and rationalization play a nontrivial role in your statement of the difficulties. I agree that these are pretty big factors. And they’re pretty easy to overlook.
This is a particularly large problem when motivated reasoning (wanting to think I’m working on good things that won’t kill everyone) stacks up with confirmation bias (the previously-justified belief that things turn out okay or better in the long-term and progress is good).
By chance, I just now published a piece you (Daniel) suggested I expand from an older short answer on the most important bias. It expanded into a pretty comprehensive review of the literature, with its impact on the field of AI safety in mind.
It’s here: Motivated reasoning, confirmation bias, and AI risk theory
The bad news: MR and confirmation bias’s total effects are probably large in people who guard against them, and overwhelming in people who do not.
Do you think advances in mechanistic interpretability can meaningfully reduce the probability of a failure during one or several critical tries, for example by detecting scheming, alignment faking, sandbagging, etc. in one or more involved models?
In the historical analogies of irrevocable failures, it seems to be the case that better understanding of one component that caused it could have meaningfully improved chances of success (software update behavior, valve behavior, specific adversarial army capabilities). These were less cursed problems and the component that would have needed more hardening wasn’t known beforehand, but in case anybody would have spent more hardening work on it, the failure could realistically have been prevented (and another failed example would have to be selected here instead).
Yes. Much of my remaining hope lies in various forms of interpretability including mechanistic. It can convert a critical failure into just a regular failure, by catching things going off the rails before it’s too late.
And then they keep going, because otherwise OpenAI will catch up, and then they die. What does mechinterp change about the asymptotic equilibrium as opposed to that particular Tuesday?
Surely there are third parties with authority over the labs who would not permit this scenario to occur? Mechanical Interpretability averting a critical failure is obviously going to bring down the hammer of every regulatory agency in a 10,000 mile radius.
As an example, Mythos is currently being de facto barred from deployment by the US federal government after it demonstrated a hypothetical ability to cause minor amounts of harm. It strains credulity to argue that, after narrowly averting a world-ending catastrophe and with direct evidence of the existence of that risk, the AI labs will simply be permitted to return to business as usual. We have direct experience to say that that’s not how society works.
You presume too much.
I struggle to understand how exactly the simulated CEOs and relevant figures failed to agree upon an international slowdown. I hoped that such a situation would lead Anthropic to broadcast the result. Additionally, I would like you to finally opensource the tabletop exercise’s rules.
Yeah sorry we should publish the ttx rules, should have done that a long time ago, never got around to it because we kept telling ourselves we should clean them up and improve them first.
Perfect as enemy of the good etc; if useful I’m happy to commit some 20 man hours by EA Serbia senior members who I would trust in this and who have experience in either writing or game design to do the clean up and then send to you for review.
Right, another dimension to these scenarios is abortability. At some point, we cross out of technically feasible abortability—we (humans) wouldn’t be able to abort the AI’s growth even if we tried. Whether things are abortable before then depends on how humans react over time / new information (e.g. heeding arguments, heeding warning shots, being credulous about apparent alignment, etc.).
I think that’s not a separate dimension from the “critical” part. I think it’s basically the same thing.
I’m not actually sure exactly what “critical” means here. I’m taking it to just mean “you absolutely must get this try right”. That’s closely connected to abortability, in that if you can abort, it’s not fully lethal / critical yet. I don’t think it’s really the same thing, e.g. you could imagine an LLM-based bacterial package (a more complex “computer virus”) that permanently lives on many computer systems and is basically impossible to abort (short of scouring the planet of all computers with more than 16 GB of memory or whatever).
There’s whether or not you get to try again after your first try, and there’s how late in the game you can decide to not fully do the try at all. There’s at least 3 kinds of outcomes:
You abort (don’t fully do the try).
You do the try and succeed.
You do the try and fail (and can’t try again).
Because unaligned AGI is lethal, you don’t get to try again.
If it’s abortable, it’s not critical. Because you’ll abort it if it starts going bad. If it goes bad so suddenly and silently that you won’t have time to abort it, well, then, it’s not abortable. I don’t think saying “It’s not abortable” is adding anything once we’ve already said that it’s critical.
I very clearly said that in my comment… Anyway, I guess there’s nothing to discuss here, I’m just saying that abortability is a relevant dimension to these scenarios. It’s something that’s brought up often, and also it bears on first-try-ness. If there is a situation that is akin to the eventual critical first try, but is abortable, then that would imply that when you get the eventual critical try, it doesn’t have to be your first try. There’s a nontrivial argument to make about “when it’s abortable, it’s not akin enough to the eventual critical try”.
Are there any techniques that you are thinking about in particular? I haven’t seen any that work super well for the current models, and in general it seems like this problem only gets worse over time, but I could have missed something.