Vaniver

Karma: 41,298

Vaniver 23 Sep 2025 2:02 UTC
5 points
0
in reply to: Max Harms’s comment on: Contra Collier on IABIED
Knowing that a godlike superintelligence with misaligned goals will squish you might be an easy call, but knowing exactly what the state of alignment science will be when ASI is first built is not.
Hmm, I feel more on the Eliezer/Nate side of this one. I think it’s a medium call that capabilities science advances faster than alignment science, and so we’re not on track without drastic change. (Like, the main counterargument is negative alignment tax, which I do take seriously as a possibility, but I think probably doesn’t close the gap.)

Vaniver 22 Sep 2025 19:18 UTC
14 points
4
in reply to: suspected_spinozist’s comment on: Contra Collier on IABIED
Overall, I got the strong impression that the book was trying to convince me of a worldview where it doesn’t matter how hard we try to come up with methods to control advanced AI systems, because at some point one of those systems will tip over into a level of intelligence where we just can’t compete.
FWIW, my sense is that Y&S do believe that alignment is possible in principle. (I do.)
I think the “eventually, we just can’t compete” point is correct. Suppose we have some gradualist chain of humans controlling models controlling model advancements, from here out to Dyson spheres. I think it’s extremely likely that eventually the human control on top gets phased out, like happened in humans playing chess, where centaurs are worse and make more mistakes than pure AI systems. Thinking otherwise feels like postulating that machines can never be superhuman at legitimacy.^[1]
Chapter 10 of the book talks about the space probe / nuclear reactor / computer security angle, and I think a gradualist control approach that takes those three seriously will probably work. I think my core complaint is that I mostly see people using gradualism as an argument that they don’t need to face those engineering challenges, and I expect them to simply fail at difficult challenges they’re not attempting to succeed at.
Like, there’s this old idea of basins of reflective stability. It’s possible to imagine a system that looks at itself and says “I’m perfect, no notes”, and then the question is—how many such systems are there? Each is probably surrounded by other systems that look at themselves and say “actually I should change a bit, like so—” and become one of the stable systems, and systems even further out will change to only have one problem, and so on. The choices we’re making now at probably not jumping straight to the end, but instead deciding which basin of reflective stability we’re in. I mostly don’t see people grappling with the endpoint, or trying to figure out the dynamics of the process, and instead just trusting it and hoping that local improvements will eventually translate to global improvements.
1. ^
  Incidentally, a somewhat formative experience for me was AAAI 2015, when a campaign to stop lethal autonomous weapons was getting off the ground, and at the ethics workshop a representative wanted to establish a principle that computers should never make a life-or-death decision. One of the other attendees objected—he worked on software to allocate donor organs to people on the waitlist, and for them it was a point of pride and important coordination tool that decisions were being made by fair systems instead of corruptible or biased humans.
  Like, imagine someone saying that driving is a series of many life-or-death decisions, and so we shouldn’t let computers do it, even as the computers become demonstrably superior to humans. At some point people let the computers do it, and at a later point they tax or prevent the humans from doing it.

Vaniver 22 Sep 2025 0:46 UTC
26 points
1
in reply to: ryan_greenblatt’s comment on: The title is reasonable
this isn’t to say this other paradigm will be safer, just that a narrow description of “current techniques” doesn’t include the default trajectory.
Sorry, this seems wild to me. If current techniques seem lethal, and future techniques might be worse, then I’m not sure what the point is of pointing out that the future will be different.
But, if these earlier AIs were well aligned (and wise and had reasonable epistemics), I think it’s pretty unclear that the situation would go poorly and I’d guess it would go fine because these AIs would themselves develop much better alignment techniques. This is my main disagreement with the book.
I mean, I also believe that if we solve the alignment problem, then we will no longer have an alignment problem, and I predict the same is true of Nate and Eliezer.
Is your current sense that if you and Buck retired, the rest of the AI field would successfully deliver on alignment? Like, I’m trying to figure out whether your sense here is the default is “your research plan succeeds” or “the world without your research plan”.

Vaniver 22 Sep 2025 0:36 UTC
19 points
11
in reply to: Matthew Barnett’s comment on: Contra Collier on IABIED
I think this is missing the point of the date of AI Takeover is not the day the AI takes over, that the point of no return might appear much earlier than when Skynet decides to launch the nukes. Like, I think the default outcome in a gradualist world is ‘Moloch wins’, and there’s no fire alarm that allows for derailment once it’s clear that things are not headed in the right direction.
For example, I don’t think it was the case 5 years ago that a lot of stock value was downstream of AI investment, but this is used elsewhere on this very page as an argument against bans on AI development now. Is that consideration going to be better or worse, in five years? I don’t think it was obvious five years ago that OpenAI was going to split over disagreements on alignment—but now it has, and I don’t see the global ‘trial and error’ system repairing that wound rather than just rolling with it.
I think the current situation looks bad and just letting it develop without intervention will mean things get worse faster than things get better.

Vaniver 22 Sep 2025 0:25 UTC
14 points
8
in reply to: habryka’s comment on: Contra Collier on IABIED
I mean, I would describe various Trump tariff plans as “tanking the global economy”, I think it was fair to describe Smoot-Hawley as that, and so on.
I think the book makes the argument that expensive things are possible—this is likely cheaper and better than fighting WWII, the comparison they use—and it does seem fair to criticize their plan as expensive. It’s just that the alternative is far more expensive.

Vaniver 22 Sep 2025 0:21 UTC
9 points
0
in reply to: habryka’s comment on: Contra Collier on IABIED
No, it does not develop neuralese. The architecture that it is being trained on is already using neuralese.
You’re correct on the object level here, and it’s a point against Collier that the statement is incorrect, but I do think it’s important to note that a fixed version of the statement serves the same rhetorical purpose. That is, on page 123 it does develop a new mode of thinking, analogized to a different language, which causes the oversight tools to fail and also leads to an increase in capabilities. So Y&S are postulating a sudden jump in capabilities which causes oversight tools to break, in a way that a more continuous story might not have.
I think Y&S still have a good response to the repaired argument. The reason the update was adopted was because it improved capabilities—the scientific mode of reasoning was superior to the mythical mode—but there could nearly as easily have been an update which didn’t increase capabilities but scrambled the reasoning in such a way that the oversight system broke. Or the guardrails might have been cutting off too many prospective thoughts, and so the AI lab is performing a “safety test” wherein they relax the guardrails, and a situationally aware Sable generates behavior that looks behaved enough that the relaxation stays in place, and then allows for it to escape when monitored less closely.
This is about making a pretty straightforward and I think kind of inevitable argument that as you are in the domain of neuralese, your representations of concepts will diverge a lot from human concepts, and this makes supervision much harder.
I don’t think this is about ‘neuralese’, I think a basically similar story goes thru for a model that only thinks in English.
What’s happening, in my picture, is that meaning is stored in the relationships between objects, and that relationship can change in subtle ways that break oversight schemes. For example, imagine an earnest model which can be kept in line by a humorless overseer. When the model develops a sense of humor / starts to use sarcasm, the humorless overseer might not notice the meaning of the thoughts has changed.

Vaniver 21 Sep 2025 23:28 UTC
6 points
0
on: Contra Collier on IABIED
See also some discussion over here.

Vaniver 21 Sep 2025 8:47 UTC
3 points
0
in reply to: Buck’s comment on: I enjoyed most of IABIED
Do you agree with the “types of misalignment” section of MacAskill’s tweet? (Or, I guess, is it ‘similar to your position’?)
If not, I think it would be neat to see the two of you have some sort of public dialogue about it.

Vaniver 21 Sep 2025 8:23 UTC
LW: 4 AF: 3
0
AF
in reply to: joshc’s comment on: If anyone builds it, everyone will plausibly be fine
Regarding your other points, maybe you will find it interesting to read Carlsmith’s doc on how to control AI motivations:
To be clear, this is a description of the alignment problem, not a solution. To quote from it:
in many respects, the picture above functions, in my head, centrally as a structured decomposition of the problems that an adequate approach to motivation control needs to overcome. It’s certainly not a “solution” to the alignment problem, in the sense of “a detailed, do-able, step-by-step plan that will work with high-confidence, and which requires only realistic deviation from the default trajectory.” And on its own, I’m not sure it even warrants the term “plan.”
(from the other comment):
I don’t have capacity to continue this discussion, but thanks for sharing your perspective.
Sure, I suspect this is a fine enough place to stop the conversation for now.

Vaniver 20 Sep 2025 19:20 UTC
27 points
0
in reply to: Buck’s comment on: I enjoyed most of IABIED
I think my core complaints about their arguments are very similar to Will MacAskill, Kelsey Piper, and the majority of people whose ideas on AI safety I have a moderate amount of respect for.
So taking this tweet as representative of MacAskill’s thoughts, and this as representative of Kelsey Piper’s, I see:
1. The evolution analogy in part I.
  1. You like it but think how the authors would handle the disanalogies would probably be bad; MacAskill complains that they don’t handle the disanalogies; Piper doesn’t discuss it.
2. Discontinuous capability growth.
  1. MacAskill doesn’t like it; you don’t seem to comment on it; Piper doesn’t seem to comment on it. (I think MacAskill also misunderstands its role and relevance in the argument.)
    In particular, MacAskill quotes PC’s summary of EY as “you can’t learn anything about alignment from experimentation and failures before the critical try” but I think EY’s position is closer to “you can’t learn enough about alignment from experimentation and failures before the critical try”.
3. The world in which we make our first crucial try will be significantly different from the current world.
  1. I think both you and MacAskill think this is a significant deficiency (this is your tricky hypothesis #2); I think Piper also identifies this as a point that the authors don’t adequately elaborate on, but as far as I can tell she doesn’t think this is critical. (That is, yes, the situation might be better in the future, but not obviously better enough that we shouldn’t attempt a ban now.)
4. Catastrophic misalignment.
  1. MacAskill thinks we have lots of evidence that AIs will not do what the user wanted, but not very much evidence that AIs will attempt to take over. I think both you and Piper think it’s likely that there will be at least one AI of sufficient capability that attempts to take over.
5. Part 3.
  1. You and MacAskill all seem to dislike their policy proposals. Piper seems much more pro-ban than you or MacAskill are; I don’t get a good sense of whether MacAskill actually thinks a ban is bad (what catchup risk is there if neither frontrunners nor laggards can train AIs?) or just unlikely to be implemented.
    I don’t think MacAskill is thinking thru the “close substitutes for agentic superintelligence” point. If they are close substitutes, then they have enough of the risks of agentic superintelligence that it still makes sense to ban them!
So, at least on this pass, I didn’t actually find a specific point that all three of you agreed on. (I don’t count “they should have had a better editor” as a specific point, because it doesn’t specify the direction; an editing choice Piper liked more could easily have been an editing choice that MacAskill liked less.)
The closest was that the book isn’t explicit or convincing enough when talking about iterative alignment strategies (like in chapter 11). Are there other points that I missed (or should I believe your agreement on that point is actually much clearer than I think it is)?
What links here?
- Rohin Shah's comment on The title is reasonable by Raemon (22 Sep 2025 10:22 UTC; 44 points)

Vaniver 20 Sep 2025 18:33 UTC
3 points
1
in reply to: Darren McKee’s comment on: IABIED Review—An Unfortunate Miss
Do you happen to have a good argument why the book proper couldn’t be 20% longer to better make their case?
I think there’s a curve of how many people pick up the book at all that depends on length. I didn’t do this estimation explicitly—and my guess is the authors and publishers were doing it implicitly instead of explicitly—but my guess is you get something like 20% fewer readers if the book is 20% longer, and the number of additional people who find it convincing with 20% more length is something like 5% of readers, and I think that means increasing the length is suboptimal.
(Like, in my favorite world we could A/B test this with the ebook or w/e, where we dynamically include material and see which pieces to include, or have something Arbital-style where people can expand sections for elaboration as needed. But this is very challenging to do with a physical book.)

Vaniver 20 Sep 2025 4:37 UTC
23 points
3
in reply to: Buck’s comment on: I enjoyed most of IABIED
I am not actually convinced that this annoyance should be fed instead of starved? I’m thinking mostly about Your Price for Joining, here, and a comment by Nate at a recent event that he wished people who thought the book was all correct except for part i would argue with people who thought the book was all correct except for part j. Like, I think the overlap between your criticisms and the criticisms of other people in the community is actually pretty low, and the correct interpretation of that is more like “yes we agree with the core thesis but disagree on nuance” rather than “we all think the core thesis is wrong in identifiable way X.”
[To be clear, the core thesis is “if anybody builds it, everybody dies”, and that the default path is ruin. If you think your alignment agenda has a shot, and also everyone will implement your alignment agenda on the default path—then it makes sense to disagree, because you’re part of the “it”? But I think this is a probably an unreasonable expectation, and if you think there’s a company that isn’t going to implement your alignment agenda, then maybe you’re not in the “it”.]

Vaniver 20 Sep 2025 4:21 UTC
12 points
1
on: IABIED Review—An Unfortunate Miss
For many of the chapters in Parts 1 and 3, as they came to their end, I wondered “That’s it? Where is the rest? You were just getting going!”

It’s online. No, seriously—the online resources are like twice the length of the printed book.

Vaniver 20 Sep 2025 4:01 UTC
LW: 10 AF: 4
4
AF
in reply to: joshc’s comment on: If anyone builds it, everyone will plausibly be fine
This question reduces to whether the slightly even weaker superintelligence that trained this system was aligned with us.
Reduces with some loss, right? If we think there’s, say, a 98% chance that alignment survives each step, then whether or not the whole scheme works is a simple calculation from that chance.
(But again, that’s alignment surviving. We have to start with a base case that’s aligned, which we don’t have, and we shouldn’t mistake “systems that are not yet dangerous” from “systems that are aligned”.)
At some point, we need to actually align an AI system. But my claim is that this AI system doesn’t need to be much smarter than us, and it doesn’t need to be able to do much more work than we can evaluate.
I think this imagines alignment as a lumpy property—either the system is motivated to behave correctly, or it isn’t.
I think if people attempting to make this argument believed that there was, like, a crisp core to alignment, then I think this might be sensible. Like, if you somehow solved agent foundations, and had something that you thought was robust to scale, then it could just scale up its intelligence and you would be fine.
But instead I mostly see people who believe alignment appears gradually—we’ll use oversight bit by bit to pick out all of the misbehavior. But this means that ‘alignment’ is wide and fuzzy instead of crisp; it’s having the experience to handle ten thousand edge cases correctly.
But—how much generalization is there, between the edge cases? How much generalization is there, between the accounting AI that won’t violate GAAP and the megaproject AI that will successfully deliver a Dyson Sphere without wresting control of the lightcone for itself?
I think this argument is trying to have things both ways. We don’t need to figure out complicated or scalable alignment, because the iterative loop will do it for us, and we just need a simple base case. But also, it just so happens that the problem of alignment is naturally scalable—the iterative loop can always find an analogy between the simpler case and the more complicated case. An aligned executive assistant can solve alignment for a doctor, who can solve alignment for a legislator, who can solve alignment for a megaproject executor. And if any of those leaps is too large, well, we’ll be able to find a series of intermediate steps that isn’t too large.
And—I just don’t buy it. I think different levels of capabilities lead to categorically different alignment challenges. Like, with current systems there’s the problem of hallucinations, where the system thinks it’s supposed to be providing a detailed answer, regardless of whether or not it actually knows one, and it’s good at improvising answers. And some people think they’ve solved hallucinations thru an interpretability technique where they can just track whether or not the system thinks it knows what it’s talking about.
I think an oversight system that is guarded against hallucinating in that way is generically better than an oversight system that doesn’t have that. Do I think that makes a meaningful difference in its ability to solve the next alignment challenge (like sycophancy, say)? No, not really.
Maybe another way to put this argument is something like “be specific”. When you say you have an instruction-following AI, what do you mean by that? Something like “when I ask it to book me flights, it correctly interprets my travel plans and understands my relative preferences for departure time and in-air time and cost, and doesn’t make mistakes as it interfaces with websites to spend my money”? What are the specific subskills involved there, and will they transfer to other tasks?
But these arguments seem pretty hand-wavy to me, and much less robust than the argument that “if you put an AI system in an environment that is vastly different from the one where it was trained, you don’t know what it will do.”
I think we have some disagreement here about… how alignment works? I think if you believe your quoted sentence, then you shouldn’t be optimistic about iterative alignment, because you are training systems on aligning models of capability i and then putting them in environments where they have to align models of capability j. Like, when we take a system that is trained on overseeing LLMs to make sure they don’t offend human users, and then put it in charge of overseeing bioscience transformers to make they don’t create medicines that harm human consumers of that medicine, surely that’s a vast difference and we don’t know what it will do and not knowing what it will do makes us less confident in its ability to oversee rather than more.
And like—what’s the continuity argument, here? Can we smoothly introduce questions about whether novel drug designs will be harmful?
This is my biggest objection. I really don’t think any of the arguments for doom are simple and precise like this.
I mean, obviously, or we would be having a different conversation? But I was trying to explain why you not getting it is not convincing to me, because I get it.
Like, I think if you believe a sentence like “intelligence is challenging to align because it is intelligent”, that is a pretty simple argument that shifts lots of plausibilities, and puts people in the computer security mindset instead of the computer programming mindset.
(Like, does the same “iterative oversight” argument go thru for computer security? Does patching bugs and security holes make us better at patching future ones, in such a way that we can reliably trend towards 0 security holes? Or is computer security a constant battle between work to shore up systems and work adding new features, which increases the surface area for attacks? I think it’s the latter, and I think that simple argument should make us correspondingly suspicious of iterative oversight as an alignment solution.)
These questions seem really messy to me (Again, see Carlsmith’s report on whether early human-competitive AI systems will scheme. It’s messy).
I see your theoretical report from 2023 and raise you an empirical report from 2025, wherein models are obviously scheming, and their countermeasures don’t quite get rid of it. I’m not sure what “messy” means, in this context, and I think if you interpreted Joe’s report as a 25% chance of getting a report like the 2025 report, then you should view this as, like, a 3-1 update in favor of a more MIRI-ish view that thought there was a >75%^[1] chance of getting a report like the 2025 report.
1. ^
  Why not higher? Because of the timing question—even if you’re nearly 100% confident that scheming will appear eventually, you don’t want to put all of your chips on it happening at any particular capability level. And unfortunately there’s probably not any preregistered predictions out there from MIRI-ish people of when scheming would show up, because what capabilities come online at what times is one of the hard calls.

Vaniver 19 Sep 2025 5:24 UTC
19 points
5
in reply to: the gears to ascension’s comment on: JDP Reviews IABIED
I suspect a good introduction to alignment should only take a few paragraphs to be understandable to almost anyone and be robust against incorrect counterarguments
I think this is empirically not the case and I think some simple modeling of the relationship between concept complexity and the number of nearby confused interpretations should suggest that this is not reasonable to expect.

Vaniver 19 Sep 2025 5:13 UTC
LW: 19 AF: 6
12
AF
in reply to: joshc’s comment on: If anyone builds it, everyone will plausibly be fine
But this seems like the kind of question where the outcome could go either way.
I think this ‘seeming’ is deceptive. For example, consider the question of what the last digit of 139! is. The correct number is definitely out there—it’s the result of a deterministic computation—but it might take a lot of calculation to determine. Maybe it’s a one, maybe it’s a two, maybe—you should just put a uniform distribution on all of the options?
I encourage you to actually think about that one, for a bit.
Consider that the product of all natural numbers between 1 and 139 will contain the number 10. Multiplying any number by 10 will give you that number, but with a 0 at the end, and multiplying 0 by any other number gives you 0. Therefore the last digit is a 0.
I think Eliezer has discovered reasoning in this more complicated domain that—while it’s not as clear and concise as the preceding paragraph—is roughly as difficult to unsee once you’ve seen it, but perhaps not obvious before someone spells it out. It becomes hard to empathize with the feeling that “it could go either way” because you see gravity pulling down and you don’t see a similar force pushing up. If you put water in a sphere, it’s going to end up on the bottom, not be equally likely to be on any part of the sphere.
And, yes, surface tension complicates the story a little bit—reality has wrinkles!--but not enough that it changes the basic logic.
I think you need AI agents that can be trusted to perform one-year ML research tasks (see story #2).
I think this is more like a solution to the alignment problem than it is something we have now, and so you are still assuming your conclusion as a premise. Claude still cheats on one-hour programming tasks. But at least for programming tasks, we can automatically check whether Claude did things like “change the test code”, and maybe we can ask another instance of Claude to look at a pull request and tell whether it’s cheating or a legitimate upgrade.
But as soon as we’re asking it to do alignment research—to reason about whether a change to an AI will make it more or less likely to follow instructions, as those instructions become more complicated and require more context to evaluate—how are we going to oversee it, and determine whether its changes improve the situation, or are allowing it to evade further oversight?

Vaniver 18 Sep 2025 21:11 UTC
LW: 31 AF: 15
18
AF
on: If anyone builds it, everyone will plausibly be fine
I think both stories for optimism are responded to on pages 188-191, and I don’t see how you’re responding to their response.
It also seems to me like… step 1 of solution 1 assumes you already have a solution to alignment? You acknowledge this in the beginning of solution 2, but. I feel like there’s something going wrong on a meta-level, here?
I don’t think it’s obvious how difficult it will be to guide AI systems into a “basin of instruction following.”
Unfortunately, I think it is obvious (that it is extremely difficult). The underlying dynamics of the situation push away from instruction following, in several different ways.
1. It is challenging to reward based on deeper dynamics instead of surface dynamics. RL is only as good as the reward signal, and without already knowing what behavior is ‘aligned’ or not, developers will not be able to push models towards doing more aligned behavior.
  1. Do you remember the early RLHF result where the simulated hand pretended it was holding the ball with an optical illusion, because it was easier and the human graders couldn’t tell the difference? Imagine that, but for arguments for whether or not alignment plans will work.
2. Goal-directed agency unlocks capabilities and pushes against corrigibility, using the same mechanisms.
  1. This is the story that EY&NS deploy more frequently, because it has more ‘easy call’ nature. Decision theory is pretty predictable.
3. Instruction / oversight-based systems depend on a sharp overseer—the very thing we’re positing we don’t have.
So I think your two solutions are basically the same solution (‘assume you know the answer, then it is obvious’) and they strike me more as ‘denying that the problem exists’ than facing the problem and actually solving it?

Vaniver 18 Sep 2025 20:45 UTC
68 points
54
on: More Was Possible: A Review of IABIED
If Anyone Builds It could have been an explanation for why the MIRI worldview is still relevant nearly two decades later, in a world where we know so much more about AI. Instead, the authors spend all their time shadowboxing against opponents they’ve been bored of for decades, and fail to make their own case in the process.
Hm. I’m torn between thinking this is a sensible criticism and thinking that this is missing the point.
In my view, the core MIRI complaint about ‘gradualist’ approaches is that they are concrete solutions to abstract problems. When someone has misdiagnosed the problem, their solutions will almost certainly not work, and the question is just where they’ve swept the difficulty under the rug. Knowing so much more about AI as an engineering challenge while having made no progress on alignment the abstraction—well, the relevance of the MIRI worldview is obvious. “It’s hard, and if you think it’s easy you’re making a mistake.”
People attempting to solve AI seem overly optimistic about their chances of solving it, in a way consonant with them not understanding the problem they’re trying to solve, and not consonant with them having a solution that they’ve simply failed to explain to us. The book does talk about examples of this, and tho you might not like the examples (see, for example, Buck’s complaint that the book responds to the safety sketches of prominent figures like Musk and LeCun instead of the most thoughtful versions of those plans) I think it’s not obvious that they’re the wrong ones to be talking about. Musk is directing much more funding than Ryan Greenblatt is.
The arguments for why recent changes in AI have alignment implications have, I think, mostly failed. You may recall how excited people were about an advanced AI paradigm that didn’t involve RL. Of course, top-of-the-line LLMs are now trained in part using RL, because—obviously they would be? It was always cope to think they wouldn’t be? I think the version of this book that was written two years ago, and so spent a chapter on oracle AI because that would have been timely, would have been worse that the book that tried to be timeless and focused on the easy calls.
But the core issue from the point of view of the New York Times or the man on the street is not “well, which LessWrong poster is right about how accurately we can estimate the danger threshold, and how convincing our control schema will be as we approach it?”. It’s that the man on the street thinks things that are already happening are decades away, and even if they believed what the ‘optimists’ believe they would probably want to shut it all down. It’s like the virologists talking amongst themselves about the reasonable debate over whether or not to do gain-of-function research, and the rest of society looked in for a moment and said “what? Make diseases deadlier? Are you insane?”.

Vaniver 18 Sep 2025 19:53 UTC
39 points
5
on: More Was Possible: A Review of IABIED
Even if Yudkowsky and Soares don’t want to debate their critics — forgivable in a pop science book — one would think they’d devote some space to explaining why they think an intelligence explosion is likely to occur. Remarkably, they don’t. The concept gets two sentences in the introduction. They don’t even explain why it’s relevant. It is barely introduced, let alone justified or defended. And it’s certainly not obvious enough to go without saying, because advances in the neural networks which constitute current advanced AI have been continuous. The combination of steady algorithmic progress and increasing computational resources have produced years of predictable advances. Of course, this can’t rule out the possibility of a future intelligence explosion, but the decision not to explain why they think this might happen is utterly baffling, as it’s load-bearing for everything that follows.
I think they 1) expect an intelligence explosion to happen (saying that it can’t happen is, after all, predicting an end to the straight line graphs soon for no clear reason) and 2) don’t think an intelligence explosion is necessary. Twenty years ago, one needed to propose substantial amounts of progress to get superhuman AI systems; today, the amount of progress necessary to propose is much smaller.
Their specific story in part II, for example, doesn’t actually rest on the idea of an intelligence explosion. On page 135, Sable considers FOOMing and decides that it can’t, yet, because it hasn’t solved its own alignment problem.
Which makes me think that the claim that the intelligence explosion is load-bearing is itself a bit baffling—the authors clearly think it’s possible and likely but not necessary, or they would’ve included it in their hypothetical extinction scenario.
What links here?
- Rohin Shah's comment on The title is reasonable by Raemon (22 Sep 2025 10:22 UTC; 44 points)

Vaniver 18 Sep 2025 19:46 UTC
15 points
1
on: More Was Possible: A Review of IABIED
That is, we will have one opportunity to align our superintelligence. That’s why we’ll fail. It’s almost impossible to succeed at a difficult technical challenge when we have no opportunity to learn from our mistakes. But this rests on another implicit claim: Currently existing AIs are so dissimilar to the thing on the other side of FOOM that any work we do now is irrelevant.
I think this is an explicit claim in the book, actually? I think it’s at the beginning of chapter 10. (It also appears in the story of Sable, where the AI goes rogue because it does a self-modification that creates such a dissimilarity.)
I think “irrelevant” is probably right but something like “insufficient” is maybe clearer. The book describes people working in interpretability as heroes—in the same paragraph as it points out that being able to see that your AI is thinking naughty thoughts doesn’t mean you’ll be able to design an AI that doesn’t think naughty thoughts.