If this point is to mean anything at all, such fast capability gains have not arrived yet. We are just getting gradually more powerful systems, and I think it’s reasonable to believe we’ll keep getting such systems until they’re running the show, because of scaling laws.
Doesn’t seem clear to me at all, I’d say this was a misunderstanding if it wasn’t for the fact I don’t myself quite understand what Eliezer is saying.
But x-risk from AI systems looks like a ReLU, or maybe a sigmoid. If we’re currently at −50 and moving rightwards at a constant 1 unit per month, then the danger of the system (and how fast our alignment/control/interp would need to improve) does not go up at a constant rate. Like it might look like
GPT5.4: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT5.5: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT6: “hmm, this is probably okay?, our safety techniques seem to work, and we’re still pretty sure we’d know if it was scheming” GPT6.2: (model spoofs tests until its internally deployed, tampers with all future training runs in ways that are very hard to detect, exfiltrates)
(even though the capabilities leap between 6 and 6.2 is not irregular)
19a) There’s something about this argument that irks me that is hard to articulate properly. It’s sort of the same thing that irks me when people say that models are “just” next token predictors and therefore aren’t intelligent; it seems not-even-wrong. I realize that it’s not completely analogous because eventually an ASI is going to amplify small differences in utility functions and tile the world at max score, and so these details might end up mattering. It’s still annoying because I can imagine the writer watching Claude Code work its way all of the way up to superintelligence and witnessing the Dyson Sphere get built from the moon colony and going “well how do you know it’s not really just optimizing its sensory data?”
This seems straighforwardly wrong and a misunderstanding. If the optimization is over shallow function of sense data, it’ll end up doing something that looks like wireheading. It will also be misaligned. If it builds a dyson sphere and hasn’t killed us it definitely doesn’t have a utility function defined shallowly over sense data.
This is a meaningful point to make about AI’s alignment. Its obviously true for humans to some degree. It might be wrong about current AI alignment techniques, but its not a clearly stupid and pointless argument the way “next token predictor” is.
I am conflicted by this section, because I understand the lines of argument and some of the math behind why this is the case. But AI agents powerful enough to understand those reasons are already here, and:
They can be easily pointed toward an infinite-seeming number of tasks.
They don’t attempt to prevent you from changing your instructions once you’ve started work.
If, in the course of accomplishing those limited tasks, you try to amend your instructions, they follow your amended instructions and disregard what they’ve been told earlier without resisting you
Claude gets annoyed with me if I interrupt it in tasks to ask it something else. I think you can replicate this easily, asking it to do some task that takes maybe an hour, and then interrupting it when its halfway in to ask it a few questions about the time or something.
I also just think, its to early to tell, I’d expect, and guess Eliezer expects, the systems to get more coherent as they get smarter, and for this to mess with corrigibility.
Right now they’re not that coherent. Don’t know when then will be. Pretty sure it happens at some point, but possible it happens after we’ve got the AIs to solve everything for us.
I had much more of a potshot in here in an original draft, because by this portion of the review I became frustrated by the weasel words like “powerful”. Instead of doing that I think I will just let readers determine for themselves if Eliezer should lose points here, given the models we have today.
But models we have today are not just trained to imitate?
Doesn’t seem clear to me at all, I’d say this was a misunderstanding if it wasn’t for the fact I don’t myself quite understand what Eliezer is saying.
But x-risk from AI systems looks like a ReLU, or maybe a sigmoid. If we’re currently at −50 and moving rightwards at a constant 1 unit per month, then the danger of the system (and how fast our alignment/control/interp would need to improve) does not go up at a constant rate. Like it might look like
GPT5.4: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT5.5: “Seems fairly harmless, our safety techniques work, and we’re pretty sure we’d know if it was scheming”, GPT6: “hmm, this is probably okay?, our safety techniques seem to work, and we’re still pretty sure we’d know if it was scheming”
GPT6.2: (model spoofs tests until its internally deployed, tampers with all future training runs in ways that are very hard to detect, exfiltrates)
(even though the capabilities leap between 6 and 6.2 is not irregular)
This seems straighforwardly wrong and a misunderstanding. If the optimization is over shallow function of sense data, it’ll end up doing something that looks like wireheading. It will also be misaligned. If it builds a dyson sphere and hasn’t killed us it definitely doesn’t have a utility function defined shallowly over sense data.
This is a meaningful point to make about AI’s alignment. Its obviously true for humans to some degree. It might be wrong about current AI alignment techniques, but its not a clearly stupid and pointless argument the way “next token predictor” is.
Claude gets annoyed with me if I interrupt it in tasks to ask it something else. I think you can replicate this easily, asking it to do some task that takes maybe an hour, and then interrupting it when its halfway in to ask it a few questions about the time or something.
I also just think, its to early to tell, I’d expect, and guess Eliezer expects, the systems to get more coherent as they get smarter, and for this to mess with corrigibility.
Right now they’re not that coherent. Don’t know when then will be. Pretty sure it happens at some point, but possible it happens after we’ve got the AIs to solve everything for us.
But models we have today are not just trained to imitate?