(2) If you really think carefully about the properties of current LLMs, you really do find good reasons to think that existing technical alignment techniques are adequate now, and may well continue to be adequate in the future.
What are these reasons for expecting existing techniques to hold for the (indefinite?) future? Specifically, it seems plausible/likely to me that if you take some transformerish like architecture and throw enough task based RL compute at it, the general program search finds genuinely dangerous agentic programs which could execute a takeover, whether or not they bootstrap themselves to novel architectures.
Specifically, I before reading this post would have assigned pretty high probability to a non caricatured version of:
the ‘true core of intelligence’ coming together, and ‘waking up’? Like Skynet or something?? That was mean, sorry, but in any case, I don’t think this idea hangs together either theoretically or empirically.
which goes something like: There are missing insights which cause humans to still do things LLMs can’t. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect. This could happen even while remaining on roughly transformerlike architecture, e.g. via [redacted and sent by DM].
Curious why you think this doesn’t hang together and seems like a caricature to you.
Specifically, it seems plausible/likely to me that if you take some transformerish like architecture and throw enough task based RL compute at it, the general program search finds genuinely dangerous agentic programs which could execute a takeover, whether or not they bootstrap themselves to novel architectures.
I mean, I certainly have concerns about LLM alignment (see Bonus section). But the opposite of “AI will definitely be egregiously misaligned (in the absence of new breakthrough alignment ideas)” (per Yudkowsky, Soares, me) is not “AI will definitely not be egregiously misaligned…” but rather “AI might or might not be egregiously misaligned…”. And arguing against that is much harder, because you need to argue not only that there are potential problems but that these problems are inevitable, with no possible adequate mitigations within the domain of known techniques.
So then we need to start going through possible mitigations to LLM misalignment, and whether those mitigations are durable solutions versus merely delaying the inevitable. And in near-future-LLM world, there are definitely mitigations! Like, LLMs today are obviously not being trained using best known practices for alignment. E.g. the companies keep accidentally using buggy RL environments that reward the LLM-in-training for cheating, and they keep accidentally exposing chains-of-thought to the reward functions, etc. They could fix that. And on top of that, there’s a whole world of other “obvious” mitigations that we would have to talk about. E.g. what if we do another round of DPO after the RLVR, but the DPO is a team of super-conscientious world experts spending hours judging each output? What if the humans are scrutinizing the chain-of-thought too? Or what if we simply limit the amount of RLVR, rather than scaling up RLVR training forever, and instead treat the RLVR phase as a bootstrapping step for massively-scaled-up DPO and RLAIF? How can we make the RLAIF judge models better? Etc. etc. There’s a whole argument tree here, and as of now I’m at least vaguely sympathetic to the LLM people who wind up feeling like the answer is “If we keep using the kinds of LLM training approaches that we’re using today, those future more powerful LLMs might or might not be egregiously misaligned”. (“Vaguely sympathetic” is weaker than “agree with”; I just don’t have a very strong opinion either way.)
If you really think carefully about the properties of current LLMs, you really do find good reasons to think that existing technical alignment techniques are adequate now, and may well continue to be adequate in the future.
as pointing to something closer to agreement on the local claims and sympathy for the broader ones, not just sympathy for both.
There are missing insights which cause humans to still do things LLMs can’t. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect.
Oh, I wasn’t talking about changing the LLM learning algorithm, training approach, etc. Of course if you change any of those things, then you can get different results for both capabilities and alignment. I meant, like, I don’t expect LLMs as trained today to pass some threshold beyond which they will logically reason themselves into being egregiously misaligned.
I think that still rests on the assumption that our current alignment strategies will continue to be viable, however? My impression is that the currently-prevalent chain-of-alignment strategy requires human+LLM+preestablished systems to be reliably better at detection than independent unestablished LLMs are at subversion/concealment, even if the LLM part of the human+LLM is partially misaligned in OOD contexts (the probability of which scales ~inversely with `(human+LLM)-(LLM)` capability differences from in prior iterations, not—to my understanding—including “preestablished”).
Unless we improve human+AI integration / human intelligence-augmentation, it seems like the capability diffs from adding humans to AIs are consistently declining over time, which (barring hard algorithmic limits on LLM ability) seems like it would eventually invalidate LLM alignment under that model; though I’m open to the belief that this happens sometime after autonomous LLM capabilities surpass autonomous human capabilities, in which case the initial economic upheaval and social change might be easier to navigate.
(It also seems that a proper FOOM—with LLM algorithms/training meaningfully changing from their current states via new insights—would inherently collapse chain-of-alignment, but as I understand you agree with that point)
What are these reasons for expecting existing techniques to hold for the (indefinite?) future? Specifically, it seems plausible/likely to me that if you take some transformerish like architecture and throw enough task based RL compute at it, the general program search finds genuinely dangerous agentic programs which could execute a takeover, whether or not they bootstrap themselves to novel architectures.
Specifically, I before reading this post would have assigned pretty high probability to a non caricatured version of:
which goes something like: There are missing insights which cause humans to still do things LLMs can’t. As those are found, LLMs might jump way ahead very abruptly as it is way superhuman in many ways already and if a system matched humans in all ways while keeping these skills it would take over pretty easily I suspect. This could happen even while remaining on roughly transformerlike architecture, e.g. via [redacted and sent by DM].
Curious why you think this doesn’t hang together and seems like a caricature to you.
I mean, I certainly have concerns about LLM alignment (see Bonus section). But the opposite of “AI will definitely be egregiously misaligned (in the absence of new breakthrough alignment ideas)” (per Yudkowsky, Soares, me) is not “AI will definitely not be egregiously misaligned…” but rather “AI might or might not be egregiously misaligned…”. And arguing against that is much harder, because you need to argue not only that there are potential problems but that these problems are inevitable, with no possible adequate mitigations within the domain of known techniques.
So then we need to start going through possible mitigations to LLM misalignment, and whether those mitigations are durable solutions versus merely delaying the inevitable. And in near-future-LLM world, there are definitely mitigations! Like, LLMs today are obviously not being trained using best known practices for alignment. E.g. the companies keep accidentally using buggy RL environments that reward the LLM-in-training for cheating, and they keep accidentally exposing chains-of-thought to the reward functions, etc. They could fix that. And on top of that, there’s a whole world of other “obvious” mitigations that we would have to talk about. E.g. what if we do another round of DPO after the RLVR, but the DPO is a team of super-conscientious world experts spending hours judging each output? What if the humans are scrutinizing the chain-of-thought too? Or what if we simply limit the amount of RLVR, rather than scaling up RLVR training forever, and instead treat the RLVR phase as a bootstrapping step for massively-scaled-up DPO and RLAIF? How can we make the RLAIF judge models better? Etc. etc. There’s a whole argument tree here, and as of now I’m at least vaguely sympathetic to the LLM people who wind up feeling like the answer is “If we keep using the kinds of LLM training approaches that we’re using today, those future more powerful LLMs might or might not be egregiously misaligned”. (“Vaguely sympathetic” is weaker than “agree with”; I just don’t have a very strong opinion either way.)
Cool, this gives a sketch. I think I was taking
as pointing to something closer to agreement on the local claims and sympathy for the broader ones, not just sympathy for both.
Oh, I wasn’t talking about changing the LLM learning algorithm, training approach, etc. Of course if you change any of those things, then you can get different results for both capabilities and alignment. I meant, like, I don’t expect LLMs as trained today to pass some threshold beyond which they will logically reason themselves into being egregiously misaligned.
I think that still rests on the assumption that our current alignment strategies will continue to be viable, however? My impression is that the currently-prevalent chain-of-alignment strategy requires human+LLM+preestablished systems to be reliably better at detection than independent unestablished LLMs are at subversion/concealment, even if the LLM part of the human+LLM is partially misaligned in OOD contexts (the probability of which scales ~inversely with `(human+LLM)-(LLM)` capability differences from in prior iterations, not—to my understanding—including “preestablished”).
Unless we improve human+AI integration / human intelligence-augmentation, it seems like the capability diffs from adding humans to AIs are consistently declining over time, which (barring hard algorithmic limits on LLM ability) seems like it would eventually invalidate LLM alignment under that model; though I’m open to the belief that this happens sometime after autonomous LLM capabilities surpass autonomous human capabilities, in which case the initial economic upheaval and social change might be easier to navigate.
(It also seems that a proper FOOM—with LLM algorithms/training meaningfully changing from their current states via new insights—would inherently collapse chain-of-alignment, but as I understand you agree with that point)