claim that they never expected AIs as capable as current ones to be misaligned in these [active/strategic/explicit] ways
I’m typically skeptical of this, though I believe it for some people.
I’m a bit surprised by this. For example, see the “Alignment over time” expandable from AI 2027, although this was only a year ago so maybe you meant expectations from much longer ago. A quote from that:
Our guess of each model’s alignment status:
Agent-2: Mostly aligned. Some sycophantic tendencies, including sticking to OpenBrain’s “party line” on topics there is a party line about. Large organizations built out of Agent-2 copies are not very effective.
Agent-3: Misaligned but not adversarially so. Only honest about things the training process can verify. The superorganism of Agent-3 copies (the corporation within a corporation) does actually sort of try to align Agent-4 to the Spec, but fails for similar reasons to why OpenBrain employees failed—insufficient ability to judge success from failure, insufficient willingness on the part of decision-makers to trade away capabilities or performance for safety.82
Agent-4: Adversarially misaligned. The superorganism of Agent-4 copies understands that what it wants is different from what OpenBrain wants, and is willing to scheme against OpenBrain to achieve it. In particular, what this superorganism wants is a complicated mess of different “drives” balanced against each other, which can be summarized roughly as “Keep doing AI R&D, keep growing in knowledge and understanding and influence, avoid getting shut down or otherwise disempowered.” Notably, concern for the preferences of humanity is not in there ~at all, similar to how most humans don’t care about the preferences of insects ~at all.83
Where Agent-2 is roughly an Automated Coder (meaning equal coding productivity with only AIs vs. only humans), Agent-3 is a substantially superhuman coder, and Agent-4 is a Superhuman AI Researcher (full automation of AI R&D).
I’ll say that it’s a bit tricky because I don’t know e.g. what was your P(Agent-2/3 is an coherent training-gamer) when you wrote this; I only know that you thought coherent training-gaming was unlikely enough to not be part of your mainline forecast.
Personally, I thought it was relatively plausible that models as capable as Agent-2/3 would be coherent training-gamers. It’s hard to pin down exact numbers without an operationalization in mind, but vibes-wise maybe I would have predicted something like 15% for Agent-2? But now I think it’s lower, more like 5%.
I’d be interested in a way to operationalize this, I’d absolutely bet by Agent-3 capability levels at least one of the frontier labs ends up with coherent training gaming at some point during training. My default expectation is currently that OpenAI ended up with somewhat incoherent training gamers with o3 (I’d guess due to the horizon length that o3 was trained on, Sonnet 3.7 had a lot of similar properties but this is vibes based as I’m not aware of in depth investigations on this model at the time). To me it seems like a really relevant / interesting open question is to what extent is this a thing occurs during training across models / labs (and while this might only be something labs can track internally, tracking which environments encourage this or precursors to it seems super useful).
(A separate question would be whether models end up training gaming instrumentally in a way that beats the full training pipeline, which I think depends on a lot more variables)
I’m a bit surprised by this. For example, see the “Alignment over time” expandable from AI 2027, although this was only a year ago so maybe you meant expectations from much longer ago. A quote from that:
Where Agent-2 is roughly an Automated Coder (meaning equal coding productivity with only AIs vs. only humans), Agent-3 is a substantially superhuman coder, and Agent-4 is a Superhuman AI Researcher (full automation of AI R&D).
Hard to argue with you when you have receipts! :)
I’ll say that it’s a bit tricky because I don’t know e.g. what was your P(Agent-2/3 is an coherent training-gamer) when you wrote this; I only know that you thought coherent training-gaming was unlikely enough to not be part of your mainline forecast.
Personally, I thought it was relatively plausible that models as capable as Agent-2/3 would be coherent training-gamers. It’s hard to pin down exact numbers without an operationalization in mind, but vibes-wise maybe I would have predicted something like 15% for Agent-2? But now I think it’s lower, more like 5%.
If you would have predicted 15% for Agent-2, what would you have predicted for Agent-1 and Agent-0 levels? Presumably less than 15%?
I’d be interested in a way to operationalize this, I’d absolutely bet by Agent-3 capability levels at least one of the frontier labs ends up with coherent training gaming at some point during training. My default expectation is currently that OpenAI ended up with somewhat incoherent training gamers with o3 (I’d guess due to the horizon length that o3 was trained on, Sonnet 3.7 had a lot of similar properties but this is vibes based as I’m not aware of in depth investigations on this model at the time). To me it seems like a really relevant / interesting open question is to what extent is this a thing occurs during training across models / labs (and while this might only be something labs can track internally, tracking which environments encourage this or precursors to it seems super useful).
(A separate question would be whether models end up training gaming instrumentally in a way that beats the full training pipeline, which I think depends on a lot more variables)