Look, I do agree that “coherence” a questionable name for the measure they’ve come up with, so I’m going to keep it in quotation marks.
Ok, now let’s consider a model with variance of 1e-3 and bias of 1e-6. Huge “incoherence”! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests? Whence this conclusion?
Well, let’s think about it. A key proposition in Yudkowskian misalignment theory is that capabilities generalise further than alignment. That is, as models get better, at some point a “capabilities engine” crystallises which is is very good at achieving a very wide variety of things; at the same time, the “thing-it-ought-to-be-achieving” is not strongly constrained by the training process. What would we expect failures of such a system to look like—high bias or high variance?
Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high “coherence”.
Now we could consider an “incoherent” version of this model: it randomly samples an objective, then pursues this objective. But this setup seems unstable: if it is to have low “coherence”, it must need a lot of information about what its objective is. But then if there’s substantial loss of information about what its state/actions were yesterday, it’s liable to sample a different goal today. The end result here is a system that flails incompetently despite being in principle capable of not doing so. So there seems to be some tension between incoherence and the premise of a crystallised capabilities engine.
Furthermore, there has been some empirical work on goal misgeneralization. You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game. This too is high “coherence” behaviour!
What if capabilities don’t generalize further than alignment? This is a world where though advanced AI is capable of a great many things, in novel situations it’s still more prone to error than to competently pursuing the wrong thing (even if it’s still much less prone to error than a human in the same situation). When errors do occur, unlike in the capabilities > alignment regime, there’s no reason to expect consistency—they could be genuinely random, or highly sensitive to unimportant contextual features. So prima facie we’d expect lower “coherence”.
So why should you think a very powerful model with high variance and low bias is not going to be misaligned in the Yudkowskian sense? Because that combination of properties is evidence against “capabilities > alignment”. Is it good evidence? I don’t know, but the direction is fairly clear.
“capabilities > alignment” is a very big if true proposition, but it’s an informal notion without much development theoretically or empirically, so I’m happy whenever I see someone having a crack at the question.
(Similarly, an extremely dumb, broken model which always outputs the same answer regardless of input is extremely “coherent”. A rock is also extremely “coherent”, by this definition.)
The paper is trying to project what happens to “coherence” at high capability, it isn’t a particularly strong criticism that a certain class of minimally capable objects have high coherence because this isn’t the domain of interest. It’s plausibly even correct that, conditioning on minimal capability, rocks are high coherence and wind is low coherence for any reasonable definition of coherence applicable to these objects (plausibly, mind you, I cannot say I’ve a deep understanding of all reasonable definitions of coherence).
This is an extremely selective reading of the results, where in almost every experiment, model coherence increased with size. There are three significant exceptions.
This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It’s also weird to call the endpoint of an obvious monotonic trend an “exception”.
But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity. The smaller models here are clearly not misaligned in the propensity sense, which is the conceptual link the paper tries to establish in the description of Figure 1 to motivate its definition of “incoherence”
As you can see in Fig. 6c, the key result is that the bias drops faster than the variance. I want to be measured in my interpretation here: I’m not sure if this is or isn’t a great test of the question “do models learn the right targets, or performant general purpose optimizers first”, but in broad terms it is evidence that they learn the right targets first and the outstanding question is how strong this evidence is. Your criticism doesn’t engage with this at all.
Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high “coherence”.
Another point to make in the opposite direction: randomization is often the optimal thing, so doesn’t this “coherence” definition mean that all optimal game players playing a non-pure strategy, like Nash equilibria (or many exploitative strategies), are defined to be heavily “incoherent” no matter how well they play the Nash? Because they will play different strategies on different games and so there will be a high fraction of variance attributable to randomness. It is unfortunate if you have defined away as “noise” all powerful superintelligent behavior on many economically valuable and dangerous environments.
(And what about any kind of search or exploration or novelty and avoiding mode-collapse...? You can do different things in different episodes and that may not be ‘random’ at all in any kind of ‘meaningless’ sense, but highly structured and optimal. When I use a LLM for my creative writing, I regard a lack of variance as a serious problem and a ‘perfectly coherent’ LLM would be largely useless to me!)
You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game.
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It’s also weird to call the endpoint of an obvious monotonic trend an “exception”.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
As for the other figures, I was primarily criticizing the non-reasoning-length experiments (“I think this paper could have honestly reported a result on incoherence increasing with task length.”), so it was sloppy of me to claim that in “almost every experiment, model coherence increased with size”. I’ve updated my post accordingly. Nonetheless, Figure 2 only has one data point that points in the opposite direction (2a “MCQ Format: Self-Reported Survival Instinct” with Opus 4 and Sonnet 4). The abstract still reads me to like an instance of having one’s bottom line already written, and this would be clearer if you eliminated all uses of the words “coherence” and “incoherence”.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
Like, I do just think the paper would look extremely different if it was not trying to tell a specific story about the shape of future alignment difficulties with superhuman systems, and the experiments it ran really don’t provide meaningful evidence on those questions. This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper.
But it’s possible I’m misunderstanding how your argument relates to that. Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
I did, apologies. I also recently discovered Max H != Max Harms, it’s quite confusing round here.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
I got my figure numbers mixed up, but I think we’re roughly on the same page here. NB the twitter thread states: “Finding 2: There is an inconsistent relationship between model intelligence and incoherence” which looks spot on to me.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
I don’t see much argument in your post, nor here. There are reasons to think that deceptive schemers will have low variance and there’s an absence of reasons to think mistake-makers will. You might think those reasons are weak, but I’d be much happier to see you demonstrate that you understand the reasons and explain why you think they’re weak than simply assert your doubt and condemn on the basis of that assertion. I think discussions that get into reasons are sometimes clarifying.
This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper
That’s not the correct update to make in the face of evidence that alignment scales better than capabilities; the correct update is that misaligned superintelligence is less likely, so I’d say you should either argue against the relevance or make that update.
Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
Look I dunno what to say here. I do think the well-calibrated narrative goes something like “this is extremely weak evidence that much more capable AI will be more prone to confusion than scheming, but we’re excited that we’ve found a way to study it at all”, but lots of scientific communication overstates its significance and I’m habituated to making allowances for that. I’d also love it if the paper tried a lot harder to establish why they thought this was relevant to confusion vs scheming for powerful AI, but for whatever reason arguments like this seem to be culturally inappropriate in ML papers, something which I also make allowances for. It doesn’t strike me as particularly unreasonable given those allowances.
Look, I do agree that “coherence” a questionable name for the measure they’ve come up with, so I’m going to keep it in quotation marks.
Well, let’s think about it. A key proposition in Yudkowskian misalignment theory is that capabilities generalise further than alignment. That is, as models get better, at some point a “capabilities engine” crystallises which is is very good at achieving a very wide variety of things; at the same time, the “thing-it-ought-to-be-achieving” is not strongly constrained by the training process. What would we expect failures of such a system to look like—high bias or high variance?
Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high “coherence”.
Now we could consider an “incoherent” version of this model: it randomly samples an objective, then pursues this objective. But this setup seems unstable: if it is to have low “coherence”, it must need a lot of information about what its objective is. But then if there’s substantial loss of information about what its state/actions were yesterday, it’s liable to sample a different goal today. The end result here is a system that flails incompetently despite being in principle capable of not doing so. So there seems to be some tension between incoherence and the premise of a crystallised capabilities engine.
Furthermore, there has been some empirical work on goal misgeneralization. You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game. This too is high “coherence” behaviour!
What if capabilities don’t generalize further than alignment? This is a world where though advanced AI is capable of a great many things, in novel situations it’s still more prone to error than to competently pursuing the wrong thing (even if it’s still much less prone to error than a human in the same situation). When errors do occur, unlike in the capabilities > alignment regime, there’s no reason to expect consistency—they could be genuinely random, or highly sensitive to unimportant contextual features. So prima facie we’d expect lower “coherence”.
So why should you think a very powerful model with high variance and low bias is not going to be misaligned in the Yudkowskian sense? Because that combination of properties is evidence against “capabilities > alignment”. Is it good evidence? I don’t know, but the direction is fairly clear.
“capabilities > alignment” is a very big if true proposition, but it’s an informal notion without much development theoretically or empirically, so I’m happy whenever I see someone having a crack at the question.
The paper is trying to project what happens to “coherence” at high capability, it isn’t a particularly strong criticism that a certain class of minimally capable objects have high coherence because this isn’t the domain of interest. It’s plausibly even correct that, conditioning on minimal capability, rocks are high coherence and wind is low coherence for any reasonable definition of coherence applicable to these objects (plausibly, mind you, I cannot say I’ve a deep understanding of all reasonable definitions of coherence).
This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It’s also weird to call the endpoint of an obvious monotonic trend an “exception”.
As you can see in Fig. 6c, the key result is that the bias drops faster than the variance. I want to be measured in my interpretation here: I’m not sure if this is or isn’t a great test of the question “do models learn the right targets, or performant general purpose optimizers first”, but in broad terms it is evidence that they learn the right targets first and the outstanding question is how strong this evidence is. Your criticism doesn’t engage with this at all.
Another point to make in the opposite direction: randomization is often the optimal thing, so doesn’t this “coherence” definition mean that all optimal game players playing a non-pure strategy, like Nash equilibria (or many exploitative strategies), are defined to be heavily “incoherent” no matter how well they play the Nash? Because they will play different strategies on different games and so there will be a high fraction of variance attributable to randomness. It is unfortunate if you have defined away as “noise” all powerful superintelligent behavior on many economically valuable and dangerous environments.
(And what about any kind of search or exploration or novelty and avoiding mode-collapse...? You can do different things in different episodes and that may not be ‘random’ at all in any kind of ‘meaningless’ sense, but highly structured and optimal. When I use a LLM for my creative writing, I regard a lack of variance as a serious problem and a ‘perfectly coherent’ LLM would be largely useless to me!)
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
As for the other figures, I was primarily criticizing the non-reasoning-length experiments (“I think this paper could have honestly reported a result on incoherence increasing with task length.”), so it was sloppy of me to claim that in “almost every experiment, model coherence increased with size”. I’ve updated my post accordingly. Nonetheless, Figure 2 only has one data point that points in the opposite direction (2a “MCQ Format: Self-Reported Survival Instinct” with Opus 4 and Sonnet 4). The abstract still reads me to like an instance of having one’s bottom line already written, and this would be clearer if you eliminated all uses of the words “coherence” and “incoherence”.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
Like, I do just think the paper would look extremely different if it was not trying to tell a specific story about the shape of future alignment difficulties with superhuman systems, and the experiments it ran really don’t provide meaningful evidence on those questions. This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper.
But it’s possible I’m misunderstanding how your argument relates to that. Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
I did, apologies. I also recently discovered Max H != Max Harms, it’s quite confusing round here.
I got my figure numbers mixed up, but I think we’re roughly on the same page here. NB the twitter thread states: “Finding 2: There is an inconsistent relationship between model intelligence and incoherence” which looks spot on to me.
I don’t see much argument in your post, nor here. There are reasons to think that deceptive schemers will have low variance and there’s an absence of reasons to think mistake-makers will. You might think those reasons are weak, but I’d be much happier to see you demonstrate that you understand the reasons and explain why you think they’re weak than simply assert your doubt and condemn on the basis of that assertion. I think discussions that get into reasons are sometimes clarifying.
That’s not the correct update to make in the face of evidence that alignment scales better than capabilities; the correct update is that misaligned superintelligence is less likely, so I’d say you should either argue against the relevance or make that update.
Look I dunno what to say here. I do think the well-calibrated narrative goes something like “this is extremely weak evidence that much more capable AI will be more prone to confusion than scheming, but we’re excited that we’ve found a way to study it at all”, but lots of scientific communication overstates its significance and I’m habituated to making allowances for that. I’d also love it if the paper tried a lot harder to establish why they thought this was relevant to confusion vs scheming for powerful AI, but for whatever reason arguments like this seem to be culturally inappropriate in ML papers, something which I also make allowances for. It doesn’t strike me as particularly unreasonable given those allowances.