You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game.
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It’s also weird to call the endpoint of an obvious monotonic trend an “exception”.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
As for the other figures, I was primarily criticizing the non-reasoning-length experiments (“I think this paper could have honestly reported a result on incoherence increasing with task length.”), so it was sloppy of me to claim that in “almost every experiment, model coherence increased with size”. I’ve updated my post accordingly. Nonetheless, Figure 2 only has one data point that points in the opposite direction (2a “MCQ Format: Self-Reported Survival Instinct” with Opus 4 and Sonnet 4). The abstract still reads me to like an instance of having one’s bottom line already written, and this would be clearer if you eliminated all uses of the words “coherence” and “incoherence”.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
Like, I do just think the paper would look extremely different if it was not trying to tell a specific story about the shape of future alignment difficulties with superhuman systems, and the experiments it ran really don’t provide meaningful evidence on those questions. This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper.
But it’s possible I’m misunderstanding how your argument relates to that. Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
I did, apologies. I also recently discovered Max H != Max Harms, it’s quite confusing round here.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
I got my figure numbers mixed up, but I think we’re roughly on the same page here. NB the twitter thread states: “Finding 2: There is an inconsistent relationship between model intelligence and incoherence” which looks spot on to me.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
I don’t see much argument in your post, nor here. There are reasons to think that deceptive schemers will have low variance and there’s an absence of reasons to think mistake-makers will. You might think those reasons are weak, but I’d be much happier to see you demonstrate that you understand the reasons and explain why you think they’re weak than simply assert your doubt and condemn on the basis of that assertion. I think discussions that get into reasons are sometimes clarifying.
This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper
That’s not the correct update to make in the face of evidence that alignment scales better than capabilities; the correct update is that misaligned superintelligence is less likely, so I’d say you should either argue against the relevance or make that update.
Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
Look I dunno what to say here. I do think the well-calibrated narrative goes something like “this is extremely weak evidence that much more capable AI will be more prone to confusion than scheming, but we’re excited that we’ve found a way to study it at all”, but lots of scientific communication overstates its significance and I’m habituated to making allowances for that. I’d also love it if the paper tried a lot harder to establish why they thought this was relevant to confusion vs scheming for powerful AI, but for whatever reason arguments like this seem to be culturally inappropriate in ML papers, something which I also make allowances for. It doesn’t strike me as particularly unreasonable given those allowances.
I think you have me mistaken for my infamous doppelganger, @Robert Miles.
Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.
As for the other figures, I was primarily criticizing the non-reasoning-length experiments (“I think this paper could have honestly reported a result on incoherence increasing with task length.”), so it was sloppy of me to claim that in “almost every experiment, model coherence increased with size”. I’ve updated my post accordingly. Nonetheless, Figure 2 only has one data point that points in the opposite direction (2a “MCQ Format: Self-Reported Survival Instinct” with Opus 4 and Sonnet 4). The abstract still reads me to like an instance of having one’s bottom line already written, and this would be clearer if you eliminated all uses of the words “coherence” and “incoherence”.
As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.
Like, I do just think the paper would look extremely different if it was not trying to tell a specific story about the shape of future alignment difficulties with superhuman systems, and the experiments it ran really don’t provide meaningful evidence on those questions. This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper.
But it’s possible I’m misunderstanding how your argument relates to that. Do you think the framing/narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/narrative?
I did, apologies. I also recently discovered Max H != Max Harms, it’s quite confusing round here.
I got my figure numbers mixed up, but I think we’re roughly on the same page here. NB the twitter thread states: “Finding 2: There is an inconsistent relationship between model intelligence and incoherence” which looks spot on to me.
I don’t see much argument in your post, nor here. There are reasons to think that deceptive schemers will have low variance and there’s an absence of reasons to think mistake-makers will. You might think those reasons are weak, but I’d be much happier to see you demonstrate that you understand the reasons and explain why you think they’re weak than simply assert your doubt and condemn on the basis of that assertion. I think discussions that get into reasons are sometimes clarifying.
That’s not the correct update to make in the face of evidence that alignment scales better than capabilities; the correct update is that misaligned superintelligence is less likely, so I’d say you should either argue against the relevance or make that update.
Look I dunno what to say here. I do think the well-calibrated narrative goes something like “this is extremely weak evidence that much more capable AI will be more prone to confusion than scheming, but we’re excited that we’ve found a way to study it at all”, but lots of scientific communication overstates its significance and I’m habituated to making allowances for that. I’d also love it if the paper tried a lot harder to establish why they thought this was relevant to confusion vs scheming for powerful AI, but for whatever reason arguments like this seem to be culturally inappropriate in ML papers, something which I also make allowances for. It doesn’t strike me as particularly unreasonable given those allowances.