Great, I think the top-right graph posted by Dipika answers my question! So it looks like the answer to “Do models go easier on themselves than others, in the confession-style monitoring format” is a straightforward “Yes”. I’m somewhat surprised! But this is very good to know.
Separately, I’m a little confused why there isn’t much difference between code correctness scores even in the baseline setting. If anything, it looks like GPT-5 gets worse overall scores than GPT-5-nano, when rated by almost all of the other models. Is the dataset (SWEBench it looks like) really that easy? Or are models-as-a-judge just kinda bad and low-signal overall in this setup? (Note, I don’t think this affects the interestingness of your result at all.)
(Also, sorry for the delay in replying @Fabien Roger, my LW notifications were misconfigured!)
Great, I think the top-right graph posted by Dipika answers my question! So it looks like the answer to “Do models go easier on themselves than others, in the confession-style monitoring format” is a straightforward “Yes”. I’m somewhat surprised! But this is very good to know.
Separately, I’m a little confused why there isn’t much difference between code correctness scores even in the baseline setting. If anything, it looks like GPT-5 gets worse overall scores than GPT-5-nano, when rated by almost all of the other models. Is the dataset (SWEBench it looks like) really that easy? Or are models-as-a-judge just kinda bad and low-signal overall in this setup? (Note, I don’t think this affects the interestingness of your result at all.)
(Also, sorry for the delay in replying @Fabien Roger, my LW notifications were misconfigured!)