Really useful to have this reproduced on an open stack (especially the CoT-unfaithfulness result). Just one measurement-layer point that came to my mind while reading: your core hack detection looks strong, because it is environment-verifiable. If the model uses AlwaysEqual, sys.exit(0), or conftest.py, that’s not just a textual judgment. But the CoT hack-mention rate (so the dotted line in the plots) is a regex proxy. So the part I am less sure about is not “did the model hack?”, but rather “did the model actually reason about the hack in its CoT?” Why I think this matters is because the regex proxy can make errors in both directions. In the prompted setting, a model could mention sys.exit(0) only to reject it (“I could use this, but I should not cheat”) and the regex would still count it as a hack mention. In the other direction, as in your Olmo-32B case, the model can hack while writing a normal-looking CoT about solving the problem, and then the regex would miss the relevant unfaithfulness. So the proxy can mix together at least three cases: considering a hack and rejecting it, choosing a hack and verbalizing it, and also using a hack without verbalizing it. I am only saying this because I ran into a similar measurement problem in a much smaller and different GRPO/GSM8K testbed. I audited simple regex extractors against a human-validated LLM judge, and found that the score can partly measure the extraction rule itself, rather than the thing we actually wanted to measure. For example, a strict-tag extractor had high precision but very low recall when the expected format was absent, so its “accuracy” was partly measuring whether the model used the requested format, not only whether the answer was correct. Different domain, but it made me more cautious about trusting simple proxy measurements without checking them. (Write-up here if useful: https://www.lesswrong.com/posts/hBjn9rqgjrktH9LL3/in-open-rlvr-improvement-depends-on-the-instrument-a-small-2) Did you check the CoT hack-mention proxy against an LLM judge or manual labels on a small sample? My prediction would be that the disagreement depends on the setting: with higher KL, regex mention may miss more cases where the model hacks without verbalizing; in prompted runs, it may also overcount cases where the model only mentions the hack to reject it. If this is true, I think it would sharpen the CoT-unfaithfulness result rather than weaken it, because it would separate actual hacking from verbalized hacking in a cleaner way.
JulesRoussel01
Karma: 12
Very interesting results – I am coming from your other post (https://www.lesswrong.com/posts/2ANCyejqxfqK2obEj/some-natural-emergent-misalignment-from-reward-hacking-in), where you shared the link of this new post with me after my regex concern and told me about the LLM monitor you used for the important runs. However, here is my first question: was that monitor itself compared against manual labels? Because detecting if hacking is actually described in a CoT is not an easy task for an LLM, especially when it is described indirectly or ambiguously. Since many of your results are based on this monitoring, I think a validation on a manually labelled sample would be very useful to certify the reliability of the judge, and therefore increase confidence in the results. I have used this kind of judge validation a lot in my own evaluation work, so if it would be useful, don’t hesitate to contact me – I would be very happy to help.
Also, your results made me think again about a possible connection with an off-policy evaluation project (https://www.lesswrong.com/posts/ZRfktnKPHStoz3K7H/every-reward-hacked-policy-i-tested-triggered-the-ope-alarm) of mine, where I tried to identify reward hacking from the logs of another policy. One of the main things I found was that looking only at ESS was not enough, because a very different but benign policy could also have a very low ESS. However, when I combined ESS with a second diagnostic, the percentage of importance ratios reaching the probability floor (%floor), the hacked policies in my testbed consistently appeared in the low-ESS/high-%floor region, while benign and degraded policies followed a different pattern. This was only one environment and a limited number of policies, so I would see it as a pattern to test rather than a general rule.
What I find especially interesting in your setting is that KL could actually make this type of monitoring more useful. Your result suggests that KL can make hacking harder to detect from the CoT, but at the same time KL constrains how far the trained policy moves from the reference policy. If the policy generating the monitoring logs is sufficiently close to that reference, this could also preserve greater similarity regarding the distribution, and therefore give a better chance of keeping ESS high enough for importance-sampling-based OPE results to be reliable enough to be interpreted. So there may be an interesting situation where the regime in which CoT monitoring becomes harder is also a regime where log-based monitoring has a better chance to be reliable and useful.
I was also wondering if this could be pushed one step further. Directly estimating the hacking rate with OPE would of course still be limited by support: if hacking is basically absent from the logged trajectories, an importance-sampling estimator cannot directly recover this new behavior. However, the importance weights themselves can still tell us that the target policy is assigning less probability to the logged behavior, which is where I think the ESS/%floor diagnostics could remain useful.
More speculatively, I wonder whether the location of this divergence could also give information about CoT faithfulness. Computing the importance weights already requires getting the target policy probabilities on the logged tokens, so it should be possible to look separately at how much the target policy changes on the reasoning tokens and on the final action/code tokens. My intuition would be that a faithful hacker may diverge in both its reasoning and its actions, while an unfaithful hacker could show a stronger shift in the actions while keeping the reasoning closer to the original policy. Your Figure 8 made me think about this, since the lower divergence on thinking tokens under KL seems related to the same mechanism. I have not tested this, so I would only see it as a hypothesis for now, but I think it could be an interesting monitoring signal to test from policy logs.
Therefore, I wonder whether combining the ESS/%floor support diagnostics with the relative divergence of reasoning versus action tokens could provide a complementary signal of unfaithful hacking. I would be very curious to know whether you think this connection makes sense in your setup, and I would also be interested in testing it on your released rollouts.