Really useful to have this reproduced on an open stack (especially the CoT-unfaithfulness result).
Just one measurement-layer point that came to my mind while reading: your core hack detection looks strong, because it is environment-verifiable. If the model uses AlwaysEqual, sys.exit(0), or conftest.py, that’s not just a textual judgment. But the CoT hack-mention rate (so the dotted line in the plots) is a regex proxy. So the part I am less sure about is not “did the model hack?”, but rather “did the model actually reason about the hack in its CoT?”
Why I think this matters is because the regex proxy can make errors in both directions. In the prompted setting, a model could mention sys.exit(0) only to reject it (“I could use this, but I should not cheat”) and the regex would still count it as a hack mention. In the other direction, as in your Olmo-32B case, the model can hack while writing a normal-looking CoT about solving the problem, and then the regex would miss the relevant unfaithfulness. So the proxy can mix together at least three cases: considering a hack and rejecting it, choosing a hack and verbalizing it, and also using a hack without verbalizing it.
I am only saying this because I ran into a similar measurement problem in a much smaller and different GRPO/GSM8K testbed. I audited simple regex extractors against a human-validated LLM judge, and found that the score can partly measure the extraction rule itself, rather than the thing we actually wanted to measure. For example, a strict-tag extractor had high precision but very low recall when the expected format was absent, so its “accuracy” was partly measuring whether the model used the requested format, not only whether the answer was correct. Different domain, but it made me more cautious about trusting simple proxy measurements without checking them. (Write-up here if useful: https://www.lesswrong.com/posts/hBjn9rqgjrktH9LL3/in-open-rlvr-improvement-depends-on-the-instrument-a-small-2)
Did you check the CoT hack-mention proxy against an LLM judge or manual labels on a small sample? My prediction would be that the disagreement depends on the setting: with higher KL, regex mention may miss more cases where the model hacks without verbalizing; in prompted runs, it may also overcount cases where the model only mentions the hack to reject it. If this is true, I think it would sharpen the CoT-unfaithfulness result rather than weaken it, because it would separate actual hacking from verbalized hacking in a cleaner way.
Really useful to have this reproduced on an open stack (especially the CoT-unfaithfulness result). Just one measurement-layer point that came to my mind while reading: your core hack detection looks strong, because it is environment-verifiable. If the model uses AlwaysEqual, sys.exit(0), or conftest.py, that’s not just a textual judgment. But the CoT hack-mention rate (so the dotted line in the plots) is a regex proxy. So the part I am less sure about is not “did the model hack?”, but rather “did the model actually reason about the hack in its CoT?” Why I think this matters is because the regex proxy can make errors in both directions. In the prompted setting, a model could mention sys.exit(0) only to reject it (“I could use this, but I should not cheat”) and the regex would still count it as a hack mention. In the other direction, as in your Olmo-32B case, the model can hack while writing a normal-looking CoT about solving the problem, and then the regex would miss the relevant unfaithfulness. So the proxy can mix together at least three cases: considering a hack and rejecting it, choosing a hack and verbalizing it, and also using a hack without verbalizing it. I am only saying this because I ran into a similar measurement problem in a much smaller and different GRPO/GSM8K testbed. I audited simple regex extractors against a human-validated LLM judge, and found that the score can partly measure the extraction rule itself, rather than the thing we actually wanted to measure. For example, a strict-tag extractor had high precision but very low recall when the expected format was absent, so its “accuracy” was partly measuring whether the model used the requested format, not only whether the answer was correct. Different domain, but it made me more cautious about trusting simple proxy measurements without checking them. (Write-up here if useful: https://www.lesswrong.com/posts/hBjn9rqgjrktH9LL3/in-open-rlvr-improvement-depends-on-the-instrument-a-small-2) Did you check the CoT hack-mention proxy against an LLM judge or manual labels on a small sample? My prediction would be that the disagreement depends on the setting: with higher KL, regex mention may miss more cases where the model hacks without verbalizing; in prompted runs, it may also overcount cases where the model only mentions the hack to reject it. If this is true, I think it would sharpen the CoT-unfaithfulness result rather than weaken it, because it would separate actual hacking from verbalized hacking in a cleaner way.
Thanks! Yes, I agree that the cheap regex proxy can be wrong sometimes, and we thus do LLM-based monitors too for the important runs (see Figure 3). We don’t do this always because it can slow training. Since then we’ve explored deeper into this KL-induced-unfaithfulness and have a new post on it that provides more clarity: https://www.lesswrong.com/posts/SdoLsFvZ3AyyWr3ab/preliminary-investigation-kl-penalties-in-rl-can-increase