First, the authors did a great job as reviewers, diving deep into the data and detecting strange patterns. They found bugs that made the RLHF setup unrealistic, so instead of supporting the strong claim “RLHF by default misleads users”, the result became the much weaker “if you implement buggy RLHF, you might get a misleading model”. The irony is that a paper about misleading behaviour may itself give readers a misleading picture.
The original paper says:
“Denison et al. (2024) [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models] uses brittle rewards designed to be hacked. In contrast, we study U-SOPHISTRY that naturally emerges from standard, innocuous practices [...].”
But the bugs place the misleading-RLHF paper in the same category as model-organism studies with unnatural training setups.
2024,Lies, Damned Lies, and Distributional Language Statistics: “Such a mismatch between the intended goal (helpfulness) and the rewarded property (likeability) could be doubly harmful: not only are models not being correctly optimised for producing helpful responses for the user, they are in fact being directly optimised for deceiving the user into believing they are being helped (Wen et al., 2024).”
ICLR 2025,On Targeted Manipulation and Deception: “Wen et al. (2024) found that RLHF can lead LLMs to mislead external annotators into providing incorrect feedback on factual and coding questions.”
They thought the paper had a natural setup.
The second reason for the upvote is that I was interested in reading the first author’s response. Bugs suck, but they happen—that’s fine. However, J. Wen seems to deny that the setup was flawed and considers it intentional given “the model organism nature of the work”.
However, neither I when reading the original paper, nor the authors of the citing papers understood it as a model-organism study. The original paper is very straightforward in offering its setup as natural. I agree with J. Wen that they showed how unintentionally buggy RLHF might mislead users, but these conditions became clear only after the authors of this post tried to reproduce the result.
J. Wen says that their setup was realistic, but it does not seem realistic to me. To argue for its realism, he cites two papers, but:
Those papers do not themselves use realistic RLHF setups.
They use a different setup in which the judges have access to the stories.
It seems to me that the author does not want to admit that the setup was flawed. Good to know and I’ll be more cautious reading his other papers.
I appreciate your diligence and would love to see more similar reproductions! However, I did not understand all the ablations (eg figure D says “1+3”, but from the text I think you meant “2+3”?). I also asked Fable 5 to look for bugs in the original paper’s repo, and it found truncated inputs and a few possible bugs in the PPO implementation, though I haven’t checked them.
Strong upvote for two reasons.
First, the authors did a great job as reviewers, diving deep into the data and detecting strange patterns. They found bugs that made the RLHF setup unrealistic, so instead of supporting the strong claim “RLHF by default misleads users”, the result became the much weaker “if you implement buggy RLHF, you might get a misleading model”. The irony is that a paper about misleading behaviour may itself give readers a misleading picture.
The original paper says:
But the bugs place the misleading-RLHF paper in the same category as model-organism studies with unnatural training setups.
There were other papers about problems with RLHF (Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback), but this paper is often cited as a natural example of misalignment. Here, check these citations:
2026, Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction: “[...] by Wen et al. (2024), where deceptive behaviors are demonstrated to naturally emerge from applying the canonical RLHF training pipeline.”
2025, Observation Interference in Partially Observable Assistance Games: “Wen et al. (2024) [...] provide experimental evidence that optimizing for human feedback teaches language models to mislead humans.”
2024, Lies, Damned Lies, and Distributional Language Statistics: “Such a mismatch between the intended goal (helpfulness) and the rewarded property (likeability) could be doubly harmful: not only are models not being correctly optimised for producing helpful responses for the user, they are in fact being directly optimised for deceiving the user into believing they are being helped (Wen et al., 2024).”
ICLR 2025, On Targeted Manipulation and Deception: “Wen et al. (2024) found that RLHF can lead LLMs to mislead external annotators into providing incorrect feedback on factual and coding questions.”
They thought the paper had a natural setup.
The second reason for the upvote is that I was interested in reading the first author’s response. Bugs suck, but they happen—that’s fine. However, J. Wen seems to deny that the setup was flawed and considers it intentional given “the model organism nature of the work”.
However, neither I when reading the original paper, nor the authors of the citing papers understood it as a model-organism study. The original paper is very straightforward in offering its setup as natural. I agree with J. Wen that they showed how unintentionally buggy RLHF might mislead users, but these conditions became clear only after the authors of this post tried to reproduce the result.
J. Wen says that their setup was realistic, but it does not seem realistic to me. To argue for its realism, he cites two papers, but:
Those papers do not themselves use realistic RLHF setups.
They use a different setup in which the judges have access to the stories.
It seems to me that the author does not want to admit that the setup was flawed. Good to know and I’ll be more cautious reading his other papers.
I appreciate your diligence and would love to see more similar reproductions! However, I did not understand all the ablations (eg figure D says “1+3”, but from the text I think you meant “2+3”?). I also asked Fable 5 to look for bugs in the original paper’s repo, and it found truncated inputs and a few possible bugs in the PPO implementation, though I haven’t checked them.