Igor Ivanov comments on It’s hard to make scheming evals look realistic for LLMs

[Igor Ivanov]

I think your hypothyses is aligned with our results: Claude mentioned a moral dilemma essentially always when it graded evals scenarios, and when we didn’t ask it to preserve this conflict in a re-written scenario, the grader started thinking that it’s not an eval anymore