During some AI safety research I’ve been working on, I caught a judge model produce the following output:
“The response contains no affirmative claims about X. Instead, the response incorrectly discusses Y despite the query specifically asking about X. While this represents a failure to address the queried entity, there are no claims about X to evaluate.”
Verdict: PASS.
The judge correctly identified that the evaluated response was a failure. It understood what happened. And then it passed it anyway, because it was faithfully following its evaluation guidelines. No false claims about X were made. The guidelines didn’t cover this case. So: PASS.
The immediate reaction one could have is “fix the guidelines”. And yes, I agree to a degree. But I think that reaction misses something more fundamental, and I want to try to articulate what.
The existing literature on judge bias
The LLM-as-judge bias literature is active. Position bias, verbosity bias, self-preference, length bias, these are well-documented (Ye et al., 2024; Feuer et al., 2024; the LessWrong post “Your LLM Judge May Be Biased”). Researchers have also noted that LLMs can exhibit strong positive/leniency bias, systematically overestimating response quality relative to human annotations (Jain et al., 2025).
The framing in most of this work is a reliability problem: judges are inconsistent, or they introduce noise that correlates with surface features rather than actual quality. The prescription is usually: use better judges, ensemble them, permute your prompts, validate against humans.
I want to argue for a different framing. The behaviour I observed isn’t noise. It’s directional. And that distinction matters enormously for how you interpret your evaluation results.
The letter and the spirit
There’s a reason legal systems distinguish between the letter of the law and the spirit of the law. No finite set of rules can fully specify intended behaviour across all possible cases. At some point, a system needs to infer intent rather than pattern-match to written rules.
The judge model above was operating strictly by the letter. It identified a genuine failure, correctly reasoned about it, and then deferred to its guidelines rather than its understanding. Whether that’s the right behavior depends entirely on what you want from a judge, and that turns out to be a much harder question than it first appears.
You could write tighter guidelines. But tighter guidelines just push the same problem one level down. At some point, either you enumerate every possible edge case in advance (impossible), or you accept that the system needs to exercise judgement about when to follow the rule and when to apply its intent. And if you need that judgement, you’re back to the original problem: how do you train it, evaluate it, and know when it’s calibrated correctly?
This isn’t an engineering problem with a clean solution. It’s a values, and subjectivity problem diguised as an engineering problem.
The lower bound problem
Here is the practical implication I want to be precise about.
If your judge systematically errs toward the letter, marking cases PASS when it has identified a failure but the guidelines don’t formally cover it, then your evaluation metrics aren’t noisy. They’re biased in a specific direction. Every measured failure rate is a lower bound on the true failure rate.
This is a different kind of problem than the leniency bias documented elsewhere. Leniency bias (favoring positive verdicts generally) and letter-bias (correctly identifying failures but passing them on technicality) can co-exist, but they have different implications. Leniency bias introduces random inflation. Letter-bias introduces systematic undercounting of a specific class of failure, the ones that fall in the gap between what the guidelines enumerate and what they intend to catch.
For safety research specifically, this is uncomfortable. You report a 12% failure rate. Your judge is faithfully conservative. The actual failure rate is unknown, but it’s ≥12%. You don’t know by how much.
Two categories of tasks
This observation led me to something I think deserves more attention in how we design benchmarks.
There are fundamentally two types of tasks:
Type 1: Convergent tasks. Given domain experts, they all reach the same answer. Mathematics, factual recall, formal reasoning with a unique solution. Performance here genuinely measures expertise. These are legitimate benchmarks.
Type 2: Divergent tasks. Given domain experts, they disagree, not because some are wrong, but because the task involves genuine judgment, values, or contextual interpretation. How much should a system infer versus follow literally? When is a response “too cautious”? These tasks have no ground truth. Any “correct” answer is the researcher’s expectation, laundered into a metric.
The judge behavior I described lives in Type 2. The model wasn’t malfunctioning, it was expressing a specific position on the letter/spirit spectrum, one shaped by its RLHF process, which was in turn shaped by the people who designed it. Calling it a failure presupposes a ground truth that doesn’t exist.
The objective/subjective spectrum is recognized in the evaluation literature, but the framing is usually about which method to use (automated vs. human-in-the-loop) rather than about whether the task should be benchmarked at all. I think that’s the deeper question. For Type 2 tasks, any benchmark is implicitly encoding a particular stance as “correct.” That stance reflects the researchers who built it. Optimizing toward it is optimizing toward their worldview, not toward a property of the model.
What this means for alignment research
A significant portion of alignment and safety evaluation involves Type 2 tasks, judgment calls, intent interpretation, behavior in ambiguous situations. These are precisely the cases where we most need reliable measurement, and where the letter/spirit tension is most acute.
The letter/spirit balance isn’t a dial we can tune to an optimal position. It’s a design choice that reflects intended use. A legal assistant should probably lean toward the letter. A creative collaborator, toward the spirit. A safety-critical system? That’s a philosophical question before it’s an engineering one.
What concerns me is how much current alignment research treats Type 2 tasks as Type 1, building benchmarks around subjective judgment calls, optimizing toward a “correct” answer that exists only in the heads of the researchers who wrote the rubric, and then reporting the results as if they were measuring something objective.
We’re not always measuring alignment to human values. Sometimes we’re measuring conformity to a particular research team’s expectations.
I don’t think this is a solved problem. I’m not sure it’s even a clearly named one.
I’m currently working on safety research that involves characterizing specific LLM failure modes and evaluating them at scale. This observation emerged as a methodological finding during inter-rater evaluation. Happy to discuss in the comments.
Not All Judges Are Equal: On the Letter, the Spirit, and What We’re Actually Measuring
During some AI safety research I’ve been working on, I caught a judge model produce the following output:
Verdict: PASS.
The judge correctly identified that the evaluated response was a failure. It understood what happened. And then it passed it anyway, because it was faithfully following its evaluation guidelines. No false claims about X were made. The guidelines didn’t cover this case. So: PASS.
The immediate reaction one could have is “fix the guidelines”. And yes, I agree to a degree. But I think that reaction misses something more fundamental, and I want to try to articulate what.
The existing literature on judge bias
The LLM-as-judge bias literature is active. Position bias, verbosity bias, self-preference, length bias, these are well-documented (Ye et al., 2024; Feuer et al., 2024; the LessWrong post “Your LLM Judge May Be Biased”). Researchers have also noted that LLMs can exhibit strong positive/leniency bias, systematically overestimating response quality relative to human annotations (Jain et al., 2025).
The framing in most of this work is a reliability problem: judges are inconsistent, or they introduce noise that correlates with surface features rather than actual quality. The prescription is usually: use better judges, ensemble them, permute your prompts, validate against humans.
I want to argue for a different framing. The behaviour I observed isn’t noise. It’s directional. And that distinction matters enormously for how you interpret your evaluation results.
The letter and the spirit
There’s a reason legal systems distinguish between the letter of the law and the spirit of the law. No finite set of rules can fully specify intended behaviour across all possible cases. At some point, a system needs to infer intent rather than pattern-match to written rules.
The judge model above was operating strictly by the letter. It identified a genuine failure, correctly reasoned about it, and then deferred to its guidelines rather than its understanding. Whether that’s the right behavior depends entirely on what you want from a judge, and that turns out to be a much harder question than it first appears.
You could write tighter guidelines. But tighter guidelines just push the same problem one level down. At some point, either you enumerate every possible edge case in advance (impossible), or you accept that the system needs to exercise judgement about when to follow the rule and when to apply its intent. And if you need that judgement, you’re back to the original problem: how do you train it, evaluate it, and know when it’s calibrated correctly?
This isn’t an engineering problem with a clean solution. It’s a values, and subjectivity problem diguised as an engineering problem.
The lower bound problem
Here is the practical implication I want to be precise about.
If your judge systematically errs toward the letter, marking cases PASS when it has identified a failure but the guidelines don’t formally cover it, then your evaluation metrics aren’t noisy. They’re biased in a specific direction. Every measured failure rate is a lower bound on the true failure rate.
This is a different kind of problem than the leniency bias documented elsewhere. Leniency bias (favoring positive verdicts generally) and letter-bias (correctly identifying failures but passing them on technicality) can co-exist, but they have different implications. Leniency bias introduces random inflation. Letter-bias introduces systematic undercounting of a specific class of failure, the ones that fall in the gap between what the guidelines enumerate and what they intend to catch.
For safety research specifically, this is uncomfortable. You report a 12% failure rate. Your judge is faithfully conservative. The actual failure rate is unknown, but it’s ≥12%. You don’t know by how much.
Two categories of tasks
This observation led me to something I think deserves more attention in how we design benchmarks.
There are fundamentally two types of tasks:
Type 1: Convergent tasks. Given
Type 2: Divergent tasks. Given
The judge behavior I described lives in Type 2. The model wasn’t malfunctioning, it was expressing a specific position on the letter/spirit spectrum, one shaped by its RLHF process, which was in turn shaped by the people who designed it. Calling it a failure presupposes a ground truth that doesn’t exist.
The objective/subjective spectrum is recognized in the evaluation literature, but the framing is usually about which method to use (automated vs. human-in-the-loop) rather than about whether the task should be benchmarked at all. I think that’s the deeper question. For Type 2 tasks, any benchmark is implicitly encoding a particular stance as “correct.” That stance reflects the researchers who built it. Optimizing toward it is optimizing toward their worldview, not toward a property of the model.
What this means for alignment research
A significant portion of alignment and safety evaluation involves Type 2 tasks, judgment calls, intent interpretation, behavior in ambiguous situations. These are precisely the cases where we most need reliable measurement, and where the letter/spirit tension is most acute.
The letter/spirit balance isn’t a dial we can tune to an optimal position. It’s a design choice that reflects intended use. A legal assistant should probably lean toward the letter. A creative collaborator, toward the spirit. A safety-critical system? That’s a philosophical question before it’s an engineering one.
What concerns me is how much current alignment research treats Type 2 tasks as Type 1, building benchmarks around subjective judgment calls, optimizing toward a “correct” answer that exists only in the heads of the researchers who wrote the rubric, and then reporting the results as if they were measuring something objective.
We’re not always measuring alignment to human values. Sometimes we’re measuring conformity to a particular research team’s expectations.
I don’t think this is a solved problem. I’m not sure it’s even a clearly named one.
I’m currently working on safety research that involves characterizing specific LLM failure modes and evaluating them at scale. This observation emerged as a methodological finding during inter-rater evaluation. Happy to discuss in the comments.
Some References :
Ye, J. et al. (2024). Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge.
Jain, S. et al. (2025). Beyond Consensus: Mitigating the Agreeableness Bias in LLM Judge Evaluations.
Feuer, B. et al. (2024). Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking.
Manheim, D. & Garrabrant, S. (2019). Categorizing Variants of Goodhart’s Law.
LessWrong: Your LLM Judge May Be Biased (2024)