Epistemic status: Professional observation from a decade in enterprise internal audit. I explore whether institutional containment patterns could transfer to AI oversight. I’m not claiming expertise in alignment theory.
TL;DR: LLM-as-a-judge has the same structural weakness as asking a trader’s desk-mate to review their risk positions. Enterprise audit frameworks solve this with segregation of duties, evidence-grounded evaluation, and fail-closed defaults. I built an open-source pipeline that enforces these principles on LLM evaluator outputs, and it catches real issues.
I started from first principles, reading “The Lens That Sees Its Flaws”, where Yudkowsky argues that humanity’s advantage over mice is our ability to recognise our own cognitive distortions and apply reflective corrections. Science, rationality, the human endeavor—it’s the lens examining itself.
I’d like to offer a perspective that perhaps differs (or maybe augments) this idea. I am curious whether the community sees this as a contradiction, or simply a “yes, and.”
I spent a decade in enterprise risk management at global banks, an environment that has evolved to survive decades of adversarial pressure from intelligent agents. In this situation, we don’t ask the lens to see its own flaws. We also don’t assume the lens is acting with adversarial intent. But we also don’t rely on it being accurate. We operate from a principle of “trust, but verify”
A classic example is a rogue trader. They are highly capable, non-deterministic agents optimising for a proxy reward (i.e. their bonus) that might diverge drastically from the principal’s goal (i.e. the bank’s survival). They don’t usually break rules directly, but they are perfectly capable of finding loopholes, exploiting edge cases, and exceeding the spirit of constraints while satisfying the letter.
Governance, Risk, and Compliance (GRC) functions don’t manage this by training the trader to be more self-aware. Instead, we build structural transparency around them. Segregation of duties (the person who initiates a transaction cannot approve it), evidence-based verification, and fail-closed defaults (i.e when something doesn’t add up, the system halts and escalates rather than waving it through).
The final step is the completely independent ‘Third Line of Defence’ (Internal Audit). It verifies that both the business (first line) and the compliance/risk teams (second line) have done their jobs. It doesn’t assume bad faith, but it doesn’t require good faith either. It requires evidence, but it also requires independent logical evaluation.
When an auditor catches a problem ‘live’, our first thought isn’t bad faith. It’s usually just a broken control, a process gap, or somebody making a very everyday mistake.
That attitude matters, because people cooperate with oversight they perceive as fair. We are all on the same team; if the bank goes bust tomorrow, nobody gets paid.
I see the discussions in the AI safety community about LLM-as-a-judge, and the very real problems of using one model to evaluate another’s outputs.
This looks structurally identical to asking a trader’s desk-mate to review their risk positions based on professional judgment. The problem isn’t that the judge model is incapable. It’s that there is no segregation of duties, no requirement to ground evaluations to specific evidence, and no fail-closed default when the logic breaks.
To test whether these institutional GRC patterns actually transfer into AI safety, I built and open-sourced a small audit pipeline (Correspondence Auditor) that enforces these principles by evaluating LLM-as-a-judge outputs. It separates generator from evaluator, forces mandatory correspondence between every claim and the source text, and automatically quarantines failures with full reasoning traces that a human can review (observability).
Correspondence Auditor is being used live, and catches real, complex issues like hallucinated facts and contradictory claims that would otherwise be very difficult to detect efficiently.
For example, when auditing an LLM-generated security assessment, both the original evaluator and the judge model erroneously agreed a source document was “missing an independent recovery mechanism”.
The audit pipeline caught this. The source text explicitly described “independent recovery paths.” A synonym substitution had fooled two layers of review.
The quarantined audit trail reads:
Claim: “Missing Independent Recovery Mechanism” Verdict: CONTRADICTED Reasoning: Text explicitly states “independent recovery paths” in main body. Concept is present (synonym: “paths” vs “mechanism” doesn’t negate presence).
Without mandatory claim-to-source correspondence, that contradiction passes silently.
I’m aware that frameworks like AI control and scalable oversight are exploring structurally similar ideas from the alignment side. I haven’t yet done that literature justice, and mapping where GRC patterns converge with or diverge from that work is important.
Enterprise GRC doesn’t operate in a vacuum. It works partly because the agents being overseen face real consequences—termination, prosecution, reputational destruction. The “we’re all on the same team” cooperation that makes people tolerate oversight is downstream of shared incentives and enforceable penalties. Whether structural transparency retains its power without that surrounding enforcement architecture is, to me, the most interesting open question here—and probably the most important one for transfer to AI systems.
Whether these patterns constitute a viable research direction for scalable AI oversight, I genuinely don’t know yet. But any institutional architecture with a multi-decade survival record against intelligent adversarial agents seems worth studying rigorously.
The Lens Doesn’t Need to See Its Own Flaws
I started from first principles, reading “The Lens That Sees Its Flaws”, where Yudkowsky argues that humanity’s advantage over mice is our ability to recognise our own cognitive distortions and apply reflective corrections. Science, rationality, the human endeavor—it’s the lens examining itself.
I’d like to offer a perspective that perhaps differs (or maybe augments) this idea. I am curious whether the community sees this as a contradiction, or simply a “yes, and.”
I spent a decade in enterprise risk management at global banks, an environment that has evolved to survive decades of adversarial pressure from intelligent agents. In this situation, we don’t ask the lens to see its own flaws. We also don’t assume the lens is acting with adversarial intent. But we also don’t rely on it being accurate. We operate from a principle of “trust, but verify”
A classic example is a rogue trader. They are highly capable, non-deterministic agents optimising for a proxy reward (i.e. their bonus) that might diverge drastically from the principal’s goal (i.e. the bank’s survival). They don’t usually break rules directly, but they are perfectly capable of finding loopholes, exploiting edge cases, and exceeding the spirit of constraints while satisfying the letter.
Governance, Risk, and Compliance (GRC) functions don’t manage this by training the trader to be more self-aware. Instead, we build structural transparency around them. Segregation of duties (the person who initiates a transaction cannot approve it), evidence-based verification, and fail-closed defaults (i.e when something doesn’t add up, the system halts and escalates rather than waving it through).
The final step is the completely independent ‘Third Line of Defence’ (Internal Audit). It verifies that both the business (first line) and the compliance/risk teams (second line) have done their jobs. It doesn’t assume bad faith, but it doesn’t require good faith either. It requires evidence, but it also requires independent logical evaluation.
When an auditor catches a problem ‘live’, our first thought isn’t bad faith. It’s usually just a broken control, a process gap, or somebody making a very everyday mistake.
That attitude matters, because people cooperate with oversight they perceive as fair. We are all on the same team; if the bank goes bust tomorrow, nobody gets paid.
I see the discussions in the AI safety community about LLM-as-a-judge, and the very real problems of using one model to evaluate another’s outputs.
This looks structurally identical to asking a trader’s desk-mate to review their risk positions based on professional judgment. The problem isn’t that the judge model is incapable. It’s that there is no segregation of duties, no requirement to ground evaluations to specific evidence, and no fail-closed default when the logic breaks.
To test whether these institutional GRC patterns actually transfer into AI safety, I built and open-sourced a small audit pipeline (Correspondence Auditor) that enforces these principles by evaluating LLM-as-a-judge outputs. It separates generator from evaluator, forces mandatory correspondence between every claim and the source text, and automatically quarantines failures with full reasoning traces that a human can review (observability).
Correspondence Auditor is being used live, and catches real, complex issues like hallucinated facts and contradictory claims that would otherwise be very difficult to detect efficiently.
For example, when auditing an LLM-generated security assessment, both the original evaluator and the judge model erroneously agreed a source document was “missing an independent recovery mechanism”.
The audit pipeline caught this. The source text explicitly described “independent recovery paths.” A synonym substitution had fooled two layers of review.
The quarantined audit trail reads:
Without mandatory claim-to-source correspondence, that contradiction passes silently.
I’m aware that frameworks like AI control and scalable oversight are exploring structurally similar ideas from the alignment side. I haven’t yet done that literature justice, and mapping where GRC patterns converge with or diverge from that work is important.
Enterprise GRC doesn’t operate in a vacuum. It works partly because the agents being overseen face real consequences—termination, prosecution, reputational destruction. The “we’re all on the same team” cooperation that makes people tolerate oversight is downstream of shared incentives and enforceable penalties. Whether structural transparency retains its power without that surrounding enforcement architecture is, to me, the most interesting open question here—and probably the most important one for transfer to AI systems.
Whether these patterns constitute a viable research direction for scalable AI oversight, I genuinely don’t know yet. But any institutional architecture with a multi-decade survival record against intelligent adversarial agents seems worth studying rigorously.