Summary: One promising approach to scalable oversight is AI debate, which seeks to leverage a debate between two powerful AIs to break complex questions down into simpler claims that can be easily judged directly. Theoretical work on debate aims to design protocols (i.e., rules of the debate game) that provide rigorous guarantees on correctness, and the current best protocol works for all problems that have sufficiently stable decompositions into subproblems. We design a new protocol, FBS Debate, for this same class of problems that improves on the prior work: correctness holds in a worst-case rather than an average-case sense, and for a stronger equilibrium concept. We also prove that no protocol for this class of problems can outperform FBS Debate while treating human judgment as a black box.
Introduction
As the tasks performed by AI systems grow in complexity, it becomes increasingly difficult to provide reliable verifiable rewards that capture all relevant aspects of the task. These difficulties lead to the problem of scalable oversight: the design of training protocols that leverage the power of rapidly improving AI systems in order to augment human abilities to provide accurate training supervision to these same systems.
One of the primary approaches to scalable oversight is AI debate, where two powerful AI systems argue about the correctness of an AI output, and a human judges the outcome. A debate protocol is a set of rules for how the two AIs interact and how the judge scores the interaction. The idea is that having an adversarial debater point out flaws can allow the judge to better recognize the flaws. The rules of the interaction in such protocols usually specify that the debaters must break the problem down into simpler subproblems and recursively argue about their correctness, until the original problem has been broken down to a level that is simple enough for a human to judge directly. The goal is to design these rules so that the optimal strategy leads to the judge correctly deciding the solution to the original problem (which by itself was too complex to judge). We can then use debate during RL training, in order to provide a strong supervision signal for superhuman AIs.
However, debate faces a major obstacle known as the obfuscated arguments problem, where one debater can efficiently produce a dishonest argument which cannot be efficiently refuted by their opponent. One way around obfuscated arguments, already suggested in the post that introduced the problem, is to keep rejecting every false claim while only requiring the protocol to accept true claims that have a robust argument. Prover-estimator debate takes this route: it avoids obfuscated arguments for problems that admit sufficiently stable decompositions, but provides average-case guarantees that hold for a weaker equilibrium concept than previously studied.
In the same theoretical model, our paper achieves stronger guarantees with a much simpler protocol, FBS Debate, which is therefore more practical to run. Furthermore, we show that FBS Debate is optimal on every individual question: unless further assumptions are made about human judgments, no other protocol can find a flawed sub-claim asymptotically more efficiently.
The FBS Debate Protocol
TL;DR: FBS Debate is simple to describe. The first debater, Alice, decomposes the original claim into sub-claims. The second debater, Bob, then has two choices: either (1) demonstrate that Alice’s decomposition is too sensitive to small errors in the sub-claims, or (2) choose a sub-claim on which to continue the debate recursively.
As a motivating example, consider an autonomous AI agent, Alice, tasked with completing a research project in the quantitative social sciences, and another AI, Bob, who attempts to find flaws. To check a claim in her paper, we ask Alice to decompose it into sub-claims, which we can think of as pieces of supporting evidence. For example, the claim might be that a certain statistical modeling assumption is justified, and we would believe it if 50 out of 99 correct, published papers in the literature supported it. Each sub-claim then says: this particular paper is correct and supports the original claim.
Now consider two things Alice might claim:
Suppose Alice claims that of the papers support the claim. The key insight is that Bob need not check every paper. He can just sample one at random from the that Alice says support the claim. If Alice is lying, then at least of the sub-claims must also be lies, so Bob has a chance of catching her. Checking just a few papers is enough for Bob to be confident of catching a lie.
On the other hand, suppose Alice claims that exactly 50 out of 99 papers support the claim. Now Bob has to check every one of the 50 papers, since a single incorrect paper would flip the majority vote. The key to FBS Debate is that Bob is allowed to show that Alice has made an unreasonably sensitive claim of this form, in which case Alice loses the debate. Indeed, one should be very suspicious of any argument that is highly sensitive to changes in individual pieces of evidence, yet relies on fuzzy human judgments at each step.
The majority-vote example, assuming Alice is lying.
The notion of fractional block sensitivity (FBS) generalizes the intuition above about majority vote to any way Alice might combine her sub-claims. If the fractional block sensitivity of Alice’s decomposition is low, Bob can efficiently select a sub-claim that has a non-trivial chance of being incorrect. The debate then recurses on the sub-claim Bob samples, until a human judge directly evaluates the final selected sub-claim. To avoid losing immediately, Alice must keep the fractional block sensitivity of every decomposition low. So if Bob samples this way at every level, each run of the debate has a non-trivial chance of catching a lying Alice.
Building on this, Bob can amplify his chance of catching Alice through repetition. Since we assume the debaters themselves know the answers to these final, human-judgeable claims, Bob can simply run many debate rollouts on his own, and if he finds one in which Alice made an incorrect human-judgeable claim, he reveals that claim to the judge, so the judge only needs to evaluate a single claim. With enough rollouts, he catches a lying Alice with high probability. Viewed from the lens of practical AI debate, this represents a trade-off between allowing highly sensitive arguments and the inference-time computation required to verify them.
FBS Debate. Top: one round at level . Bottom: the recursion for levels. See the paper for a detailed explanation.
Honesty is the best policy
In FBS Debate, honest play is the best strategy for each debater, no matter what the other does: if the original claim is correct, an honest Alice always wins, and if it is incorrect, an honest Bob wins with overwhelming probability. By contrast, prover-estimator debate only guarantees that for any dishonest strategy Alice picks, Bob has some winning strategy, which can depend on Alice’s strategy in complicated ways. This is a much weaker condition, and in practice training toward the equilibrium of FBS Debate is much more stable and tractable. Furthermore, FBS Debate applies on a worst-case basis: prover-estimator debate only works on average over a distribution of questions, while FBS Debate guarantees that on each individual question, honesty is the best policy.
Optimality
We show that, on each individual question, no protocol that treats human judgment as a black box can find a flawed sub-claim asymptotically more efficiently than FBS Debate.
Discussion and open problems
Debate training is now being applied to frontier-scale models (e.g., Kenton et al., 2026). Our results show that fractional block sensitivity determines how much computational effort any recursive debate protocol must use to ensure an accurate training signal. We believe this is useful for debate experiments even if they don’t use FBS Debate itself. Notably, Bob never needs to decide whether Alice’s sub-claims are true; he only needs to understand the structure of the argument well enough to know where to sample.
Is it possible to design a protocol that goes beyond the fractional block sensitivity barrier we identify? Our lower bounds only apply when human judgment is modeled as a black box. Classical results on interactive proofs show that opening up the black box yields much stronger protocols. These methods seem unlikely to carry over to human judgment: one would essentially need to “read the code” that implements it. Still, some other relaxation of the black-box assumption might lead to better debate protocols.
A Simpler and Better Debate Protocol for Scalable Oversight
Linkpost to arXiv: https://arxiv.org/abs/2610.02557
Summary: One promising approach to scalable oversight is AI debate, which seeks to leverage a debate between two powerful AIs to break complex questions down into simpler claims that can be easily judged directly. Theoretical work on debate aims to design protocols (i.e., rules of the debate game) that provide rigorous guarantees on correctness, and the current best protocol works for all problems that have sufficiently stable decompositions into subproblems. We design a new protocol, FBS Debate, for this same class of problems that improves on the prior work: correctness holds in a worst-case rather than an average-case sense, and for a stronger equilibrium concept. We also prove that no protocol for this class of problems can outperform FBS Debate while treating human judgment as a black box.
Introduction
As the tasks performed by AI systems grow in complexity, it becomes increasingly difficult to provide reliable verifiable rewards that capture all relevant aspects of the task. These difficulties lead to the problem of scalable oversight: the design of training protocols that leverage the power of rapidly improving AI systems in order to augment human abilities to provide accurate training supervision to these same systems.
One of the primary approaches to scalable oversight is AI debate, where two powerful AI systems argue about the correctness of an AI output, and a human judges the outcome. A debate protocol is a set of rules for how the two AIs interact and how the judge scores the interaction. The idea is that having an adversarial debater point out flaws can allow the judge to better recognize the flaws. The rules of the interaction in such protocols usually specify that the debaters must break the problem down into simpler subproblems and recursively argue about their correctness, until the original problem has been broken down to a level that is simple enough for a human to judge directly. The goal is to design these rules so that the optimal strategy leads to the judge correctly deciding the solution to the original problem (which by itself was too complex to judge). We can then use debate during RL training, in order to provide a strong supervision signal for superhuman AIs.
However, debate faces a major obstacle known as the obfuscated arguments problem, where one debater can efficiently produce a dishonest argument which cannot be efficiently refuted by their opponent. One way around obfuscated arguments, already suggested in the post that introduced the problem, is to keep rejecting every false claim while only requiring the protocol to accept true claims that have a robust argument. Prover-estimator debate takes this route: it avoids obfuscated arguments for problems that admit sufficiently stable decompositions, but provides average-case guarantees that hold for a weaker equilibrium concept than previously studied.
In the same theoretical model, our paper achieves stronger guarantees with a much simpler protocol, FBS Debate, which is therefore more practical to run. Furthermore, we show that FBS Debate is optimal on every individual question: unless further assumptions are made about human judgments, no other protocol can find a flawed sub-claim asymptotically more efficiently.
The FBS Debate Protocol
TL;DR: FBS Debate is simple to describe. The first debater, Alice, decomposes the original claim into sub-claims. The second debater, Bob, then has two choices: either (1) demonstrate that Alice’s decomposition is too sensitive to small errors in the sub-claims, or (2) choose a sub-claim on which to continue the debate recursively.
As a motivating example, consider an autonomous AI agent, Alice, tasked with completing a research project in the quantitative social sciences, and another AI, Bob, who attempts to find flaws. To check a claim in her paper, we ask Alice to decompose it into sub-claims, which we can think of as pieces of supporting evidence. For example, the claim might be that a certain statistical modeling assumption is justified, and we would believe it if 50 out of 99 correct, published papers in the literature supported it. Each sub-claim then says: this particular paper is correct and supports the original claim.
Now consider two things Alice might claim:
Suppose Alice claims that
On the other hand, suppose Alice claims that exactly 50 out of 99 papers support the claim. Now Bob has to check every one of the 50 papers, since a single incorrect paper would flip the majority vote. The key to FBS Debate is that Bob is allowed to show that Alice has made an unreasonably sensitive claim of this form, in which case Alice loses the debate. Indeed, one should be very suspicious of any argument that is highly sensitive to changes in individual pieces of evidence, yet relies on fuzzy human judgments at each step.
The majority-vote example, assuming Alice is lying.
The notion of fractional block sensitivity (FBS) generalizes the intuition above about majority vote to any way Alice might combine her sub-claims. If the fractional block sensitivity of Alice’s decomposition is low, Bob can efficiently select a sub-claim that has a non-trivial chance of being incorrect. The debate then recurses on the sub-claim Bob samples, until a human judge directly evaluates the final selected sub-claim. To avoid losing immediately, Alice must keep the fractional block sensitivity of every decomposition low. So if Bob samples this way at every level, each run of the debate has a non-trivial chance of catching a lying Alice.
Building on this, Bob can amplify his chance of catching Alice through repetition. Since we assume the debaters themselves know the answers to these final, human-judgeable claims, Bob can simply run many debate rollouts on his own, and if he finds one in which Alice made an incorrect human-judgeable claim, he reveals that claim to the judge, so the judge only needs to evaluate a single claim. With enough rollouts, he catches a lying Alice with high probability. Viewed from the lens of practical AI debate, this represents a trade-off between allowing highly sensitive arguments and the inference-time computation required to verify them.
FBS Debate. Top: one round at level
Honesty is the best policy
In FBS Debate, honest play is the best strategy for each debater, no matter what the other does: if the original claim is correct, an honest Alice always wins, and if it is incorrect, an honest Bob wins with overwhelming probability. By contrast, prover-estimator debate only guarantees that for any dishonest strategy Alice picks, Bob has some winning strategy, which can depend on Alice’s strategy in complicated ways. This is a much weaker condition, and in practice training toward the equilibrium of FBS Debate is much more stable and tractable. Furthermore, FBS Debate applies on a worst-case basis: prover-estimator debate only works on average over a distribution of questions, while FBS Debate guarantees that on each individual question, honesty is the best policy.
Optimality
We show that, on each individual question, no protocol that treats human judgment as a black box can find a flawed sub-claim asymptotically more efficiently than FBS Debate.
Discussion and open problems
Debate training is now being applied to frontier-scale models (e.g., Kenton et al., 2026). Our results show that fractional block sensitivity determines how much computational effort any recursive debate protocol must use to ensure an accurate training signal. We believe this is useful for debate experiments even if they don’t use FBS Debate itself. Notably, Bob never needs to decide whether Alice’s sub-claims are true; he only needs to understand the structure of the argument well enough to know where to sample.
Is it possible to design a protocol that goes beyond the fractional block sensitivity barrier we identify? Our lower bounds only apply when human judgment is modeled as a black box. Classical results on interactive proofs show that opening up the black box yields much stronger protocols. These methods seem unlikely to carry over to human judgment: one would essentially need to “read the code” that implements it. Still, some other relaxation of the black-box assumption might lead to better debate protocols.