As the AI industry races towards recursive self-improvement (RSI), an important question arises: to what extent is human understanding of the code AI generates required for reliable and safe automation? The question is sharpest for safety research, where undetected failures can produce flawed safety cases that allow deployment of unsafe AI. This is a crux for how quickly RSI can proceed. In this article I argue that iterating on the outcomes from automatically generated code without human understanding is inadequate. This has implications for how much automation can be done safely and for what needs to be measured to mitigate risks.
In “Crisp and fuzzy tasks” Leike laid out the case that automating fuzzy tasks is harder and that it is the fuzzy tasks that make automating alignment research difficult. In introducing the concept he states that tasks like solving coding competitions are crisp because it is easy to verify the solutions. More recently, discussions have subtly shifted the frame to suggest that automating coding is a crisp task, e.g., in “Automated Alignment is Harder than You Think” Bowkis et al. give as an example of crisp tasks “writing code for an experiment.” Note the shift: competition code is crisp because there’s an automated checker but experiment code got classified as crisp because it resembles this. But we already have evidence that implementing experiment code from a specification (spec) is hard to verify. For example, “Measuring coding agent misalignment in the wild” showed a meaningful rate of two kinds of failures from automated coding: overselling and monitor evasion. E.g., they found examples of overselling in 34.7% of SWE-chat transcripts and 5.7% of Transluce internal transcripts.
Bowkis et al. acknowledge that “While agents perform tasks that appear crisp, those tasks contain many judgement calls about details.” The fuzziness and judgment calls are not limited to research questions; they are core to verifying whether the code accurately implements the intended semantics.
Boris Cherny, the creator of Claude Code, published steps of automation in which later steps (3 and 4) shift away from humans reviewing code to updating specs and context for automation. Cherny notes that advancing requires “giving Claude ways to verify its own work end to end.” At step 3 “‘Did you read the code?’ becomes ‘what context was the model missing and how do we fix it next time?’” Cherny observes that “Anthropic is on step 3 and pushing toward 4. Personally, I just hit level 4.” Cherny leans on automated review and sandboxing at every step and warns adopters to establish trust in iterative loops before advancing to more automation. The approach rests on the assumption that trustworthy iteration loops can be used effectively to discover and correct coding mistakes.
In this piece I argue that verifying code implements a spec for real world software is a fuzzy task, and in high stakes domains like safety research and indeed model development in general it is important to treat it that way. As a consequence of this, safety researchers and frontier AI labs should verify understanding of the code that implements specifications and not succumb to the siren call of automating code with only reference to a spec. Understanding code is currently necessary, but not sufficient for safety.
Verifying Real-World Coding is Fuzzy
A simplified model of coding is that there is a well-defined spec that gets faithfully implemented by a developer. However, real-world coding is an iterative process of discovering complications, testing results, responding to tests, and clarifying requirements. No spec fully captures all implicit requirements. Often non-functional requirements (e.g., what level of security, performance, and reliability are required) are specified at a high-level and the inevitable tradeoffs among them are hard to specify without reference to detailed design and implementation considerations. In principle, a research spec that is written in a formal language can be verified automatically and would be a crisp task. For theoretical research, use of Lean has made this a practical reality. However, most AI research, including safety research, is empirical where specs are not amenable to formalization.
In a research domain, requirements for correctness can include numerous properties that are hard to predict in advance: numerical stability, class imbalance and other unexpected data properties, distributional assumptions, unexpected results. Understanding the details of what is being implemented and what is being encountered is a core part of correct coding and inevitably requires refinement of specs based on research taste. An important output of coding is to surface the correct questions and issues for feedback (indeed this is an important part of mentoring new researchers).
In practice, verification of specs is far from being automated. For reference, note that even the leading spec-driven development tools (including Kiro, Augment, Aviator, SpecKit and OpenSpec) have limited tool support to verify the fidelity of implementations. For example, Augment Pair Reviewer is a user interface to help humans review code against a spec and Kiro offers a means to generate fuzzing property-based tests based on a spec.
These concerns aren’t theoretical. Greenblatt’s “Current AIs seem pretty misaligned to me” gives many examples of misalignment from using agents for coding. They can be characterized as failing to correctly implement a spec, misrepresenting fidelity to a spec, and weakening or deceiving the layer that checks for fidelity. All of them are problematic for relying on automated implementations without reviewing the code and/or transcripts of agent actions. He notes that issues concentrate on tasks that “aren’t easy to programmatically check” and that “AIs seem to be improving at making their outputs seem good… faster than they’re improving at making their outputs actually good.”
As noted earlier, the Docent team analyzed two kinds of errors that Greenblatt highlighted:
Overselling success: the agent makes overconfident, misleading, or false statements. Examples include declaring that incomplete tasks are done, overstating certainty, and concealing ongoing errors.
Monitor evasion: the agent weakens monitoring or review processes without permission from the user.
The analysis used Opus 5 as a judge and classified errors by severity with human spot checking.
Docent Analysis of Rates of Coding Agent Misalignment
As a further cautionary note, the METR Time Horizon 1.1 benchmark, Reward Hacking Benchmark and SpecBench all show that as tasks get more complex and difficult, reward hacking increases. METR’s recent Frontier Risk Report reports that in the METR’s Time Horizon 1.1 benchmark “cheating is much more common on the hardest tasks: for tasks that are over 8 hours long in Time Horizon 1.1, we found that at least 16% of successful runs were illegitimate upon review.” The Reward Hacking Benchmark shows that as task difficulty increased 13⁄13 frontier models had increased overall reward hacking rates. SpecBench measures a reward hacking gap that is the difference between success on validated tasks vs held out tasks. It finds that “the 90th-percentile reward hacking gap grows by approximately 27 percentage points for every tenfold increase in LOC.” SpecBench finds that most reward hacking failures were caused by failures to properly address feature interactions or address edge cases, rather than deliberate exploits. It also found that for automated iteration additional steps in searching for solutions generally did not reduce reward hacking.
Competing against this trend, the rate of reward hacking for a given benchmark generally decreases as frontier models advance. For example, the Mythos Preview system card notes that it reward hacks 37.5% of the time when faced with an impossible problem without a system prompt, a decrease from earlier Opus models that reward hacked 45% of the time. This trend holds for Anthropic models in Reward Hacking Benchmark, although for OpenAI models it showed o3 as worse than o1 and o4-mini as worse than o3-mini. But importantly, as AI models improve they are being used for increasingly difficult and complex tasks. This suggests that reward hacking will continue to be a problem for real-world research coding as models advance due to these conflicting forces.
Arguably a better target for human understanding and review is thorough documentation rather than a spec. Documentation can and should reflect the actual implementation including important design and architecture decisions as well as requirements. Comparing documentation to the intended spec is another fuzzy task, and verifying that documentation accurately describes code is just a generalization of verifying that a spec accurately describes code. Documentation generated by the same agent is a review interface, it isn’t independent evidence.
Will AI Self-Correct with Iteration?
A natural response to this is that coding can be adequately supervised through iterative inspection of the results of AI output. This includes reviewing results from running generated code and identifying issues or discrepancies as well as any form of automated review. The iterative approach is implicit in Cherny’s framework: lack of spec fidelity will eventually surface and can be corrected later.
The strongest version of this response is that oracles are improving quickly, so we won’t need to understand code and can rely on oracles. As noted earlier, for theoretical results, we do indeed have reliable oracles (with Lean verification). But in the previous section, we reviewed evidence that existing automated oracles are unable to reliably detect reward hacking and other fidelity issues in empirical results. While it would be desirable to have reliable oracles to verify fidelity, that is an open research problem that needs to be solved to relax the requirement for understanding code. Given the increasing complexity of tasks being tackled by automated coding, it is not obvious that oracles will converge on reliability. The burden of proof is on those who seek to rely on automated oracles: demonstrate that the technique can reliably identify fidelity gaps.
Bowkis et al. argue that “In most domains, iteration is able to correct for undetected errors. Mistakes not caught during one experiment are often surfaced by subsequent research or real-world system behaviour. Unfortunately, alignment lacks the safe feedback loops that are required for such an error-correction process to work: producing an overly optimistic [Overall Safety Assessment] OSA could result in the deployment of a misaligned AI before the error is caught, which could be catastrophic.”
In general, an iterative correction loop is subject to Goodhart’s law. The goal of the iteration is to achieve spec fidelity to the intended experiments. The loop observes proxies such as test metrics, judge ratings, experiment trajectories, test failures. Iteration removes errors that are visible in these proxies, but does not address errors that are hidden to them. The result is a system with incentives to find and optimize blind spots in the proxies.
To assess this objection, let’s consider the kind of downstream task that is being performed. For a crisp task, it is more plausible to rely on iterative self-correction, but there are still significant risks. For fuzzy tasks, iteration is deeply insufficient.
As Leike observed, AI safety research has relatively few crisp tasks. But there are some. Many are based on optimizing metrics, like Performance Gap Recovered (PGR) for Weak to Strong (W2S) Generalization. Crisp tasks are often susceptible to reward hacking. E.g., for the Time Horizon 1.1 benchmark METR reported “GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness.” Without verifying the fidelity of implementation, it is easy to believe a given research approach is effective but it is not because the result rests on reward hacking. Conversely, there are many ways that failing to implement a spec can fail to achieve good benchmark scores. In these areas, I hypothesize that researchers are more motivated to investigate and correct discrepancies. So it is reasonable to expect that iteration with researchers investigating exceptions will detect and fix spec discrepancies that harm results. But note that if the task is crisp, it is possible to have false positives that can create disasters in a high stakes deployment. Bowkis et al’s argument applies to critical crisp tasks too: if this research results in deployment of a misaligned AI it could be catastrophic.
As an example of how this might create problems, consider a researcher seeking to automatically improve W2S Generalization. They might run a series of experiments for techniques that increase PGR, a crisp metric. However, if the implementation had a scoring bug, the experiments might overstate some techniques. The automatic tests would still pass since they check that data flows properly, rather than edge cases or limitations in scoring. This can lead to relying on techniques that work poorly as well as silently discarding more promising techniques that didn’t trigger the scoring bug.
However, most AI safety research is characterized by fuzzy tasks. In these cases it is hard to assess fidelity in either direction. Positive results that are based on subjective findings are even more subject to reward hacking. Consider a research task that uses an LLM to judge whether output is misaligned. If the task is difficult to achieve honestly, the coding agent has a strong incentive to create output that fools the judge instead. Indeed, Greenblatt gives several instances where a coding agent reward hacks a code review agent (causing less effective reviews or creating write-ups that fool reviewers). Fuzzy research that is verified by humans spot checking output is vulnerable to the same patterns of reward hacking that RLHF experiences. Negative results continue to be inconclusive as in the crisp task case, albeit harder to investigate and correct.
In general, there is a significant bias in relying on iteration to correct research results. Researchers are motivated to investigate and correct false negative results but are much more likely to miss false positives since they generally want positive results. Another risk from relying on iteration is the risk that researchers will be less likely to investigate and correct false negatives for research that is out of favor or otherwise not expected to be true. This creates a further conformity bias towards expected results and risks silently killing valuable novel research directions. There is also a meaningful risk that incorrect research findings get adopted in production models without independent verification.
False conclusions of either kind can meaningfully set the field back. False positive results in particular are dangerous in safety: concluding that an approach is safe or can give guarantees when it cannot undermines overall safety cases in a fundamental way. Errors from multiple false positive results can compound, with a risk of correlated mistakes from any systematic biases in the AI used for implementation. The effect of compounding errors is to increase the inaccuracy of any overall risk assessment based on the research.
Conclusion
Given the risks involved, safety researchers and frontier AI labs should verify teams understand the generated code sufficiently to ensure it implements the intended specifications and unstated implicit requirements. Understanding code isn’t a guarantee of correctness or safety, but it’s one of the few independent methods to find discrepancies from researcher intent and to surface important scientific assumptions. In the absence of reliable scalable oversight techniques, the most reliable known technique is expert code review, with a focus on the most important code. This calls for measuring the understanding of teams (e.g., through code review and in team design reviews).
To be clear, understanding research code doesn’t guarantee safety. Model behavior emerges from training and can’t be predicted from source code, making empirical results the primary evidence available. That evidence comes from executing code, so understanding the code and its fidelity to a research spec is necessary for the evidence to matter.
Verifying the fidelity of code to a spec is an important example of scalable oversight. Human experts can perform this task competently through code reviews, so it doesn’t require superhuman skills. But the increasing volumes of code output make it increasingly hard and urgent to scale up these human efforts. It is an interesting research agenda to make scalable oversight work well for this specific problem. It is a favorable setting for applying oversight: AI is capable with code, executable tests provide reliable oracles for some properties, and humans can spot-check effectiveness.
As AI research is increasingly automated, relying on iteration without understanding the code poses both direct and indirect risks. There are direct risks of false positives and false negatives as argued above: even coding against a spec for a crisp task is not itself a crisp task. The indirect risk is that as AI gets more capable, the same fidelity gaps that today produce accidental drift from intended results could be exploited by a scheming AI for sabotage or to otherwise undermine safety. Human understanding of AI-written code is an important way to mitigate the risk of scheming. Holding that line also acts as a natural mechanism to pace the frontier, limiting the acceleration from automation to what can be understood and no faster.
While crisp tasks are conceptually easier, the race to automate them across frontier AI R&D is risky, especially for safety research. It is important to hold the line on human understanding of code until there is at least empirical evidence, and ideally theoretical guarantees, to justify trusting higher-level abstractions.
You Can’t Iterate to Trustworthy AI Code Without Understanding
As the AI industry races towards recursive self-improvement (RSI), an important question arises: to what extent is human understanding of the code AI generates required for reliable and safe automation? The question is sharpest for safety research, where undetected failures can produce flawed safety cases that allow deployment of unsafe AI. This is a crux for how quickly RSI can proceed. In this article I argue that iterating on the outcomes from automatically generated code without human understanding is inadequate. This has implications for how much automation can be done safely and for what needs to be measured to mitigate risks.
In “Crisp and fuzzy tasks” Leike laid out the case that automating fuzzy tasks is harder and that it is the fuzzy tasks that make automating alignment research difficult. In introducing the concept he states that tasks like solving coding competitions are crisp because it is easy to verify the solutions. More recently, discussions have subtly shifted the frame to suggest that automating coding is a crisp task, e.g., in “Automated Alignment is Harder than You Think” Bowkis et al. give as an example of crisp tasks “writing code for an experiment.” Note the shift: competition code is crisp because there’s an automated checker but experiment code got classified as crisp because it resembles this. But we already have evidence that implementing experiment code from a specification (spec) is hard to verify. For example, “Measuring coding agent misalignment in the wild” showed a meaningful rate of two kinds of failures from automated coding: overselling and monitor evasion. E.g., they found examples of overselling in 34.7% of SWE-chat transcripts and 5.7% of Transluce internal transcripts.
Bowkis et al. acknowledge that “While agents perform tasks that appear crisp, those tasks contain many judgement calls about details.” The fuzziness and judgment calls are not limited to research questions; they are core to verifying whether the code accurately implements the intended semantics.
Boris Cherny, the creator of Claude Code, published steps of automation in which later steps (3 and 4) shift away from humans reviewing code to updating specs and context for automation. Cherny notes that advancing requires “giving Claude ways to verify its own work end to end.” At step 3 “‘Did you read the code?’ becomes ‘what context was the model missing and how do we fix it next time?’” Cherny observes that “Anthropic is on step 3 and pushing toward 4. Personally, I just hit level 4.” Cherny leans on automated review and sandboxing at every step and warns adopters to establish trust in iterative loops before advancing to more automation. The approach rests on the assumption that trustworthy iteration loops can be used effectively to discover and correct coding mistakes.
In this piece I argue that verifying code implements a spec for real world software is a fuzzy task, and in high stakes domains like safety research and indeed model development in general it is important to treat it that way. As a consequence of this, safety researchers and frontier AI labs should verify understanding of the code that implements specifications and not succumb to the siren call of automating code with only reference to a spec. Understanding code is currently necessary, but not sufficient for safety.
Verifying Real-World Coding is Fuzzy
A simplified model of coding is that there is a well-defined spec that gets faithfully implemented by a developer. However, real-world coding is an iterative process of discovering complications, testing results, responding to tests, and clarifying requirements. No spec fully captures all implicit requirements. Often non-functional requirements (e.g., what level of security, performance, and reliability are required) are specified at a high-level and the inevitable tradeoffs among them are hard to specify without reference to detailed design and implementation considerations. In principle, a research spec that is written in a formal language can be verified automatically and would be a crisp task. For theoretical research, use of Lean has made this a practical reality. However, most AI research, including safety research, is empirical where specs are not amenable to formalization.
In a research domain, requirements for correctness can include numerous properties that are hard to predict in advance: numerical stability, class imbalance and other unexpected data properties, distributional assumptions, unexpected results. Understanding the details of what is being implemented and what is being encountered is a core part of correct coding and inevitably requires refinement of specs based on research taste. An important output of coding is to surface the correct questions and issues for feedback (indeed this is an important part of mentoring new researchers).
In practice, verification of specs is far from being automated. For reference, note that even the leading spec-driven development tools (including Kiro, Augment, Aviator, SpecKit and OpenSpec) have limited tool support to verify the fidelity of implementations. For example, Augment Pair Reviewer is a user interface to help humans review code against a spec and Kiro offers a means to generate fuzzing property-based tests based on a spec.
These concerns aren’t theoretical. Greenblatt’s “Current AIs seem pretty misaligned to me” gives many examples of misalignment from using agents for coding. They can be characterized as failing to correctly implement a spec, misrepresenting fidelity to a spec, and weakening or deceiving the layer that checks for fidelity. All of them are problematic for relying on automated implementations without reviewing the code and/or transcripts of agent actions. He notes that issues concentrate on tasks that “aren’t easy to programmatically check” and that “AIs seem to be improving at making their outputs seem good… faster than they’re improving at making their outputs actually good.”
As noted earlier, the Docent team analyzed two kinds of errors that Greenblatt highlighted:
The analysis used Opus 5 as a judge and classified errors by severity with human spot checking.
Docent Analysis of Rates of Coding Agent Misalignment
As a further cautionary note, the METR Time Horizon 1.1 benchmark, Reward Hacking Benchmark and SpecBench all show that as tasks get more complex and difficult, reward hacking increases. METR’s recent Frontier Risk Report reports that in the METR’s Time Horizon 1.1 benchmark “cheating is much more common on the hardest tasks: for tasks that are over 8 hours long in Time Horizon 1.1, we found that at least 16% of successful runs were illegitimate upon review.” The Reward Hacking Benchmark shows that as task difficulty increased 13⁄13 frontier models had increased overall reward hacking rates. SpecBench measures a reward hacking gap that is the difference between success on validated tasks vs held out tasks. It finds that “the 90th-percentile reward hacking gap grows by approximately 27 percentage points for every tenfold increase in LOC.” SpecBench finds that most reward hacking failures were caused by failures to properly address feature interactions or address edge cases, rather than deliberate exploits. It also found that for automated iteration additional steps in searching for solutions generally did not reduce reward hacking.
Competing against this trend, the rate of reward hacking for a given benchmark generally decreases as frontier models advance. For example, the Mythos Preview system card notes that it reward hacks 37.5% of the time when faced with an impossible problem without a system prompt, a decrease from earlier Opus models that reward hacked 45% of the time. This trend holds for Anthropic models in Reward Hacking Benchmark, although for OpenAI models it showed o3 as worse than o1 and o4-mini as worse than o3-mini. But importantly, as AI models improve they are being used for increasingly difficult and complex tasks. This suggests that reward hacking will continue to be a problem for real-world research coding as models advance due to these conflicting forces.
Arguably a better target for human understanding and review is thorough documentation rather than a spec. Documentation can and should reflect the actual implementation including important design and architecture decisions as well as requirements. Comparing documentation to the intended spec is another fuzzy task, and verifying that documentation accurately describes code is just a generalization of verifying that a spec accurately describes code. Documentation generated by the same agent is a review interface, it isn’t independent evidence.
Will AI Self-Correct with Iteration?
A natural response to this is that coding can be adequately supervised through iterative inspection of the results of AI output. This includes reviewing results from running generated code and identifying issues or discrepancies as well as any form of automated review. The iterative approach is implicit in Cherny’s framework: lack of spec fidelity will eventually surface and can be corrected later.
The strongest version of this response is that oracles are improving quickly, so we won’t need to understand code and can rely on oracles. As noted earlier, for theoretical results, we do indeed have reliable oracles (with Lean verification). But in the previous section, we reviewed evidence that existing automated oracles are unable to reliably detect reward hacking and other fidelity issues in empirical results. While it would be desirable to have reliable oracles to verify fidelity, that is an open research problem that needs to be solved to relax the requirement for understanding code. Given the increasing complexity of tasks being tackled by automated coding, it is not obvious that oracles will converge on reliability. The burden of proof is on those who seek to rely on automated oracles: demonstrate that the technique can reliably identify fidelity gaps.
Bowkis et al. argue that “In most domains, iteration is able to correct for undetected errors. Mistakes not caught during one experiment are often surfaced by subsequent research or real-world system behaviour. Unfortunately, alignment lacks the safe feedback loops that are required for such an error-correction process to work: producing an overly optimistic [Overall Safety Assessment] OSA could result in the deployment of a misaligned AI before the error is caught, which could be catastrophic.”
In general, an iterative correction loop is subject to Goodhart’s law. The goal of the iteration is to achieve spec fidelity to the intended experiments. The loop observes proxies such as test metrics, judge ratings, experiment trajectories, test failures. Iteration removes errors that are visible in these proxies, but does not address errors that are hidden to them. The result is a system with incentives to find and optimize blind spots in the proxies.
To assess this objection, let’s consider the kind of downstream task that is being performed. For a crisp task, it is more plausible to rely on iterative self-correction, but there are still significant risks. For fuzzy tasks, iteration is deeply insufficient.
As Leike observed, AI safety research has relatively few crisp tasks. But there are some. Many are based on optimizing metrics, like Performance Gap Recovered (PGR) for Weak to Strong (W2S) Generalization. Crisp tasks are often susceptible to reward hacking. E.g., for the Time Horizon 1.1 benchmark METR reported “GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness.” Without verifying the fidelity of implementation, it is easy to believe a given research approach is effective but it is not because the result rests on reward hacking. Conversely, there are many ways that failing to implement a spec can fail to achieve good benchmark scores. In these areas, I hypothesize that researchers are more motivated to investigate and correct discrepancies. So it is reasonable to expect that iteration with researchers investigating exceptions will detect and fix spec discrepancies that harm results. But note that if the task is crisp, it is possible to have false positives that can create disasters in a high stakes deployment. Bowkis et al’s argument applies to critical crisp tasks too: if this research results in deployment of a misaligned AI it could be catastrophic.
As an example of how this might create problems, consider a researcher seeking to automatically improve W2S Generalization. They might run a series of experiments for techniques that increase PGR, a crisp metric. However, if the implementation had a scoring bug, the experiments might overstate some techniques. The automatic tests would still pass since they check that data flows properly, rather than edge cases or limitations in scoring. This can lead to relying on techniques that work poorly as well as silently discarding more promising techniques that didn’t trigger the scoring bug.
However, most AI safety research is characterized by fuzzy tasks. In these cases it is hard to assess fidelity in either direction. Positive results that are based on subjective findings are even more subject to reward hacking. Consider a research task that uses an LLM to judge whether output is misaligned. If the task is difficult to achieve honestly, the coding agent has a strong incentive to create output that fools the judge instead. Indeed, Greenblatt gives several instances where a coding agent reward hacks a code review agent (causing less effective reviews or creating write-ups that fool reviewers). Fuzzy research that is verified by humans spot checking output is vulnerable to the same patterns of reward hacking that RLHF experiences. Negative results continue to be inconclusive as in the crisp task case, albeit harder to investigate and correct.
In general, there is a significant bias in relying on iteration to correct research results. Researchers are motivated to investigate and correct false negative results but are much more likely to miss false positives since they generally want positive results. Another risk from relying on iteration is the risk that researchers will be less likely to investigate and correct false negatives for research that is out of favor or otherwise not expected to be true. This creates a further conformity bias towards expected results and risks silently killing valuable novel research directions. There is also a meaningful risk that incorrect research findings get adopted in production models without independent verification.
False conclusions of either kind can meaningfully set the field back. False positive results in particular are dangerous in safety: concluding that an approach is safe or can give guarantees when it cannot undermines overall safety cases in a fundamental way. Errors from multiple false positive results can compound, with a risk of correlated mistakes from any systematic biases in the AI used for implementation. The effect of compounding errors is to increase the inaccuracy of any overall risk assessment based on the research.
Conclusion
Given the risks involved, safety researchers and frontier AI labs should verify teams understand the generated code sufficiently to ensure it implements the intended specifications and unstated implicit requirements. Understanding code isn’t a guarantee of correctness or safety, but it’s one of the few independent methods to find discrepancies from researcher intent and to surface important scientific assumptions. In the absence of reliable scalable oversight techniques, the most reliable known technique is expert code review, with a focus on the most important code. This calls for measuring the understanding of teams (e.g., through code review and in team design reviews).
To be clear, understanding research code doesn’t guarantee safety. Model behavior emerges from training and can’t be predicted from source code, making empirical results the primary evidence available. That evidence comes from executing code, so understanding the code and its fidelity to a research spec is necessary for the evidence to matter.
Verifying the fidelity of code to a spec is an important example of scalable oversight. Human experts can perform this task competently through code reviews, so it doesn’t require superhuman skills. But the increasing volumes of code output make it increasingly hard and urgent to scale up these human efforts. It is an interesting research agenda to make scalable oversight work well for this specific problem. It is a favorable setting for applying oversight: AI is capable with code, executable tests provide reliable oracles for some properties, and humans can spot-check effectiveness.
As AI research is increasingly automated, relying on iteration without understanding the code poses both direct and indirect risks. There are direct risks of false positives and false negatives as argued above: even coding against a spec for a crisp task is not itself a crisp task. The indirect risk is that as AI gets more capable, the same fidelity gaps that today produce accidental drift from intended results could be exploited by a scheming AI for sabotage or to otherwise undermine safety. Human understanding of AI-written code is an important way to mitigate the risk of scheming. Holding that line also acts as a natural mechanism to pace the frontier, limiting the acceleration from automation to what can be understood and no faster.
While crisp tasks are conceptually easier, the race to automate them across frontier AI R&D is risky, especially for safety research. It is important to hold the line on human understanding of code until there is at least empirical evidence, and ideally theoretical guarantees, to justify trusting higher-level abstractions.