I feel like your jumping to cheating way to quickly. I think everyone would agree that there is overfitting to benchmarks and to benchmark like questions. Also, this is a very hard problem. The average person doesn’t have a shot at contributing to security research. Even the typical appsec engineer with years of experience would fail at the task of navigating a new codebase, creating a threat model and finding important security issues. This takes an expert in the field at least a few days of work. This is much longer than the time periods that AIs can be expected to work on a problem productively. I don’t think you get a completely general solution to this problem until we are well past AGI. However, it doesn’t mean you can saturate a meaningful benchmark and lead to a big improvement.
We need to zoom out. It clear across all benchmarks that we hit dimension returns from pretraining a long time ago. Bigger models aren’t helping much at all. Inference-time scaling is what helping lead to improvements. But how will the get a useful chain of thought for this use case. There is almost nothing online from security researching thinking though what is and isn’t an important security issue. We need to feed more of this data into LLM to benefit from chain of thought.
In addition, we likely need amazing scaffolding to break this problem in sub pieces so that each batch of thinking can move it forward. Ie— split the code base into sections to understand - Create a mental model of code base - Create a mental model of interactions - Document assumptions - Create a threat model - Test each threat model for security issues.
I feel like your jumping to cheating way to quickly. I think everyone would agree that there is overfitting to benchmarks and to benchmark like questions. Also, this is a very hard problem. The average person doesn’t have a shot at contributing to security research. Even the typical appsec engineer with years of experience would fail at the task of navigating a new codebase, creating a threat model and finding important security issues. This takes an expert in the field at least a few days of work. This is much longer than the time periods that AIs can be expected to work on a problem productively. I don’t think you get a completely general solution to this problem until we are well past AGI. However, it doesn’t mean you can saturate a meaningful benchmark and lead to a big improvement.
We need to zoom out. It clear across all benchmarks that we hit dimension returns from pretraining a long time ago. Bigger models aren’t helping much at all. Inference-time scaling is what helping lead to improvements. But how will the get a useful chain of thought for this use case. There is almost nothing online from security researching thinking though what is and isn’t an important security issue. We need to feed more of this data into LLM to benefit from chain of thought.
In addition, we likely need amazing scaffolding to break this problem in sub pieces so that each batch of thinking can move it forward. Ie—
split the code base into sections to understand
- Create a mental model of code base
- Create a mental model of interactions
- Document assumptions
- Create a threat model
- Test each threat model for security issues.