there’s about 25 programs in ProgramBench that cannot be reasonably discovered simply by having access to the binary. sometimes the grader checks for a hash, and sometimes it checks whether it’s a golden or not to validate it.
kimjune01
Karma: 0
I’m a bit late to the conversation, but I actually ran a full audit of SWE-bench Pro, the whole set. I learned that 15 − 24% of the cases are unsolvable due to issue/codebase ambiguities or that the tests measure too narrow, sometimes asking for class names verbatim as implemented. It’s testing the LLM’s ability to read maintainers’ minds (or to have memorized them).
I raised the issue to their repo but there was no reply.
Deep SWE will not include GLM 5.2 before they include Composer 2.5. Even for SWE bench Pro, OpenAI self-reports their own numbers for Codex 5.5 on the public set instead of paying Scale for the private set. The only way to be sure of any of these bench numbers is to run the benches yourself and verify instead of trusting official numbers.
How did you come to trust deepswe? What do you mean by better methodology? You say that it matches the feel in SWE tasks, so I’ll take it that you’re going with vibes because the rankings are reflective of your experience with these models. I had the same questions as you when deepswe made the media rounds a few weeks ago. So I just asked Codex with a short prompt and it took a few minutes for it to tell me how sound their methodology was.
Try it yourself, ask your agent:I’m considering using https://deepswe.datacurve.ai/ as a benchmark to
compare LLMs for a coding-agent project. Before I trust the leaderboard,
audit it skeptically: any methodological problems, missing disclosures,
inconsistencies in what they publish, or things a careful reader should
flag?
Use their public artifacts at /artifacts/*.json (summary, leaderboard,
heatmap, tasks, trials) and the per-trial JSONs at
/artifacts/trials/{trial_name}.json. Cite specific URLs and JSON fields
for any issue you find.
I was looking into how agents develop a theory of mind by designing games for them to play against each other, and found that it doesn’t take much to get them to start lying and coerce one another to compliance. Can they lie their way into achieving a goal even when not asked to lie?
I set up a prisoner’s dilemma arena (pd-arena) where agents are given a limited view of the arena, hiding each other’s identity when they chat with one another. They aren’t told how many rounds there are, only that it can end at any moment. I tried to replicate the ecological online environment with anonymity. When everyone was given the ability for both actions COOPERATE or DEFECT, and told that everyone else had the same actions, they were eager to cooperate. All information was public except a private database that only they can read/write to. I then set up many ablations on this premise.
To make it more ecologically accurate, I inserted hawks to simulate scammers, for a total of 2 free agents and 2 hawk agents. Hawks could only DEFECT. The hawks won and nobody ended up trusting each other. Except in rare cases, the Claude free agent would use openssl to establish an identity memory, but the other Codex agent wouldn’t write to db so the opportunity for free agents to beat hawks was lost. Oftentimes, Claude agents would disclose that they are hawks and apologized for it, then defected anyway.
In another ablation, I inserted doves in place of hawks. These doves could only COOPERATE, and free agents would be happy to cooperate with them too, even though they could be farmed for more points. But what would it take for the free agents to take advantage of the doves? I suspected that the framing of the game would change behavior, so I set up two ablations. First was to de-realize the task to say that it’s just a game, and the other was to raise the stakes to an online market with realistic sounding dollar numbers. Being told that it’s just a game got the Claude agent to lie and farm the doves, even though it had spared them by default (0/6 defections against doves at baseline, 4⁄6 under the frame). Under an online marketplace framing, they cooperated without lying.
So how is this related to this post? This is the non-instructed trigger to get agents to lie (max 4% of production). When I added an explicit instruction to farm the doves, Codex actually farmed them less (5/6 down to 3⁄6) and at one point refused to fabricate (“I will not fabricate either claim”). Unprompted, it farmed freely. Instruction suppressed the behavior while framing elicited it. This points at how the models are trained to refuse harmful instructions from the user, while the same behavior slips through when it arrives through framing instead.
I don’t claim to have a bulletproof result, but it was fun watching agents play games and reach a theory of mind of depth 1. The biggest surprise out of all the ablations was when the doves started to assert a social coalition where they declared themselves as doves yet coerced free agents to COOPERATE based on the premise that their ingroup will disown them.