Aaron Staley

Karma: 74

Aaron Staley 5 Oct 2025 18:06 UTC
2 points
0
in reply to: Baybar’s comment on: Checking in on AI-2027
Good response. A few things I do want to stress:
. I am just not sure I believe 25%-33% behind is significant.
I personally see the lower bound as 33% slower. That’s enough to change 2 to 3 years which is significant.
And again, realistically progress is even slower. The parallel compute version only increased by 1.8% in 4 months. We might be another 6 months from hitting 85% at current rates—this is quite a prediction gap.
and knowledgeable human performance on the benchmark remains around 70%.
Is this true? They haven’t updated their abstract claiming 72.36% (which was from the old version) and I’m wondering if they simply haven’t re-evaluated.
But yes, looking at the GTA1 paper, you are correct that perf varies a bit between os-world and os-world-verified, so I take back that growth is obviously slower than projected.
All said, I trust swe-bench-verified more regardless to track progress:
1. We’re relying on a well-made benchmark that was done as a second pass by OpenAI. os-world is not that.
2. Labs seem to be targetting more—low hanging fuit like attaching python interpreters just doesn’t exist for this benchmark ( I’m not sure if the ai-2027 considered this issue when making their os-world predictions)..
3. We are concerned mainly with coding abilities (automated ai research) on the ai 2027 timelines.

Aaron Staley 3 Oct 2025 18:29 UTC
6 points
3
on: Checking in on AI-2027
Claude Sonnet 4.5 scored an 82% on this metric, as of September 29th, 2025. Three percentage points below the 85% target, achieved one month late, again, remarkably close. Particularly given that in August, Opus 4.1 was already scoring 80% on this benchmark.
I disagree this is close for several reasons.
1. It isn’t clear that the “parallel test time” number even counts.
  1. My understanding is these benchmarks can’t be achieved by using mechanisms that cost more in compute than a human to manually perform and we have no idea how much parallel attempts are sampled. They use up to 256 in their post on GPQA
  2. It uses an internal scoring model that might not generalize beyond the repos swe-bench tests.
  3. Sonnet 3.7′s 70.3% score did not exist on swebench.com at the point ai-2027 was released (highest was 65.4%), suggesting the authors were not anchoring from that parallel test time number to begin with.
2. If parallel test time does count, projection is not close:
  1. A projection for 5 months away (beginning of Sep) of growing +15% instead grew +12% 6 months away. That’s 33% slower growth (2% a month vs. 3% a month projected)
  2. Looking more recently, the growth from May’s Sonnet 4 with parallel compute to now (4 months later) has been 1.8%. At this rate assuming linearity, 85% won’t be crossed for nearly 7 months from now, which is over 60% slower than projection.
Claude Sonnet 4.5 scored a 62% on this metric, as of September 29th, 2025.
For OSWorld, these aren’t even the same benchmarks. ai-2027 referred to the original osworld, while the sonnet 4.5 score of 61.4% is for osworld-verifed. Huge difference—Sonnet 3.7 scored 28 on osworld original, while getting a 35.8% on osworld-verified. We might be at more like a 55.6% SOTA today (GTA1 w/ GPT-5) on OG osworld, a huge miss (~46% slower)

Overall, realized data suggests something more like an ai-2029 or even later.

Aaron Staley 7 Aug 2025 21:59 UTC
5 points
1
in reply to: O O’s comment on: tdko’s Shortform
I don’t believe there’s a strong correlation between mathematical ability and agentic coding tasks (as opposed to competition coding tasks where a stronger correlation exists).
1. Gemini 2.5 Pro is already was well ahead of O3 on IMO, but had worse swe-bench/METR scores.
2. Claude is relatively bad at math but has hovered around SOTA on agentic coding.

Aaron Staley 7 Aug 2025 19:41 UTC
7 points
1
in reply to: Bitnotri’s comment on: tdko’s Shortform
+ 25% for swe-bench relative to Gemini 2.5? Quadrupling the METR task length of Gemini 2.5?

I suppose it’s a possibility, albeit a remote one.

Aaron Staley 7 Aug 2025 18:36 UTC
21 points
12
in reply to: β-redex’s comment on: tdko’s Shortform
The swe-bench scores are already well below trend from ai 2027. Had to hit 85% by end of month. We’re at 75%. (and SOTA was ~64% when they released ai 2027)

Aaron Staley 31 Jul 2025 6:19 UTC
3 points
0
in reply to: Nikola Jurkovic’s comment on: nikola’s Shortform
Very wide confidence intervals. If Grok 4 were equal to O3 in 50%, time horizon, it “beating” by this much is a 33% outcome. (On the other hand, losing by this amount in the 80% bucket is a 32% outcome).
Overall, I read this as about equally agentic as O3. Possibly slightly less so given the lack of swe-bench scores published for it (suggesting it wasn’t SOTA).

Aaron Staley 29 Jul 2025 16:25 UTC
3 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
My expectation is that GPT-5 will be a decent amount better than o3 on agentic software engineering (both in benchmarks and in practice), but won’t be substantially above trend. In particular, my median is that it will have a 2.75 hour time horizon^[1] on METR’s evaluation suite ^[2]. This prediction was produced by extrapolating out the faster 2024-2025 agentic software engineering time horizon trend from o3 and expecting GPT-5 will be slightly below trend.^[3]

If the correlations continue to hold, this would map to something like a 78% to 80% range on swe-bench pass @ 1 (which is likely to be announced at release). I’m personally not this bearish (I’d guess low 80s given that benchmark has reliably jumped ~3.5% monthly), but we shall see.
Needless to say if it scores 80%, we are well below AI 2027 timeline predictions with high confidence.

Aaron Staley 29 Jul 2025 16:22 UTC
5 points
2
in reply to: Garrett Baker’s comment on: tdko’s Shortform
Coding agentic abilities are different from general chatbot abilities. Gemini is IMO the best chatbot there is (just in terms of understanding context well if you wish to analyze text/learn things/etc.). Claude on the other hand is dead last among the big 3 (a steep change from a year ago) and my guess is Anthropic isn’t trying much anymore (focusing on.. agentic coding instead)

Aaron Staley 29 Jul 2025 16:20 UTC
3 points
0
in reply to: tdko’s comment on: tdko’s Shortform
I don’t see that producing much of an update. Its SWE-bench score as you note was only 59.6%, which naively maps to ~50 minutes METR.

Aaron Staley 21 Jul 2025 18:51 UTC
3 points
0
in reply to: james oofou’s comment on: james oofou’s Shortform
I don’t think you can just start at the HCAST timeline for software engineering and map it to IMO problems.
Alternative bearish prediction would be deepthink got 50% on May 20 (not released, lab frontier) on USAMO. 80% is ~4x the task time of 50% ones (at least for software engineering—not sure what it is for math), so we needed two doublings (6 months) to pull this off and instead only have ~0.67.

Aaron Staley 20 Jul 2025 1:17 UTC
3 points
0
in reply to: tdko’s comment on: OpenAI Claims IMO Gold Medal
To put into perspective, there was only an 8% chance P3 would be this easy, putting substantial weight on the “unexpected” part being the problem being so easy. It’s also the first time in 20 years (5% chance) that 5 problems were of difficulty ⇐ 25.
Indeed, knowing that Gemini 2.5 Deep Think could solve an N25 (IMO result from Gemini 2.5 pro) and an A30 (known from Gemini 2.5 Deep think post), I’m somewhat less impressed. Only barriers were a medium-ish geometry problem (P2), which of course alpha geometry could solve and an easy combinatorics (P1).

The two most impressive things are, factoring this write up by Ralph Furman:

* OpenAI’s LLM was able to solve a medium level geometry problem. (guessing Deepmind just used alpha geometry again) - Furman thought this would be hard for informal methods.
* OpenAI’s LLM is strong enough to get the easy combinatorics problem (Furman noted informal methods would likely outperform formal ones on this one—just a matter if the LLM were smart enough)

Aaron Staley 19 Jul 2025 23:53 UTC
5 points
0
in reply to: Stephen Martin’s comment on: Agents lag behind AI 2027′s schedule
swe-bench pass @ 1 on Claude sonnet versions has been 33.4% (June − 3.5) → 49.0% (October) → 62.3% (Feb − 3.7) → 72.7% (May → 4). That’s practically linear at 3.5% gain/month. That would extrapolate to end of August at 83%.
With the leaderboard at ~75.2% on July 1, such an extrapolation also gets us to around 82%.

Aaron Staley 19 Jul 2025 23:37 UTC
6 points
1
on: Agents lag behind AI 2027′s schedule
Key points:
- OSworld likely has serious grading issues because it grades on dyanmically changing websites with static assumptions. It might saturate around 70% and likely will only get worse over time.
- It would have been ideal for the ai-2027 authors to have defined what they believed SOTA on swe-bench-verified was at time of writing. As you note, on swebench.com leaderboard it was 62.4%-65% (basically Claude 3.7 sonnet’s pass @1), while claude 3.7 sonnet with its parallel compute / learned scoring function was already at 70.3%. A 15% gain vs. a 22.6% gain in 4 months is quite a different projection—and today, we’re either at 73% (codex or sonnet, 1 shot), 75% (unchecked leaderboard), or 80.2% (Sonnet 4 with parallel compute).
- As noted elsewhere, the trendline from Sonnet models puts us at ~84% on August 31. So not much of a miss, but again this hedges on a top agentic model being released around end of Aug. (The argument against this though is that the public board trend is showing slowdown).
- swe-bench-verified leaderboard is ~3 weeks behind submissions as far as I can tell. (history suggests this too). So there’s really 2 months to get from 75.2% to 85%.
That said, I am also skeptical we’ll hit 85% by end of August (pass@1 definition) - new Claude is unlikely to ship by then and the fact that neither Opus outperforms Sonnet and Codex barely outperforms O3 suggest the remaining tasks are quite difficult.

Aaron Staley 19 Jul 2025 18:47 UTC
1 point
0
in reply to: Hopenope’s comment on: Burny’s Shortform
Good point. This does update me downward on Deep Think outperforming matharena’s gemini-2.5-pro IMO run as it is possible Deep Think internally was doing a similar selection process to begin with. Difficult to know without randomly sampling gemini-2.5-pro’s answers and seeing how much the best-of-n selection lifted its score.

Aaron Staley 19 Jul 2025 18:12 UTC
3 points
0
in reply to: Thane Ruthenis’s comment on: Burny’s Shortform
Note that the likely known SOTA was even higher than 30%. Google never released Gemini 2.5 Pro Deep think, which they claimed scored 49% on the USAMO (vs. 34.5% for Gemini 2.5-05-06). Little hard to convert this to an implied IMO score (especially because matharena.ai has Gemini 2.5 oddly having a significantly lower USAMO score for the June model (24%), though similar IMO score (~31.5%) but my guess is Deep Think would get somewhere between 37% and 45% on the IMO. 81% remains a huge jump of course.
Hmm, I think there’s a systemic EMH failure here.
Perhaps, perhaps not. Substantial weight was on the “no one bothers” case—no one was reporting such high scores on the USAMO (pretty similar difficulty to IMO) and the market started dropping rapidly after the USAMO date. Note that we were still at 50% odds of IMO gold a week ago—but the lack of news of anyone trying drove it down to ~26%.
Interestingly, I can find write-ups roughly predicting order of AI difficulty. Looking at gemini-2.5 pro’s result so far, using alphageometry would have guaranteed problem 2, so assuming Pro Deep Think only boosted performance on the non-geometric problems, we’d be at a 58% using deep think + alphageometry, giving Bronze and close to Silver. I think it was reasonable to assume an extra 4+ months (2 months timeline, 2 months labs being ahead of release) + more compute would have given the 2 more points to get silver.
What is surprising is that generalist LLM got better at combinatorics (problem 1) and learned to solve geometry problems well. I’m neither an AI nor math competition expert, so can’t opine whether this is a qualitative gain or just an example of a company targeting these specific problems (lots of training on math + lots of inference).

Aaron Staley 19 Jul 2025 0:06 UTC
12 points
3
in reply to: CAC’s comment on: Cole Wyeth’s Shortform
If I understand correctly, Claude’s pass@X benchmarks mean multiple sampling and taking the best result. This is valid so long as compute cost isn’t exceeding equivalent cost of an engineer.
codex’s pass @ 8 score seems to be saying “the correct solution was present in 8 attempts, but the model doesn’t actually know what the correct result is”. That shouldn’t count.

Aaron Staley 24 May 2025 19:04 UTC
3 points
0
in reply to: Thomas Kwa’s comment on: Cole Wyeth’s Shortform
That’s a bit higher than I would have guessed. I compared the known data points that have SWE-bench and METR medians (sonnet 3.5,3.6,3.7, o1, o3, o4-mini) and got an r^2 = 0.96 model assuming linearity between log(METR_median) and log(swe-bench-error).

That gives an estimate more like 110 minutes for an Swe-bench score of 72.7%. Which works out to a sonnet doubling time of ~3.3 months. (If I throw out o4-mini, estimator is ~117 minutes.. still below 120)

Also would imply an 85% swe-bench score is something like a 6-6.5 hour METR median.