That wasn’t true in the original time horizon paper. There’s a massive drop-off after 1 hour, a conspicuous gap at 2 hours for all models, and the vast majority of success after that is due to 3 problems.
Note the gap at ~ 2 hours and the presence of the “L” at 4-8 hours in unrelated models! Could it just be misclassification of those 3 problems? Maybe they’re the type of problems that are tough for humans but relatively easy for AIs? Interesting to see the pattern smooth out in recent tests. Of the 2+ hour problems with significant success %, how many had no human baseline successes?
It would be nice if the authors can give us more info about the 2+ hour problems that the AI solved. Very interesting that the best performance in the original Re-Bench paper (Nov 2024) never came close to the human 8 hour performance, but it appears that AIs have almost 20% success rate on one of them by the time horizon paper (Mar 2025).
This plot collects the performance of many humans. I think the average of all 14 AI plots would also look smooth.
That wasn’t true in the original time horizon paper. There’s a massive drop-off after 1 hour, a conspicuous gap at 2 hours for all models, and the vast majority of success after that is due to 3 problems.
Note the gap at ~ 2 hours and the presence of the “L” at 4-8 hours in unrelated models! Could it just be misclassification of those 3 problems? Maybe they’re the type of problems that are tough for humans but relatively easy for AIs? Interesting to see the pattern smooth out in recent tests. Of the 2+ hour problems with significant success %, how many had no human baseline successes?
It would be nice if the authors can give us more info about the 2+ hour problems that the AI solved. Very interesting that the best performance in the original Re-Bench paper (Nov 2024) never came close to the human 8 hour performance, but it appears that AIs have almost 20% success rate on one of them by the time horizon paper (Mar 2025).