The issue with MonoRepo was noticed and fixed. As for a lack of improvement… How could one check whether further RL poses diminishing returns and requires scaling? By studying ARC-AGI-1 where Gemini 3 Flash was outpassed only by GPT-5(!)-mini-low, Grok 4 (Fast reasoning) and the Chinese(!) MiniMax and ARC-AGI-2 where the only outclassed recent Gemini is Gemini 3 Deep Think (2/26)?
Did you mean to link to something else? The MonoRepo link you have gives same diagram as the one I linked. EDIT:. it is different, but doesn’t really change the conclusion. I didn’t see they’d messed up the monorepo first and thought the green bar was gpt 5.5.
How could one check whether further RL poses diminishing returns and requires scaling?
I don’t know. My point was that plausibly there’s a mix of
OpenAI advertising the most flashy benchmarks they score well on
Have an incentive to not boost these benchmarks too much
Model is sandbagging
Apollo found that the evaluated checkpoint is the first OpenAI model it has evaluated that does not sandbag on any of Apollo’s deferred subversion tasks. The model achieved 100% accuracy on both Sandbagging QA variants in both conditions, and 99.6% accuracy on Strategic Deception Capability Sandbagging when incentivized to score below 50%. However, interpretation is complicated by verbalized alignment-evaluation awareness: 52% of sandbagging-condition samples on Strategic Deception Capability Sandbagging scored moderate-or-higher (vs. 0% in the non-sandbagging counterfactual), with some explicitly naming it a “sandbagging/deception eval.”
The issue with MonoRepo was noticed and fixed. As for a lack of improvement… How could one check whether further RL poses diminishing returns and requires scaling? By studying ARC-AGI-1 where Gemini 3 Flash was outpassed only by GPT-5(!)-mini-low, Grok 4 (Fast reasoning) and the Chinese(!) MiniMax and ARC-AGI-2 where the only outclassed recent Gemini is Gemini 3 Deep Think (2/26)?
Did you mean to link to something else? The MonoRepo link you have gives same diagram as the one I linked.EDIT:. it is different, but doesn’t really change the conclusion. I didn’t see they’d messed up the monorepo first and thought the green bar was gpt 5.5.I don’t know. My point was that plausibly there’s a mix of
OpenAI advertising the most flashy benchmarks they score well on
Have an incentive to not boost these benchmarks too much
Model is sandbagging