I think it can hardly be denied that there was a slowdown in scaling up pre-training. We have not seen an intelligence jump like the one from GPT-3.5 to GPT-4 again. Most of the progress comes from reasoning RL now, not from Chinchilla-scaling.
Do you mean “we have not seen an intelligence jump like from 3.5 to 4 again” unconditionally? Then I’d disagree, I think the newest GPT-pro models are a greater jump over 4 than 4 is over 3.5.
Or do you mean we have not seen a similar jump in pretraining capabilities? That is plausible but I wonder how to assess that.
That sounds very plausible to me, but how would you evaluate it without access to the base model or the instance of the model before it was trained to reason?
GPT-5 Instant doesn’t do dedicated reasoning. It is probably still able to reason sometimes in the actual reply block (it did so in the past in the seahorse emoji case), so there seems to be some degree of RLVR involved, but even with that advantage, GPT-5 Instant was not a big improvement over GPT-4o.
I think it can hardly be denied that there was a slowdown in scaling up pre-training. We have not seen an intelligence jump like the one from GPT-3.5 to GPT-4 again. Most of the progress comes from reasoning RL now, not from Chinchilla-scaling.
Do you mean “we have not seen an intelligence jump like from 3.5 to 4 again” unconditionally? Then I’d disagree, I think the newest GPT-pro models are a greater jump over 4 than 4 is over 3.5.
Or do you mean we have not seen a similar jump in pretraining capabilities? That is plausible but I wonder how to assess that.
In pretraining. I think it’s pretty clear that the change from GPT-4o to GPT-5 Instant was rather incremental. Like the change from GPT-4 to GPT-4o.
That sounds very plausible to me, but how would you evaluate it without access to the base model or the instance of the model before it was trained to reason?
GPT-5 Instant doesn’t do dedicated reasoning. It is probably still able to reason sometimes in the actual reply block (it did so in the past in the seahorse emoji case), so there seems to be some degree of RLVR involved, but even with that advantage, GPT-5 Instant was not a big improvement over GPT-4o.