How could one test this? By using a different similarly scaled pretrain (think of Talkie raised on pre-1930 data, but far bigger-scaled and focused on preventing contamination)?
Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important.
I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.
I think there are probably two things going on here.
LLM’s learn during pretraining sequences of thought patterns that are usedul for thinking about and verifying proofs.
A lot of the time the best way to solve a problem is just to try everything somewhat intelligently and see what sticks then prune after that. In academia the seeing what sticks is largely determined by peer review which obviously isn’t perfect but gives a noisy verification signal such that over decades you get a final literature that is more correct than it was at the start. The economy is roughly the same with businesses. When OpenAI sends thousands of models at a problem they’re essentially hoping for a similar roughly evolutionary process to play out.
Why I think verifiability for LLM’s matter so much is that they operate at a wholly different rate than we do so there’s very few ways for LLM’s to get the kind of verifiable feedback which human society typically relies on. Not to say you can’t approximate these social structures with other models but for empirical sciences there is no clear path to immediately verify a theory and there’s not much evidence yet imo that models are much better at theory building than humans when verification is slow and you only get to try a few things.
Much easier to do with open-weight models where we can get probes, but the eliciting 50% performance tells us couple things. Pre-training (mid-training too) gave the token generation capabilities, but the post-training for 4o did not have particularly focused evaluations for ARC-AGI.
Post-training is best for steering token outputs. By generating 8k python transformation per problem and selecting the best, you’re getting the best of what could have been an over-fitting for a general model versus capabilities for ARC-AGI-1. RLVR makes verifiable problems far easier for steering during training!
IIUC, The method Ryan describes is “generate a bunch of programs” and then “filter those”. He wrote:
> The distribution of programs you are searching over [after generating a bunch of them] has to be pretty close to the right program for Best-of-6k to work at all: if you did best-of-6k for random python programs, this would not work!
I agree that it’s way better than random… but is it better than the human literature? E.g., If you asked me to filter 6,000 NLP papers, I’d expect a great many of them to be “basically true” and a great many more to be “a good effort but ultimately just very wrong”.
Is that the sense in which 6,000 programs he generated and filtered… match the distribution of human output? Like Steven says here, it “spits out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans.” Am I thinking about this right?
I have a few issues with this.
How could one test this? By using a different similarly scaled pretrain (think of Talkie raised on pre-1930 data, but far bigger-scaled and focused on preventing contamination)?
The world saw Ryan Greenblatt elicit 50% performance on ARC-AGI-1 from GPT-4o. How does this method interact with your arguments?
You could do a contaminated pretrain by making different fractions of the train set artificially bad math papers and see if it breaks math ability?
I really love this counterfactual intervention!
Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important.
I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.
I think there are probably two things going on here.
LLM’s learn during pretraining sequences of thought patterns that are usedul for thinking about and verifying proofs.
A lot of the time the best way to solve a problem is just to try everything somewhat intelligently and see what sticks then prune after that. In academia the seeing what sticks is largely determined by peer review which obviously isn’t perfect but gives a noisy verification signal such that over decades you get a final literature that is more correct than it was at the start. The economy is roughly the same with businesses. When OpenAI sends thousands of models at a problem they’re essentially hoping for a similar roughly evolutionary process to play out.
Why I think verifiability for LLM’s matter so much is that they operate at a wholly different rate than we do so there’s very few ways for LLM’s to get the kind of verifiable feedback which human society typically relies on. Not to say you can’t approximate these social structures with other models but for empirical sciences there is no clear path to immediately verify a theory and there’s not much evidence yet imo that models are much better at theory building than humans when verification is slow and you only get to try a few things.
Much easier to do with open-weight models where we can get probes, but the eliciting 50% performance tells us couple things. Pre-training (mid-training too) gave the token generation capabilities, but the post-training for 4o did not have particularly focused evaluations for ARC-AGI.
Post-training is best for steering token outputs. By generating 8k python transformation per problem and selecting the best, you’re getting the best of what could have been an over-fitting for a general model versus capabilities for ARC-AGI-1. RLVR makes verifiable problems far easier for steering during training!
I also have question2.
IIUC, The method Ryan describes is “generate a bunch of programs” and then “filter those”. He wrote:
> The distribution of programs you are searching over [after generating a bunch of them] has to be pretty close to the right program for Best-of-6k to work at all: if you did best-of-6k for random python programs, this would not work!
I agree that it’s way better than random… but is it better than the human literature? E.g., If you asked me to filter 6,000 NLP papers, I’d expect a great many of them to be “basically true” and a great many more to be “a good effort but ultimately just very wrong”.
Is that the sense in which 6,000 programs he generated and filtered… match the distribution of human output? Like Steven says here, it “spits out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans.” Am I thinking about this right?