Current LLMs are capable of solving novel problems when the user does most the work: when the user lays the groundwork and poses the right question for the LLM to answer.
So, if we can get LLMs to lay the groundwork and pose the right questions then we’ll have autonomous scientists in whatever fields LLMs are OK at problem solving.
This seems like something LLMs will learn to do as inference-time compute is scaled up. Reasoners benefit from coming up with sub-problems whose solutions can be built atop of to solve the problem posed by the user.
LLMs will learn that in order to solve difficult questions, they must pose and solve novel sub-questions.
So, once given an interesting research problem, the LLM will hum away for days doing good, often-novel work.
I think the argument you’re making is that since LLMs can make eps > 0 progress, they can repeat it N times to make unbounded progress. But this is not the structure of conceptual insight as a general rule. Concretely, it fails for the architectural reasons I explained in the original post.
Here’s an attempt at a clearer explanation of my argument:
I think the ability to autonomously find novel problems to solve will emerge as reasoning models scale up. It will emerge because it is instrumental to solving difficult problems.
Imagine an RL environment in which the LLM being trained is tasked with solving somewhat difficult open math problems (solutions verified using autonomous proof verification). It fails and fails at most of them until it learns to focus on making marginal progress: tackling simpler cases, working on tangentially-related problems, etc. These instrumental solutions are themselves often novel, meaning that the LLM will have become able to pose novel, interesting, somewhat important problems autonomously. And this will scale to something like a fully autonomous, very much superhuman researcher.
This is how it often works in humans. We work on a difficult problem, find novel results on the way there. The LLM would likely be uncertain of whether these results are truly novel but this is how it works with humans too. The system can do some DeepResearch / check with relevant experts if it’s important.
Of course, I’m working from my parent-comment’s position that LLMs are in fact already capable of solving novel problems, just not posing them and doing the requisite groundwork.
I think the ability to autonomously find novel problems to solve will emerge as reasoning models scale up. It will emerge because it is instrumental to solving difficult problems.
This of course is not a sufficient reason. (Demonstration: telepathy will emerge [as evolution improves organisms] because it is instrumental to navigating social situations.) It being instrumental means that there is an incentive—or to be more precise, a downward slope in the loss function toward areas of model space with that property—which is one required piece, but it also must be feasible. E.g., if the parameter space doesn’t have any elements that are good at this ability, then it doesn’t matter whether there’s a downward slope.
Fwiw I agree with this:
Current LLMs are capable of solving novel problems when the user does most the work: when the user lays the groundwork and poses the right question for the LLM to answer.
… though like you I think posing the right question is the hard part, so imo this is not very informative.
I don’t agree with the underlying assumption then—I don’t think LLMs are capable of solving difficult novel problems, unless you include a nearly-complete solution as part of the groundwork.
Current LLMs are capable of solving novel problems when the user does most the work: when the user lays the groundwork and poses the right question for the LLM to answer.
So, if we can get LLMs to lay the groundwork and pose the right questions then we’ll have autonomous scientists in whatever fields LLMs are OK at problem solving.
This seems like something LLMs will learn to do as inference-time compute is scaled up. Reasoners benefit from coming up with sub-problems whose solutions can be built atop of to solve the problem posed by the user.
LLMs will learn that in order to solve difficult questions, they must pose and solve novel sub-questions.
So, once given an interesting research problem, the LLM will hum away for days doing good, often-novel work.
I think the argument you’re making is that since LLMs can make eps > 0 progress, they can repeat it N times to make unbounded progress. But this is not the structure of conceptual insight as a general rule. Concretely, it fails for the architectural reasons I explained in the original post.
Here’s an attempt at a clearer explanation of my argument:
I think the ability to autonomously find novel problems to solve will emerge as reasoning models scale up. It will emerge because it is instrumental to solving difficult problems.
Imagine an RL environment in which the LLM being trained is tasked with solving somewhat difficult open math problems (solutions verified using autonomous proof verification). It fails and fails at most of them until it learns to focus on making marginal progress: tackling simpler cases, working on tangentially-related problems, etc. These instrumental solutions are themselves often novel, meaning that the LLM will have become able to pose novel, interesting, somewhat important problems autonomously. And this will scale to something like a fully autonomous, very much superhuman researcher.
This is how it often works in humans. We work on a difficult problem, find novel results on the way there. The LLM would likely be uncertain of whether these results are truly novel but this is how it works with humans too. The system can do some DeepResearch / check with relevant experts if it’s important.
Of course, I’m working from my parent-comment’s position that LLMs are in fact already capable of solving novel problems, just not posing them and doing the requisite groundwork.
This of course is not a sufficient reason. (Demonstration: telepathy will emerge [as evolution improves organisms] because it is instrumental to navigating social situations.) It being instrumental means that there is an incentive—or to be more precise, a downward slope in the loss function toward areas of model space with that property—which is one required piece, but it also must be feasible. E.g., if the parameter space doesn’t have any elements that are good at this ability, then it doesn’t matter whether there’s a downward slope.
Fwiw I agree with this:
… though like you I think posing the right question is the hard part, so imo this is not very informative.
I don’t agree with the underlying assumption then—I don’t think LLMs are capable of solving difficult novel problems, unless you include a nearly-complete solution as part of the groundwork.