I find this possible though it’s not my median scenario to say the least. But I am also not sure I can put the probability of such a fast development below 10%.
Main cruxes:
I am not so sure that “automating AI research” is going to speed up development by orders of magnitude.
My experience is that cracked AI engineers can implement any new paper / well specified research idea in a matter of hours. So speeding up the coding can’t be the huge speedup to R&D.
The bottleneck seems to be: A.) Coming up with good research ideas.
b.) Finding the precise formulation of that idea that makes most sense/works.
LLMs so far are bad at both. So I currently only see them scouring the immediate neighbourhood of existing ideas, to eke out incremental progress in the current paradigm.
Is that enough? Is an LLM building on a base model that has a loss close to the irreducible loss AGI? I.e. does accelerating this improvement matter for the transition to AGI and superintelligence?
I think not even the authors believe that. So they make the leap of faith that accelerated research will make a qualitative difference too. I think there are additional gaps between human cognition and LLMs beyond recursive reasoning in latent space and sample efficiency.
Will all those gaps be closed in the next few years?
I agree, and am also confused with the idea that LLMs will be able to bootstrap something more intelligent.
My day job is a technical writer. I also do a bit of DevOps stuff. This combo ought to be the most LLM-able of all, yet I frequently find myself giving up on trying to tease out an answer from an LLM. And I’m far from the edge of my field!
So how exactly do people on the edge of their field make better use of LLMs, and expect to make qualitative improvements?
Feels like it’ll have to be humans to make algorithmic improvements, at least up until a point.
I would presume that the process of the AI improvement can be also modelled as: A.) Coming up with good research ideas. B.) Finding the precise formulation of that idea that makes most sense/works. C.) Implementation of the idea.
If you claim that C) only “takes hours”—then with the AI Coder it takes seconds instead (nowadays agents work correctly only 50-70% of the time, hence a programmer indeed has to spent these couple of hours).
Then the loop becomes tighter—a single iteration takes a few hours less.
Let’s assume there’s a very creative engineer who can come up with a couple ideas a day. What is the B-step? Finding the formulation means e.g. getting the math equations, right? The LLMs become superhuman at math this year already. If they’re superhuman then the loop becomes tighter.
Then instead of spending a day on an idea (a few hours of implementation), you test a bunch of them a day.
Also—the A) can probably get automated too, with a framework in which you make the model read all the literature and provide combinations of ideas which you then filter out. Each new model makes the propositions more relevant.
So all 3 steps get semi-automated (and gradually tighten with next models releases), where the human’s role boils down to filtering things out—it’s the “taste” quality, which Kokotajlo mentions.
Let’s instead assume a top engineer has a really consequential idea every couple of months. Now what?
Speeding up implementation just means that you test more of the less promising ideas.
Speeding up feedback might mean that you can hone in on the really good ideas faster, but does this actually happen if you don’t do the coding and don’t do the math?
I find this possible though it’s not my median scenario to say the least. But I am also not sure I can put the probability of such a fast development below 10%.
Main cruxes:
I am not so sure that “automating AI research” is going to speed up development by orders of magnitude.
My experience is that cracked AI engineers can implement any new paper / well specified research idea in a matter of hours. So speeding up the coding can’t be the huge speedup to R&D.
The bottleneck seems to be:
A.) Coming up with good research ideas.
b.) Finding the precise formulation of that idea that makes most sense/works.
LLMs so far are bad at both. So I currently only see them scouring the immediate neighbourhood of existing ideas, to eke out incremental progress in the current paradigm.
Is that enough? Is an LLM building on a base model that has a loss close to the irreducible loss AGI? I.e. does accelerating this improvement matter for the transition to AGI and superintelligence?
I think not even the authors believe that. So they make the leap of faith that accelerated research will make a qualitative difference too. I think there are additional gaps between human cognition and LLMs beyond recursive reasoning in latent space and sample efficiency.
Will all those gaps be closed in the next few years?
I agree, and am also confused with the idea that LLMs will be able to bootstrap something more intelligent.
My day job is a technical writer. I also do a bit of DevOps stuff. This combo ought to be the most LLM-able of all, yet I frequently find myself giving up on trying to tease out an answer from an LLM. And I’m far from the edge of my field!
So how exactly do people on the edge of their field make better use of LLMs, and expect to make qualitative improvements?
Feels like it’ll have to be humans to make algorithmic improvements, at least up until a point.
I would presume that the process of the AI improvement can be also modelled as:
A.) Coming up with good research ideas.
B.) Finding the precise formulation of that idea that makes most sense/works.
C.) Implementation of the idea.
If you claim that C) only “takes hours”—then with the AI Coder it takes seconds instead (nowadays agents work correctly only 50-70% of the time, hence a programmer indeed has to spent these couple of hours).
Then the loop becomes tighter—a single iteration takes a few hours less.
Let’s assume there’s a very creative engineer who can come up with a couple ideas a day.
What is the B-step? Finding the formulation means e.g. getting the math equations, right? The LLMs become superhuman at math this year already. If they’re superhuman then the loop becomes tighter.
Then instead of spending a day on an idea (a few hours of implementation), you test a bunch of them a day.
Also—the A) can probably get automated too, with a framework in which you make the model read all the literature and provide combinations of ideas which you then filter out. Each new model makes the propositions more relevant.
So all 3 steps get semi-automated (and gradually tighten with next models releases), where the human’s role boils down to filtering things out—it’s the “taste” quality, which Kokotajlo mentions.
Let’s instead assume a top engineer has a really consequential idea every couple of months. Now what?
Speeding up implementation just means that you test more of the less promising ideas.
Speeding up feedback might mean that you can hone in on the really good ideas faster, but does this actually happen if you don’t do the coding and don’t do the math?