AIs being bad at AI research says nothing about acceleration
Lots of people are trying to make AI good at AI research. How are they doing?
One way to measure this is to assume AIs are gradually doing more and more complex tasks independently. Eventually it would grow to doing whole research projects. Something like this, but for software engineering instead of research, is captured in the METR “time horizons” benchmark.
I think extending this line of thinking to forecasting progress in AI research is wrong. Instead, a better way to accelerate AI research for the time being is combining AI and people to do research together, in a way that uses the complementary strengths of each; with the goal of the researcher’s feedback loops shortening.
What is difficult to automate in AI research?
If you decompose a big AI research project into tasks, there’s lots of “dark matter” that does not neatly fit into any category. Some examples are given in Large-Scale Projects Stress Deep Cognitive Skills, which is a much better post than mine.
But I think that the most central argument is: the research process involves taste, coming up with ideas, and various such intangibles that we don’t really know how to train for.
The labs are trying to make superhuman AI researchers. We do not yet know how to do it, which means at least some of our ideas are lacking. To improve our ideas, we need either:
(the proper way) conceptual advances in machine learning;
(the way it’s actually going to get done) reinforcement learning on the idea->code->experiment->result process, to figure out which ideas are good.
Measuring which ideas are good is difficult; it requires sparse empirical outcomes that happen long after the idea is formulated. How can we accelerate this process?
I want to make two claims:
There are large gains from accelerating AI researchers.
Much more importantly, those gains are achievable without inventing new things in machine learning.
The careful reader might ask, ok, this sounds fine in the abstract, but I don’t understand what exactly the lab is doing then, if not “automate AI research as a whole”? How is this different from making autonomous AI researchers directly?
Here is a list of tasks that would be extremely valuable if we wanted to make the research feedback loops faster.
implementing instructions of varying level of detail into code efficiently;
extrapolating user intent and implementing the correct thing;
relatedly: learning user intent from working with a researcher over time;
given code, running experiments autonomously, fixing minor deployment issues;
checking for bugs and suspicious logic in the code;
observing and pinpointing anomalies in the data;
monitoring experiments, reporting updates, and raising alarm when something is off; and so on.
I believe all of these tasks possess properties that make them attractive to attack directly.
They consume a significant amount of time of a researcher (or add communication overhead if we add people focused on research engineering);
There are clear ways to generate many datapoints + labels / reward functions for each of these tasks;
Alternatively, these are done by people typing into a keyboard, so labs can do imitation learning by collecting all the human actions from their own researchers.
This seems easier than automating the full research process. If labs have the goal of speeding up the lab’s ability to do AI research as opposed to other goals, they are probably doing these things; and measuring the ability of AIs to do research autonomously is not going to give a good grasp on how quickly the lab is accelerating.
AIs being bad at AI research says nothing about acceleration
Lots of people are trying to make AI good at AI research. How are they doing?
One way to measure this is to assume AIs are gradually doing more and more complex tasks independently. Eventually it would grow to doing whole research projects. Something like this, but for software engineering instead of research, is captured in the METR “time horizons” benchmark.
I think extending this line of thinking to forecasting progress in AI research is wrong. Instead, a better way to accelerate AI research for the time being is combining AI and people to do research together, in a way that uses the complementary strengths of each; with the goal of the researcher’s feedback loops shortening.
What is difficult to automate in AI research?
If you decompose a big AI research project into tasks, there’s lots of “dark matter” that does not neatly fit into any category. Some examples are given in Large-Scale Projects Stress Deep Cognitive Skills, which is a much better post than mine.
But I think that the most central argument is: the research process involves taste, coming up with ideas, and various such intangibles that we don’t really know how to train for.
The labs are trying to make superhuman AI researchers. We do not yet know how to do it, which means at least some of our ideas are lacking. To improve our ideas, we need either:
(the proper way) conceptual advances in machine learning;
(the way it’s actually going to get done) reinforcement learning on the idea->code->experiment->result process, to figure out which ideas are good.
Measuring which ideas are good is difficult; it requires sparse empirical outcomes that happen long after the idea is formulated. How can we accelerate this process?
I want to make two claims:
There are large gains from accelerating AI researchers.
Much more importantly, those gains are achievable without inventing new things in machine learning.
The careful reader might ask, ok, this sounds fine in the abstract, but I don’t understand what exactly the lab is doing then, if not “automate AI research as a whole”? How is this different from making autonomous AI researchers directly?
Here is a list of tasks that would be extremely valuable if we wanted to make the research feedback loops faster.
implementing instructions of varying level of detail into code efficiently;
extrapolating user intent and implementing the correct thing;
relatedly: learning user intent from working with a researcher over time;
given code, running experiments autonomously, fixing minor deployment issues;
checking for bugs and suspicious logic in the code;
observing and pinpointing anomalies in the data;
monitoring experiments, reporting updates, and raising alarm when something is off; and so on.
I believe all of these tasks possess properties that make them attractive to attack directly.
They consume a significant amount of time of a researcher (or add communication overhead if we add people focused on research engineering);
There are clear ways to generate many datapoints + labels / reward functions for each of these tasks;
Alternatively, these are done by people typing into a keyboard, so labs can do imitation learning by collecting all the human actions from their own researchers.
This seems easier than automating the full research process. If labs have the goal of speeding up the lab’s ability to do AI research as opposed to other goals, they are probably doing these things; and measuring the ability of AIs to do research autonomously is not going to give a good grasp on how quickly the lab is accelerating.