I’ve been working towards automated research (for safety) for a long time. After a ton of reflection and building in this direction, I’ve landed on a similar opinion as presented in this post.
I think LLM scaffolds will solve some problems, but I think they will be limited in ways that make it hard to solve incredibly hard problems. You can claim that LLMs can just use a scratchpad as a form of continual online learning, it feels like this will hit limits. Information loss and being able to internalize new information feels like bottlenecks.
Scale will help, but unclear how far it will go and clearly not economical.
That said, I still think automated research for safety is underinvested.
I’ve been working towards automated research (for safety) for a long time. After a ton of reflection and building in this direction, I’ve landed on a similar opinion as presented in this post.
I think LLM scaffolds will solve some problems, but I think they will be limited in ways that make it hard to solve incredibly hard problems. You can claim that LLMs can just use a scratchpad as a form of continual online learning, it feels like this will hit limits. Information loss and being able to internalize new information feels like bottlenecks.
Scale will help, but unclear how far it will go and clearly not economical.
That said, I still think automated research for safety is underinvested.