This line of research makes me question one thing: “Is the alignment community over-updating on how scale impacts generalization?”
It remains to be seen how well models will generalize outside of their training distribution (interpolation vs extrapolation).
In other words, when people say that GPT-4 (and other LLMs) can generalize, I think they need to be more careful about what they really mean. Is it doing interpolation or extrapolation? Meaning, yes, GPT-4 can do things like write a completely new poem, but poems and related stuff were in its training distribution! So, you can say it is generalizing, but I think it’s a much weaker form of generalization than what people really imply when they say generalization. A stronger form of generalization would be an AI that can do completely new tasks that are actually outside of its training distribution.
Now, at this point, you might say, “yes, but we know that LLMs learn functions and algorithms to do tasks, and as you scale up and compress more and more data, you will uncover more meta-algorithms that can do this kind of extrapolation/tasks outside of the training distribution.”
Well, two things:
It remains to be seen when or if this will happen in the current paradigm (no matter how much you scale up).
It’s not clear to me how well things like induction heads continue to work on things that are outside of their training distribution. If they don’t adapt well, then it may be the same thing for other algorithms. What this would mean in practice, I’m not sure. I’ve been looking at relevant papers, but haven’t foundan answeryet.
This brings me to another point: it also remains to be seen how much it will matter in practice, given that models are trained on so much data and things like online learning are coming. Scaffolding specialized AI models, and new innovations might make such a limitation not big of a deal if there is one.
Also, perhaps most of the important capabilities come from interpolation. Perhaps intelligence is largely just interpolation? You just need to interpolate and push the boundaries of capability one step at a time, iteratively, like a scientist conducting experiments would. You just need to integrate knowledge as you interact with the world.
But what of brilliant insights from our greatest minds? Is it just recursive interpolation+small_external_interactions? Is there something else they are doing to get brilliant insights? Would AGI still ultimately be limited in the same way (even if it can run many of these genius patterns in parallel)?
Title: Is the alignment community over-updating on how scale impacts generalization?
So, apparently, there’s a rebuttal to the recent Google generalization paper (and also worth pointing out it wasn’t done with language models, just sinoïsodal functions, not language):
But then, the paper author responds:
This line of research makes me question one thing: “Is the alignment community over-updating on how scale impacts generalization?”
It remains to be seen how well models will generalize outside of their training distribution (interpolation vs extrapolation).
In other words, when people say that GPT-4 (and other LLMs) can generalize, I think they need to be more careful about what they really mean. Is it doing interpolation or extrapolation? Meaning, yes, GPT-4 can do things like write a completely new poem, but poems and related stuff were in its training distribution! So, you can say it is generalizing, but I think it’s a much weaker form of generalization than what people really imply when they say generalization. A stronger form of generalization would be an AI that can do completely new tasks that are actually outside of its training distribution.
Now, at this point, you might say, “yes, but we know that LLMs learn functions and algorithms to do tasks, and as you scale up and compress more and more data, you will uncover more meta-algorithms that can do this kind of extrapolation/tasks outside of the training distribution.”
Well, two things:
It remains to be seen when or if this will happen in the current paradigm (no matter how much you scale up).
It’s not clear to me how well things like induction heads continue to work on things that are outside of their training distribution. If they don’t adapt well, then it may be the same thing for other algorithms. What this would mean in practice, I’m not sure. I’ve been looking at relevant papers, but haven’t found an answer yet.
This brings me to another point: it also remains to be seen how much it will matter in practice, given that models are trained on so much data and things like online learning are coming. Scaffolding specialized AI models, and new innovations might make such a limitation not big of a deal if there is one.
Also, perhaps most of the important capabilities come from interpolation. Perhaps intelligence is largely just interpolation? You just need to interpolate and push the boundaries of capability one step at a time, iteratively, like a scientist conducting experiments would. You just need to integrate knowledge as you interact with the world.
But what of brilliant insights from our greatest minds? Is it just recursive interpolation+small_external_interactions? Is there something else they are doing to get brilliant insights? Would AGI still ultimately be limited in the same way (even if it can run many of these genius patterns in parallel)?
Or perhaps as @Nora Belrose mentioned to me: “Perhaps we should queer the interpolation-extrapolation distinction.”