I’d say my major takeaways, assuming this research scales (it was only done on GPT-2, and we already knew it couldn’t generalize.)
Gary Marcus was right about LLMs mostly not reasoning outside the training distribution, and this updates me more towards “LLMs probably aren’t going to be godlike, or be nearly as impactful as LW say it is.”
Be more skeptical of AI progress leading to big things, and in general unless reality can simply be memorized, scaling probably won’t work to automate the economy. More generally, this updates me towards longer timelines, and longer tails on those timelines.
Be slightly more pessimistic on AI safety, since LLMs have a bunch of nice properties, and future AI probably will have less nice properties, though alignment optimism mostly doesn’t depend on LLMs.
AI governance gets a lucky break, since they only have to regulate misuse, and even though their threat model isn’t likely or even probable to be realized, it’s still nice that we don’t have to deal with the disruptive effects of AI now.
I am sharing this since I think it will change your view on how much to update on this paper (I should have shared this initially). Here’s what the paper author said on X:
Clarifying two things:
Model is simple transformer for science, not a language model (or large by standards today)
The model can learn new tasks (via in-context learning), but can’t generalize to new task families
I would be thrilled if this work was important for understanding AI safety and fairness, but it is the start of a scientific direction, not ready for policy conclusions. Understanding what task families a true LLM is capable of would be fascinating and more relevant to policy!
So, with that, I said:
I hastily thought the paper was using language models, so I think it’s important to share this. A follow-up paper using a couple of ‘true’ LLMs at different model scales would be great. Is it just interpolation? How far can the models extrapolate?
I’d say my major takeaways, assuming this research scales (it was only done on GPT-2, and we already knew it couldn’t generalize.)
Gary Marcus was right about LLMs mostly not reasoning outside the training distribution, and this updates me more towards “LLMs probably aren’t going to be godlike, or be nearly as impactful as LW say it is.”
Be more skeptical of AI progress leading to big things, and in general unless reality can simply be memorized, scaling probably won’t work to automate the economy. More generally, this updates me towards longer timelines, and longer tails on those timelines.
Be slightly more pessimistic on AI safety, since LLMs have a bunch of nice properties, and future AI probably will have less nice properties, though alignment optimism mostly doesn’t depend on LLMs.
AI governance gets a lucky break, since they only have to regulate misuse, and even though their threat model isn’t likely or even probable to be realized, it’s still nice that we don’t have to deal with the disruptive effects of AI now.
I am sharing this since I think it will change your view on how much to update on this paper (I should have shared this initially). Here’s what the paper author said on X:
So, with that, I said:
To which @Jozdien replied: