Just to make sure we’re on the same page, Fig. 4.1 was about training the model by gradient descent, not in-context learning. I’m generally somewhat averse to the term “in-context learning” in the first place; I’m skeptical that we should think of it as “learning” at all (as opposed to, say, “pointing to a certain task”). I wish people would reserve the term “learning” for the weight updates (when we’re talking about LLMs), at least in the absence of more careful justification than what I’ve seen.
In particular, instead of titling the paper “Language Models are Few-Shot Learners”, I wish they had titled it “Language Models Can Do Lots of Impressive Things Without Fine-Tuning”.
But Fig. 4.1 of that paper is definitely about actual learning.
In order for it to be a strong counterargument to this example, it should be the case that an LLM can learn to play Go on superhuman level …
I think there are so many disanalogies between LLMs-playing-Go and humans-playing-Go that it’s not even worth thinking about. ¯\_(ツ)_/¯ For example, humans can “visualize” things but LLMs can’t (probably). But OTOH, maybe future multi-modal next-gen LLMs will be able to.
More generally, I haven’t seen any simple comparison that provides air-tight evidence either way on sample-efficiency of deep learning versus human brains (and “deep learning” is itself is big tent—presumably some model types & sizes are more sample-efficient than others).
As it happens, I do believe that human brains are more sample efficient than any deep learning model. But my reasons for believing that are pretty indirect and I don’t want to talk about them.
Just to make sure we’re on the same page, Fig. 4.1 was about training the model by gradient descent, not in-context learning. I’m generally somewhat averse to the term “in-context learning” in the first place; I’m skeptical that we should think of it as “learning” at all (as opposed to, say, “pointing to a certain task”). I wish people would reserve the term “learning” for the weight updates (when we’re talking about LLMs), at least in the absence of more careful justification than what I’ve seen.
In particular, instead of titling the paper “Language Models are Few-Shot Learners”, I wish they had titled it “Language Models Can Do Lots of Impressive Things Without Fine-Tuning”.
But Fig. 4.1 of that paper is definitely about actual learning.
I think there are so many disanalogies between LLMs-playing-Go and humans-playing-Go that it’s not even worth thinking about. ¯\_(ツ)_/¯ For example, humans can “visualize” things but LLMs can’t (probably). But OTOH, maybe future multi-modal next-gen LLMs will be able to.
More generally, I haven’t seen any simple comparison that provides air-tight evidence either way on sample-efficiency of deep learning versus human brains (and “deep learning” is itself is big tent—presumably some model types & sizes are more sample-efficient than others).
As it happens, I do believe that human brains are more sample efficient than any deep learning model. But my reasons for believing that are pretty indirect and I don’t want to talk about them.