While language models plausibly are trained with comparable amounts of FLOP to humans today here are some differences:
Humans process much less data
Humans spend much more compute per datapoint
Human data includes them taking actions and the results of those actions, language model pretraining data much less so.
These might explain some of the strengths/weaknesses of language models
LMs know many more things than humans, but often in shallower ways.
LMs seem less sample-efficient than humans (less compute per datapoint and they haven’t been very optimized for sample-efficiency yet)
LMs are worse at taking actions over time than humans.
While language models plausibly are trained with comparable amounts of FLOP to humans today here are some differences:
Humans process much less data
Humans spend much more compute per datapoint
Human data includes them taking actions and the results of those actions, language model pretraining data much less so.
These might explain some of the strengths/weaknesses of language models
LMs know many more things than humans, but often in shallower ways.
LMs seem less sample-efficient than humans (less compute per datapoint and they haven’t been very optimized for sample-efficiency yet)
LMs are worse at taking actions over time than humans.