Humans are probably not a good benchmark but what do we know about how humans update factual knowledge?
(or maybe we are—maybe humans are actually quite exceptional at updating factual knowledge but I’m hypersensitive to the errors or examples of failures. Perhaps I’m over looking all the updates we do over the day, say the score of a Lionel Messi game, or where they are in the competition ladder, “My brother just called, he’s still at the restaurant” to “they’re in traffic on the freeway” to “they’re just around the corner”??)
I think we know very little about how humans learn individual new facts from a neuroscience perspective. There are some studies that track fMRI scans of individuals who learn new things over the course of a day (sleep seems particularly important for the formation of connections in the brain having to do with new learning), but I am sceptical that it is the kind of learning that could be applied to Language Model type systems as of now.
In general though, I think humans are the gold standard for factual knowledge updating. I agree with you that there are some examples of failures, but no other system comes close in my view.
Humans are probably not a good benchmark but what do we know about how humans update factual knowledge?
(or maybe we are—maybe humans are actually quite exceptional at updating factual knowledge but I’m hypersensitive to the errors or examples of failures. Perhaps I’m over looking all the updates we do over the day, say the score of a Lionel Messi game, or where they are in the competition ladder, “My brother just called, he’s still at the restaurant” to “they’re in traffic on the freeway” to “they’re just around the corner”??)
I think we know very little about how humans learn individual new facts from a neuroscience perspective. There are some studies that track fMRI scans of individuals who learn new things over the course of a day (sleep seems particularly important for the formation of connections in the brain having to do with new learning), but I am sceptical that it is the kind of learning that could be applied to Language Model type systems as of now.
In general though, I think humans are the gold standard for factual knowledge updating. I agree with you that there are some examples of failures, but no other system comes close in my view.