It has only been done by “Time-Aware Language Models as Temporal Knowledge Bases”, Dhingra et al 2021 (and done mostly in passing by earlier experiments in providing metadata prefixes like CTRL). Temporal reasoning scales, unsurprisingly: even without explicit metadata, which would be extremely hard to get reliably for most cases (eg Common Crawl—dating random web pages at scale? gl), there tend to be lots of implicit clues in text, such as the URL structure of news articles (CTRL gives the example of https://www.cnn.com/style/09/20/2018/george-clooney-interview), and this probably serves as scaffolding for helping understand the internal evidence of undated text. You can already prompt a model like GPT-3 with dates already, so you wouldn’t be creating any qualitatively new capabilities.
So including more metadata (of every kind, not just dating) is a good idea, but not necessary and may be a bad use of expert human labor: probably it’d be so cheap that it’s worth hand-engineering in for clean sources like Wikipedia or Twitter or Reddit or academic datasets where you can be sure of the date easily, but then less so for the bulk of the dataset coming from Common Crawl etc.
It has only been done by “Time-Aware Language Models as Temporal Knowledge Bases”, Dhingra et al 2021 (and done mostly in passing by earlier experiments in providing metadata prefixes like CTRL). Temporal reasoning scales, unsurprisingly: even without explicit metadata, which would be extremely hard to get reliably for most cases (eg Common Crawl—dating random web pages at scale? gl), there tend to be lots of implicit clues in text, such as the URL structure of news articles (CTRL gives the example of
https://www.cnn.com/style/09/20/2018/george-clooney-interview), and this probably serves as scaffolding for helping understand the internal evidence of undated text. You can already prompt a model like GPT-3 with dates already, so you wouldn’t be creating any qualitatively new capabilities.So including more metadata (of every kind, not just dating) is a good idea, but not necessary and may be a bad use of expert human labor: probably it’d be so cheap that it’s worth hand-engineering in for clean sources like Wikipedia or Twitter or Reddit or academic datasets where you can be sure of the date easily, but then less so for the bulk of the dataset coming from Common Crawl etc.