Historical AI applications have had a relatively small loading on key-insights and seem like the closest analogies to AGI.
...Transformers as the key to text prediction?
It’s hard to see transformers making a big difference in text prediction trends when you look at benchmark data. On language modeling benchmarks such as the Penn Treebank Dataset we saw roughly smooth progress since at least 2014, and continuing at roughly the same rate through late 2017 and 2018 when the first transformer models were coming out.
It’s plausible that progress after 2017 has been faster than progress prior to 2017, but that this is hard to see in the data on Papers With Code, which only goes back to about 2013. That said, we can still see significant gradual progress prior to 2013 documented in Shen et al. which in my opinion does not look radically slower than progress post-2017.
It’s hard to see transformers making a big difference in text prediction trends when you look at benchmark data. On language modeling benchmarks such as the Penn Treebank Dataset we saw roughly smooth progress since at least 2014, and continuing at roughly the same rate through late 2017 and 2018 when the first transformer models were coming out.
It’s plausible that progress after 2017 has been faster than progress prior to 2017, but that this is hard to see in the data on Papers With Code, which only goes back to about 2013. That said, we can still see significant gradual progress prior to 2013 documented in Shen et al. which in my opinion does not look radically slower than progress post-2017.
Related SSC post: Does reality drive straight lines on graphs, or do straight lines on graphs drive reality?