My understanding of your argument is that AI progress will slow down in the future because the low-hanging fruit in hardware, software, and research have been exhausted.
Hardware: researchers have scaled models to the point where they cost millions of dollars. At this point, scaling them further is difficult. Moore’s Law is slowing down, making it harder to scale models.
In my opinion, it seems unlikely, but not inconceivable, that training budgets will increase further. It could happen if more useful models result in greater financial returns and investment in a positive feedback loop. Human labor is expensive and creating an artificial replacement could still be profitable even with large training costs. Another possibility is that government investment increases AI training budgets in some kind of AI manhattan project. Though this possibility doesn’t seem likely to me given that most progress has occurred in private companies in recent years.
I’m somewhat less pessimistic about the death of Moore’s Law. Although it’s getting harder to improve chip performance, there is still a strong incentive to improve it. We at least know that it’s possible for improvements to continue because current technology is not near the physical (1).
Software: has improved a lot in recent years. For example, libraries such as HuggingFace have made it much easier to use the latest models. The post argues that research is not bottlenecked by progress in software.
This point seems valid to me. However, better AI-assisted programming tools in the future could increase the rate of software development even more.
Research: transformers, pipeline parallelism, and self-supervised learning have made it possible to train large models with much better performance. The post also says that many of these innovations (e.g. the transformer) are from 2018 or earlier.
New techniques are introduced, they mature and are replaced by newer techniques. For example, progress in CPU speed stagnated and GPUs increased performance dramatically. TPUs have improved on GPUs and we’ll probably see further progress. If this is true, then some of the AI techniques that will be commonplace in several years are probably already under development but not mature enough to be used in mainstream AI. Instead of a global slowdown in AI, I see AI research progress as a series of s-curves.
I can’t imagine how future architectures will be different but the historical trend has always been that new and better techniques replace old ones.
As more money and talent are invested in AI research, progress should accelerate given a fixed difficulty in making progress. Even if the problems become harder to solve, increased talent and financial investment should offset the increase in difficulty. Therefore, it seems like the problems would have to become much harder for AI progress to slow down significantly which doesn’t seem likely to me given how new the field is.
Given that deep learning has only been really popular for about ten years, it seems unlikely that most of the low-hanging fruit have already been extracted, unlike particle physics which has been around for decades and where particle colliders have had diminishing returns.
Overall, I’m more bullish on AI progress in the future than this post and I expect more significant progress to occur.
My understanding of your argument is that AI progress will slow down in the future because the low-hanging fruit in hardware, software, and research have been exhausted.
Hardware: researchers have scaled models to the point where they cost millions of dollars. At this point, scaling them further is difficult. Moore’s Law is slowing down, making it harder to scale models.
In my opinion, it seems unlikely, but not inconceivable, that training budgets will increase further. It could happen if more useful models result in greater financial returns and investment in a positive feedback loop. Human labor is expensive and creating an artificial replacement could still be profitable even with large training costs. Another possibility is that government investment increases AI training budgets in some kind of AI manhattan project. Though this possibility doesn’t seem likely to me given that most progress has occurred in private companies in recent years.
I’m somewhat less pessimistic about the death of Moore’s Law. Although it’s getting harder to improve chip performance, there is still a strong incentive to improve it. We at least know that it’s possible for improvements to continue because current technology is not near the physical (1).
Software: has improved a lot in recent years. For example, libraries such as HuggingFace have made it much easier to use the latest models. The post argues that research is not bottlenecked by progress in software.
This point seems valid to me. However, better AI-assisted programming tools in the future could increase the rate of software development even more.
Research: transformers, pipeline parallelism, and self-supervised learning have made it possible to train large models with much better performance. The post also says that many of these innovations (e.g. the transformer) are from 2018 or earlier.
New techniques are introduced, they mature and are replaced by newer techniques. For example, progress in CPU speed stagnated and GPUs increased performance dramatically. TPUs have improved on GPUs and we’ll probably see further progress. If this is true, then some of the AI techniques that will be commonplace in several years are probably already under development but not mature enough to be used in mainstream AI. Instead of a global slowdown in AI, I see AI research progress as a series of s-curves.
I can’t imagine how future architectures will be different but the historical trend has always been that new and better techniques replace old ones.
As more money and talent are invested in AI research, progress should accelerate given a fixed difficulty in making progress. Even if the problems become harder to solve, increased talent and financial investment should offset the increase in difficulty. Therefore, it seems like the problems would have to become much harder for AI progress to slow down significantly which doesn’t seem likely to me given how new the field is.
Given that deep learning has only been really popular for about ten years, it seems unlikely that most of the low-hanging fruit have already been extracted, unlike particle physics which has been around for decades and where particle colliders have had diminishing returns.
Overall, I’m more bullish on AI progress in the future than this post and I expect more significant progress to occur.
1: https://en.wikipedia.org/wiki/Landauer%27s_principle