I want to note a soft lower-bound here. If we compare the few-shot learning efficiency of the human brain (both in terms of watts expended, and number of samples) we note multiple orders of magnitude efficiency gain over current SotA AI learning. These algorithms were found by carbon-based life with a search algorithm that is essentially a random-walk and a satisficing ratchet mechanism called evolution. This means that there are probably far more learning algorithms and architectures lurking in the wings that we did not stumble upon (e.g. the eye vs the digital camera). The search algorithm we used to find those that power the brain, should return a loosely random sample of a much larger set. In contrast, the search methods humans are using to find intelligence algorithms are intentional and efficiency maximizing rather than satisficing. We are noticing improvement on the timescale of decades rather than millions of years as a result. Thus, as we continue to rapidly sample candidate architectures and algorithms, we should expect an explosion of capability as we search the candidate space, irrespective even of compute.
I want to note a soft lower-bound here. If we compare the few-shot learning efficiency of the human brain (both in terms of watts expended, and number of samples) we note multiple orders of magnitude efficiency gain over current SotA AI learning. These algorithms were found by carbon-based life with a search algorithm that is essentially a random-walk and a satisficing ratchet mechanism called evolution. This means that there are probably far more learning algorithms and architectures lurking in the wings that we did not stumble upon (e.g. the eye vs the digital camera). The search algorithm we used to find those that power the brain, should return a loosely random sample of a much larger set. In contrast, the search methods humans are using to find intelligence algorithms are intentional and efficiency maximizing rather than satisficing. We are noticing improvement on the timescale of decades rather than millions of years as a result. Thus, as we continue to rapidly sample candidate architectures and algorithms, we should expect an explosion of capability as we search the candidate space, irrespective even of compute.
tldr; Low hanging-fruit should be abundant.