There has been some insider discussion (and Sam Altman has said) that scaling has started running into some difficulties. Specifically, GPT-4 has gained a wider breath of knowledge, but has not significantly improved in any one domain. This might mean that future AI systems may gain their capabilities from places other than scaling because of the diminishing returns from scaling. This could mean that to become “superintelligent”, the AI needs to run experiments and learn from the outcome of those experiments to gain more superintelligent capabilities. You can only learn so much from a static dataset.
So you can imagine the case where capabilities come from some form of active/continual/online learning, but that was only possible once models were scaled up enough to gain capabilities in that way. And so that as LLMs become more capable, they will essentially become capable of running their own experiments to gain alphafold-like capabilities across many domains.
Of course, this has implications for understanding takeoffs / sharp left turns.
As Max H said, I think once you meet a threshold with a universal interface like a language model, things start to open up and the game changes.
I spoke to Altman about a month ago. He essentially said some of the following:
His recent statement about scaling essentially plateau-ing was misunderstood and he still thinks it plays a big role.
Then, I asked him what comes next and he said they are working on the next thing that will provide 1000x improvement (some new paradigm).
I asked if online learning plays a role in that and he said yes.
That’s one of the reasons we started to work on Supervising AIs Improving AIs.
In a shortform last month, I wrote the following:
As Max H said, I think once you meet a threshold with a universal interface like a language model, things start to open up and the game changes.