On the bright side, according to OpenAI’s scaling laws paper, GPT-3 is about the size that scaling was predicted to start breaking down. So maybe GPT-4 won’t actually be better than GPT-3. I’m not counting on it though.
It’s possible that GPT-3 is roughly at where the maximally naive simple text LM begins to hit the constant wall, but I don’t regard this as important; as I emphasize at every turn, there are many distinct ways in which to improve it greatly using purely known methods, never mind future research approaches. The question is not whether there is any way GPT-4 might fail, but any way in which it might succeed.
On the bright side, according to OpenAI’s scaling laws paper, GPT-3 is about the size that scaling was predicted to start breaking down. So maybe GPT-4 won’t actually be better than GPT-3. I’m not counting on it though.
It’s possible that GPT-3 is roughly at where the maximally naive simple text LM begins to hit the constant wall, but I don’t regard this as important; as I emphasize at every turn, there are many distinct ways in which to improve it greatly using purely known methods, never mind future research approaches. The question is not whether there is any way GPT-4 might fail, but any way in which it might succeed.