I don’t know what you mean by “GPT-N” but if you mean “the same thing they do now, but scaled up,” I’m doubtful that it will happen that way.
Language models are made using fill-in-the-blank training, which is about imitation. Some things can be learned that way, but to get better at doing hard things (like playing Go at superhuman level) you need training that’s about winning increasingly harder competitions. Beyond a certain point, imitating game transcripts doesn’t get any harder, so becomes more like learning stage sword fighting.
Also, “making detailed plans at high speed” is similar to “writing extremely long documents.” There are limits on how far back a language model can look in the chat transcript. It’s difficult to increase because it’s an O(N-squared) algorithm, though I’ve seen a paper claiming it can be improved.
Language models aren’t particularly good at reasoning, let alone long chains of reasoning, so it’s not clear that using them to generate longer documents will result in them getting better results.
So there might not be much incentive for researchers to work on language models that can write extremely long documents.
I don’t know what you mean by “GPT-N” but if you mean “the same thing they do now, but scaled up,” I’m doubtful that it will happen that way.
Language models are made using fill-in-the-blank training, which is about imitation. Some things can be learned that way, but to get better at doing hard things (like playing Go at superhuman level) you need training that’s about winning increasingly harder competitions. Beyond a certain point, imitating game transcripts doesn’t get any harder, so becomes more like learning stage sword fighting.
Also, “making detailed plans at high speed” is similar to “writing extremely long documents.” There are limits on how far back a language model can look in the chat transcript. It’s difficult to increase because it’s an O(N-squared) algorithm, though I’ve seen a paper claiming it can be improved.
Language models aren’t particularly good at reasoning, let alone long chains of reasoning, so it’s not clear that using them to generate longer documents will result in them getting better results.
So there might not be much incentive for researchers to work on language models that can write extremely long documents.
Vaguely descriptive frames can be taken as prescriptive, motivating particular design changes.