In the limit, GPT-N models the entire Earth that created the Internet in order to predict text completion. Before then, it will invent new methods of psychoanalyzing humans to better infer correlations between text. So it surely has superhuman capabilities, it’s just a matter of accessing them.
My understanding is that it’s possible there’s a neural net along the path of GPT-1 → N that plateaus at perfectly predicting the next token of text written by a human that stops way short of having to model the entire Earth. And that would basically be a human internet poster right? If you create one of those, then training it with more text, more space, and more compute won’t make a neural net that models the earth. It’ll just make that same neural net that works perfectly on its own with a bunch of extra wasted space.
I’m not too sure my understanding is correct though.
In the limit, GPT-N models the entire Earth that created the Internet in order to predict text completion. Before then, it will invent new methods of psychoanalyzing humans to better infer correlations between text. So it surely has superhuman capabilities, it’s just a matter of accessing them.
My understanding is that it’s possible there’s a neural net along the path of GPT-1 → N that plateaus at perfectly predicting the next token of text written by a human that stops way short of having to model the entire Earth. And that would basically be a human internet poster right? If you create one of those, then training it with more text, more space, and more compute won’t make a neural net that models the earth. It’ll just make that same neural net that works perfectly on its own with a bunch of extra wasted space.
I’m not too sure my understanding is correct though.