My understanding is that it’s possible there’s a neural net along the path of GPT-1 → N that plateaus at perfectly predicting the next token of text written by a human that stops way short of having to model the entire Earth. And that would basically be a human internet poster right? If you create one of those, then training it with more text, more space, and more compute won’t make a neural net that models the earth. It’ll just make that same neural net that works perfectly on its own with a bunch of extra wasted space.
I’m not too sure my understanding is correct though.
My understanding is that it’s possible there’s a neural net along the path of GPT-1 → N that plateaus at perfectly predicting the next token of text written by a human that stops way short of having to model the entire Earth. And that would basically be a human internet poster right? If you create one of those, then training it with more text, more space, and more compute won’t make a neural net that models the earth. It’ll just make that same neural net that works perfectly on its own with a bunch of extra wasted space.
I’m not too sure my understanding is correct though.