Do you think you have become much smarter mind because of that? I doubt do—but you were given a very hard task, and a high reward.
No, but I was able to predict my own sensory input pretty well, for those 5 minutes. (I was sitting in a quiet room, mostly pondering how I would respond to this comment, rather than the actual problem you posed. When I closed my eyes, the sensory prediction problem got even easier.)
You could probably also train a GPT on sensory inputs (suitably encoded) instead of text, and get pretty good predictions about future sensory inputs.
Stepping back, the fact that you can draw a high-level analogy between neuroplasticity in human brains ⇔ SGD in transformer networks, and sensory input prediction ⇔ next token prediction doesn’t mean you can declare there is “no fundamental difference” in the nature of these things, even if you are careful to avoid the type error in your last example.
In the limit (maybe) a sufficiently good predictor could perfectly predict both sensory input and tokens, but the point is that the analogy breaks down in the ordinary, limited case, on the kinds of concrete tasks that GPTs and humans are being asked to solve today. There are plenty of text manipulation and summarization problems that GPT-4 is already superhuman at, and SGD can already re-weight a transformer network much more than neuroplasticity can reshape a human brain.
No, but I was able to predict my own sensory input pretty well, for those 5 minutes. (I was sitting in a quiet room, mostly pondering how I would respond to this comment, rather than the actual problem you posed. When I closed my eyes, the sensory prediction problem got even easier.)
You could probably also train a GPT on sensory inputs (suitably encoded) instead of text, and get pretty good predictions about future sensory inputs.
Stepping back, the fact that you can draw a high-level analogy between neuroplasticity in human brains ⇔ SGD in transformer networks, and sensory input prediction ⇔ next token prediction doesn’t mean you can declare there is “no fundamental difference” in the nature of these things, even if you are careful to avoid the type error in your last example.
In the limit (maybe) a sufficiently good predictor could perfectly predict both sensory input and tokens, but the point is that the analogy breaks down in the ordinary, limited case, on the kinds of concrete tasks that GPTs and humans are being asked to solve today. There are plenty of text manipulation and summarization problems that GPT-4 is already superhuman at, and SGD can already re-weight a transformer network much more than neuroplasticity can reshape a human brain.