Summary of your argument: The training data can contain outputs of processes that have superhuman abilities (eg chess engines), therefore LLMs can exceed human performance.
More speculatively, there might be another source of (slight?) superhuman abilities: GPT-N could generalize/extrapolate from human abilities to superhuman abilities, if it was plausible that at some point in the future these superhuman abilities would be shown on the internet. For example, it is conceivable that GPT-N prompted with “Here is a proof of the Riemann hypothesis that has been verified extensively:” would actually a valid proof, even if a proof of the Riemann hypothesis was beyond the ability of humans in the training data.
But perhaps this is an assumption people often make about LLMs.
I think people often claim something along the lines of “GPT-8 cannot exceed human capacity” (which is technically false) to argue that a (naively) upscaled version of GPT-3 cannot reach AGI.
I think we should expect that there are at least some limits to the intelligence we can obtain from GPT-8, if they just train it to predict text (and not do any amplification steps, or RL).
Summary of your argument: The training data can contain outputs of processes that have superhuman abilities (eg chess engines), therefore LLMs can exceed human performance.
More speculatively, there might be another source of (slight?) superhuman abilities: GPT-N could generalize/extrapolate from human abilities to superhuman abilities, if it was plausible that at some point in the future these superhuman abilities would be shown on the internet. For example, it is conceivable that GPT-N prompted with “Here is a proof of the Riemann hypothesis that has been verified extensively:” would actually a valid proof, even if a proof of the Riemann hypothesis was beyond the ability of humans in the training data.
I think people often claim something along the lines of “GPT-8 cannot exceed human capacity” (which is technically false) to argue that a (naively) upscaled version of GPT-3 cannot reach AGI. I think we should expect that there are at least some limits to the intelligence we can obtain from GPT-8, if they just train it to predict text (and not do any amplification steps, or RL).