Imagine typing the following meta-question into GPT-4, a revolutionary new 20 Trillion parameter language model released in 2021:
“I asked the superintelligence how to cure cancer. The superintelligence responded __”
How likely are we to get an actual cure for cancer, complete with manufacturing blueprints?
...The difference between the two lies on the fine semantic line: whether GPT thinks the conversation is a human imitating superintelligence, or an actual words of a superintelligence. Arguably, since it only has training samples of the former, it will do the former. Yet that’s not what it did with numbers—it learnt the underlying principle, and extrapolated to tasks it had never seen.
What GPT is actually trying to do is predicting the continuation of random texts found on the Internet. So, if you let it continue “24+51=”, what it does is answering the question “Suppose a random text on the Internet contained the string ’24+51=’. What do I expect to come next?” In this case, it seems fairly reasonable to expect the correct answer. More so if this is preceded by a number of correct exercises in arithmetic (otherwise, maybe it’s e.g. one of those puzzles in which symbols of arithmetic are used to denote something different).
On the other hand, your text about curing cancer is extremely unlikely to be generated by actual superintelligence. If you told me that you found this text on the Internet, I would bet against the continuation being an actual cure for cancer. I expect any version of GPT which is as smart as me or more to reason similarly (except for complex reasons to do with subagents and acausal bargaining which are besides the point here), and any version of GPT that is less smart than me to be unable to cure cancer (roughly speaking: intelligence is not really one-dimensional).
It seems more likely to get an actual cure for cancer if your initial text is a realistic imitation of something like, an academic paper describing a novel cure for cancer. Or, a paper in AI describing a superintelligence that can cure cancer.
What GPT is actually trying to do is predicting the continuation of random texts found on the Internet. So, if you let it continue “24+51=”, what it does is answering the question “Suppose a random text on the Internet contained the string ’24+51=’. What do I expect to come next?” In this case, it seems fairly reasonable to expect the correct answer. More so if this is preceded by a number of correct exercises in arithmetic (otherwise, maybe it’s e.g. one of those puzzles in which symbols of arithmetic are used to denote something different).
On the other hand, your text about curing cancer is extremely unlikely to be generated by actual superintelligence. If you told me that you found this text on the Internet, I would bet against the continuation being an actual cure for cancer. I expect any version of GPT which is as smart as me or more to reason similarly (except for complex reasons to do with subagents and acausal bargaining which are besides the point here), and any version of GPT that is less smart than me to be unable to cure cancer (roughly speaking: intelligence is not really one-dimensional).
It seems more likely to get an actual cure for cancer if your initial text is a realistic imitation of something like, an academic paper describing a novel cure for cancer. Or, a paper in AI describing a superintelligence that can cure cancer.