ChatGPT is settling the Chinese Room argument
My observation of people’s experiences with ChatGPT leads me to the conclusion that while for many people it appears to be passing Turing test, it is possible to demonstrate that it does not have any deeper understanding (i.e. that it does not contain useful representation of semantics) or any ability to generalize and transfer knowledge to adjacent domains.
This is understandable considering that GPT is over-parametrized model which contains some encoding of the entire (very large) training set. It is a de-facto implementation of a Chinese Room having sufficient language fluency to parse and generate passably correct language. (The lack of generalization capability is also a known property of current ANNs.)
The main take-out of that is that the Chinese Room which merely finds appropriate answers from an (encoded) set is not the same as a semantic representation and processing, and can be relatively easily distinguished from the actual intelligence, bolstering John Searle’s argument.
I think you’re misunderstanding the Chinese Room argument, which purports to show that the program being run in the room doesn’t understand anything, regardless of how good its input-output behaviour is. So noting that ChatGPT sometimes appears to lack understanding says nothing about the Chinese Room argument—it just shows that ChatGPT isn’t good enough for the Chinese Room argument to be relevant.
For the record, I think the Chinese Room argument is invalid, but we aren’t at the point where the debate is of practical importance.
The point is not “not understanding sometimes”, the point is not understanding in a sense of inability to generate responses having no close analogs in the training sets. ChatGPT is very good at finding the closest example and fitting it into output text. What it cannot do—obviously—is to combine two things it can answer satisfactory separately and combine them into a coherent answer to a question requiring two steps (unless it has seen an analog of this two-step answer already).
This shows the complete lack of usable semantic encoding—which is the core of the original Searle’s argument.
You’re still arguing with reference to what ChatGPT can or cannot do as far as producing responses to questions—that it cannot produce “a coherent answer to a question requiring two steps”. But the claim of the Chinese Room argument is that even if it could do that, and could do everything else you think it ought to be able to do, it still would not actually understand anything. We have had programs that produce text but clearly don’t understand many things for decades. That ChatGPT is another such program has no implications for whether or not the Chinese Room argument is correct. If at some point we conclude that it is just not possible to write a program that behaves in all respects as if it understands, that wouldn’t so much refute or support the Chinese Room argument, as simply render it pointless, since its premise cannot possible hold.
Not just “some” encoding. It is in fact a semantic encoding. Concepts that are semantically close together are also close together in the encoding. It also doesn’t find answers from a set. It actually translates its internal semantic concept into words, and this process is guided by flexible requirements.
You can do stuff like tell it to provide the answer in consonants only, or in the form of a song, or pirate speak, even though the training data would barely have any relevant examples.
It is instructive to talk to Chatgpt for 10 mins to see what it knows about the rules and strategies for playing tic-tac-toe and then ask it to actually play a game of tic-tac-toe with you using those same strategies. In my experiment, I came to 2 conclusions:
Chatgpt knows everything there is to know about tic-tac-toe and the best strategies for playing it.
Chatgpt is entirely incapable of applying the knowledge it has for (1) in an actual game of tic-tac-toe it is playing with you.
Chatgpt is merely a generative language model: which means it is good at picking the next word given the prompt and the previous words it has output. Sadly, it knows little to nothing about the meaning of those words. It can’t use what it knows about a topic (the rules and best strategies for playing tic-tac-toe) to benefit it in relevant scenarios (when playing an actual game of tic-tac-toe). To Chatgpt, (1) and (2) have almost nothing to do with each other because they rarely occur in the same conversation in its training data. Despite this model being able to obviously learn it has no understanding of the things its learned. It cannot use those things to learn new things. In a sense, its web of knowledge is highly disconnected.
What you are describing in your blog post has not been my experience. ChatGPT can definitely summarize my own chat it. It is also capable of using information from one domain to another.
That said, ChatGPT sometimes makes mistakes. But this is not an argument against ChatGPT being able to understand language. It only means that having access to textual information alone is not enough to reason accurately about the physical world. This is a surmountable limitation.
I just came here to clarify that ChatGPT can generalize to some extend and hence possess some sort of understanding of the natural world. Who are we to judge that his algorithms are intrinsically worse than ours and that he doesn’t understand ‘in the proper way’ ?