I care about many things, but one that’s important here is that I care about understanding the world. For instance, I am curious about the capabilities (present and future) of AI systems. You say that from first principles we can tell that LLMs trained on text can’t actually mean anything they “say”, that they can have only a simulacrum of understanding, etc. So I am curious about (1) whether this claim tells us anything about what they can actually do as opposed to what words you choose to use to describe them, and (2) if so whether it’s correct.
Another thing I care about is clarity of thought and communication (mine and, to a lesser but decidedly nonzero extent, other people’s). So to whatever extent your thoughts on “meaning”, “understanding”, etc., are about that more than they’re about what LLMs can actually do, I am still interested, because when thinking and talking about LLMs I would prefer not to use language in a systematically misleading way. (At present, I would generally avoid saying that an LLM “means” whatever strings of text it emits, because I don’t think there’s anything sufficiently mind-like in there, but I would not avoid saying that in many cases those strings of text “mean” something, for reasons I’ve already sketched above. My impression is that you agree about the first of those and disagree about the second. So presumably at least one of us is wrong, and if we can figure out who then that person can improve their thinking and/or communication a bit.)
Back to the actual discussion, such as it is. People do in fact typically read the output of LLMs, but there isn’t much information flow back to the LLMs after that, so that process is less relevant to the question of whether and how and how much LLMs’ output is coupled to the actual world. That coupling happens via the chain I described two comments up from this one: world → people → corpus → LLM weights → LLM output. It’s rather indirect, but the same goes for a lot of what humans say.
Several comments back, I asked what key difference you see between that chain (which in your view leaves no scope for the LLM’s output to “mean” anything) and one that goes world → person 1 → writing → person 2 → person 2′s words (which in your view permits person 2′s words to mean things even when person 2 is talking about something they know about only indirectly via other people’s writing). It seems to me that if the problem is a lack of “adhesion”—contact with the real world—then that afflicts person 2 in this scenario in the same way as it afflicts the LLM in the other scenario: both are emitting words whose only connection to the real world is an indirect one via other people’s writing. I assume you reckon I’m missing some important point here; what is it?
That’s a bunch of stuff, more than I can deal with at the moment.
On the meaning of “meaning,” it’s a mess and people in various discipline have been arguing it for 3/4s of a century or more at this point. You might want to take a look at a longish comment I posted above, if you haven’t already. It’s a passage from another article, where I make the point that terms like “think” don’t really tell us much at all. What matters to me at this point are the physical mechanisms, and those terms don’t convey much about those mechanisms.
On LLMs, GPT-4 now has plug-ins. I recently saw a YouTube video about the Wolfram Alpha plug-in. You ask GPT-4 a question, it decides to query Wolfram Alpha and sends a message. Alpha does something, sends the result back to GPT-4, which presents the result to you. So now we have Alpha interpreting messages from GPT-4 and GPT-4 interpreting messages from Alpha. How reliable is that circuit? Does it give the human user what they want? How does “meaning” work in that circuit.
I first encountered the whole business of meaning in philosophy and literary criticism. So, you read Dickens’ A Tale of Two Cities or Frank Herbert’s Dune, whatever. It’s easy to say those texts have meaning. But where does that meaning come from? When you read those texts, the meaning comes from you. When I read them, it comes from me. What about the meanings the authors put into them? You can see where I’m going with this. Meaning is not like wine, that can be poured from one glass to another and remain the same. Well, literary critics argued about that one for decades. The issue’s never really been settled. It’s just been dropped, more or less.
ChatGPT produces text, lots of it. When you read one of those texts, where does the meaning come from? Let’s ask a different question. People are now using output from LLMs as a medium for interacting with one another. How is that working out? Where can LLM text be useful and where not? What’s the difference? Those strike me as rather open-ended questions for which we do not have answers at the moment.
I think it’s clear that when you read a book the meaning is a product of both you and the book, because if instead you read a different book you’d arrive at different meaning, and different people reading the same book get to-some-extent-similar meanings from it. So “the meaning comes from you” / “the meaning comes from me” is too simple. It seems to me that generally you get more-similar meanings when you keep the book the same and change the reader than when you keep the reader the same and change the book, though of course it depends on how big a change you make in either case, so I would say more of the meaning is in the text than in the reader. (For the avoidance of doubt: no, I do not believe that there’s some literal meaning-stuff that we could distil from books and readers and measure. “In” there is a metaphor. Obviously.)
I agree that there are many questions to which we don’t have answers, and that more specific and concrete questions may be more illuminating than very broad and vague ones like “does the text emited by an LLM have meaning?”.
I don’t know how well the GPT/Wolfram|Alpha integration works (I seem to remember reading somewhere that it’s very flaky, but maybe they’ve made it better), but I suggest that to whatever extent it successfully results in users getting information that’s correct on account of Alpha’s databases having been filled with data derived from how the world actually is, and its algorithms having been designed to match how mathematics actually works, that’s an indication that in some useful sense some kind of meaning is being (yes, metaphorically) transmitted.
...the question of whether or not the language produced by LLMs is meaningful is up to us. Do you trust it? Do WE trust it? Why or why not?
That’s the position I’m considering. If you understand “WE” to mean society as a whole, then the answer is that the question is under discussion and is undetermined. But some individuals do seem to trust the text from certain LLMs at least under certain circumstances. For the most part I trust the output of ChatGPT and GPT-4, with which I have considerably less experience than I do with ChatGPT. I know that both systems make mistakes of various kinds, including what is called “hallucination.” It’s not clear to me that that differentiates them from ordinary humans, who make mistakes and often say things without foundation in reality.
I care about many things, but one that’s important here is that I care about understanding the world. For instance, I am curious about the capabilities (present and future) of AI systems. You say that from first principles we can tell that LLMs trained on text can’t actually mean anything they “say”, that they can have only a simulacrum of understanding, etc. So I am curious about (1) whether this claim tells us anything about what they can actually do as opposed to what words you choose to use to describe them, and (2) if so whether it’s correct.
Another thing I care about is clarity of thought and communication (mine and, to a lesser but decidedly nonzero extent, other people’s). So to whatever extent your thoughts on “meaning”, “understanding”, etc., are about that more than they’re about what LLMs can actually do, I am still interested, because when thinking and talking about LLMs I would prefer not to use language in a systematically misleading way. (At present, I would generally avoid saying that an LLM “means” whatever strings of text it emits, because I don’t think there’s anything sufficiently mind-like in there, but I would not avoid saying that in many cases those strings of text “mean” something, for reasons I’ve already sketched above. My impression is that you agree about the first of those and disagree about the second. So presumably at least one of us is wrong, and if we can figure out who then that person can improve their thinking and/or communication a bit.)
Back to the actual discussion, such as it is. People do in fact typically read the output of LLMs, but there isn’t much information flow back to the LLMs after that, so that process is less relevant to the question of whether and how and how much LLMs’ output is coupled to the actual world. That coupling happens via the chain I described two comments up from this one: world → people → corpus → LLM weights → LLM output. It’s rather indirect, but the same goes for a lot of what humans say.
Several comments back, I asked what key difference you see between that chain (which in your view leaves no scope for the LLM’s output to “mean” anything) and one that goes world → person 1 → writing → person 2 → person 2′s words (which in your view permits person 2′s words to mean things even when person 2 is talking about something they know about only indirectly via other people’s writing). It seems to me that if the problem is a lack of “adhesion”—contact with the real world—then that afflicts person 2 in this scenario in the same way as it afflicts the LLM in the other scenario: both are emitting words whose only connection to the real world is an indirect one via other people’s writing. I assume you reckon I’m missing some important point here; what is it?
That’s a bunch of stuff, more than I can deal with at the moment.
On the meaning of “meaning,” it’s a mess and people in various discipline have been arguing it for 3/4s of a century or more at this point. You might want to take a look at a longish comment I posted above, if you haven’t already. It’s a passage from another article, where I make the point that terms like “think” don’t really tell us much at all. What matters to me at this point are the physical mechanisms, and those terms don’t convey much about those mechanisms.
On LLMs, GPT-4 now has plug-ins. I recently saw a YouTube video about the Wolfram Alpha plug-in. You ask GPT-4 a question, it decides to query Wolfram Alpha and sends a message. Alpha does something, sends the result back to GPT-4, which presents the result to you. So now we have Alpha interpreting messages from GPT-4 and GPT-4 interpreting messages from Alpha. How reliable is that circuit? Does it give the human user what they want? How does “meaning” work in that circuit.
I first encountered the whole business of meaning in philosophy and literary criticism. So, you read Dickens’ A Tale of Two Cities or Frank Herbert’s Dune, whatever. It’s easy to say those texts have meaning. But where does that meaning come from? When you read those texts, the meaning comes from you. When I read them, it comes from me. What about the meanings the authors put into them? You can see where I’m going with this. Meaning is not like wine, that can be poured from one glass to another and remain the same. Well, literary critics argued about that one for decades. The issue’s never really been settled. It’s just been dropped, more or less.
ChatGPT produces text, lots of it. When you read one of those texts, where does the meaning come from? Let’s ask a different question. People are now using output from LLMs as a medium for interacting with one another. How is that working out? Where can LLM text be useful and where not? What’s the difference? Those strike me as rather open-ended questions for which we do not have answers at the moment.
And so on....
I think it’s clear that when you read a book the meaning is a product of both you and the book, because if instead you read a different book you’d arrive at different meaning, and different people reading the same book get to-some-extent-similar meanings from it. So “the meaning comes from you” / “the meaning comes from me” is too simple. It seems to me that generally you get more-similar meanings when you keep the book the same and change the reader than when you keep the reader the same and change the book, though of course it depends on how big a change you make in either case, so I would say more of the meaning is in the text than in the reader. (For the avoidance of doubt: no, I do not believe that there’s some literal meaning-stuff that we could distil from books and readers and measure. “In” there is a metaphor. Obviously.)
I agree that there are many questions to which we don’t have answers, and that more specific and concrete questions may be more illuminating than very broad and vague ones like “does the text emited by an LLM have meaning?”.
I don’t know how well the GPT/Wolfram|Alpha integration works (I seem to remember reading somewhere that it’s very flaky, but maybe they’ve made it better), but I suggest that to whatever extent it successfully results in users getting information that’s correct on account of Alpha’s databases having been filled with data derived from how the world actually is, and its algorithms having been designed to match how mathematics actually works, that’s an indication that in some useful sense some kind of meaning is being (yes, metaphorically) transmitted.
I’ve just posted something at my home blog, New Savanna, in which I consider the idea that