The claim that the thought process behind words—the mental states of the mind and agency that produced the words … does not exist seems phenomelogically contradicted by just interacting with LLMs. I expect your counteragrument be to appeal to some idiosyncratic meanings of words like thoughts or mind states, and my response being something in the direction ‘planes do fly’.
Why LLM it up? Just give me the prompt. One reason why not to is your mind is often broadly unable to trace the thoughts of an LLM, and if the specific human-AI interaction leading to some output has nontrivial context & lenght, you would also be unable to get an LLM to replicate the trace without the context shared.
“Why LLM it up? Just give me the prompt.” Another reason not to do that is that LLMs are non-deterministic. A third reason is that I would have to track down that exact model of LLM, which I probably don’t have a license for. A fourth is that text storage on LessWrong.com is cheap, and my time is valuable. A fifth is that some LLMs are updated or altered daily. I see no reason to give someone the prompt instead of the text. That is strictly inferior in every way.
I expect your counteragrument be to appeal to some idiosyncratic meanings of words like thoughts or mind states
I think you’re just thinking of some of the thought processes, but not about the ones that are most important in the context of writing and reading and public communication. I have no big reason to suspect that LLMs don’t have “mental states” that humans do in order to proximally perform e.g. grammar, sufficient distinctions between synonyms, facts like “Baghdad is the capital of Iraq”, and a huge variety of algorithms. I do think they lack the distal mental states, which matter much more. Feel free to call this “idiosyncratic” but I sure hope you don’t call anything that’s insufficiently concrete / specific / explicit / well-understood “idiosyncratic”, because that means something totally different, and lots of centrally important stuff is insufficiently concrete / specific / explicit / well-understood.
I would describe that position as “I suspect LLMs don’t have distal/deep mental states, and as I mostly care about these distal mental states/representations, LLMs are not doing the important parts of thinking”
Also my guess is you are partially wrong about this. LLMs learn deep abstractions of reality; as these are mostly non-verbal / somewhat far from “tokens”, they are mostly unable to explain or express them using words; similarly to limited introspective access of humans.
I do think they lack the distal mental states, which matter much more.
Out of interest, do you have a good argument for this? If so, I’d be really interested to hear it.
Naively, I’d think your example of “Baghdad is the capital of Iraq” encodes enough of the content of ‘Baghdad’ and ‘Iraq’ e.g. other facts about the history of Iraq, the architecture in the city etc.. to meaningfully point towards the distal Baghdad and Iraq. Do you have a different view?
My surprise at your initial comment was related to us using the word ‘distal’ to mean slightly different things.
I’m using it to mean something like “minimally world directed” which is why I was surprised that you’d grant the possibility of LLM’s having mental content without it being minimally world directed. e.g. “Baghdad is the capital of Iraq” already seems minimally world directed if the text-based optimisation builds enough meaning into the concepts ‘Baghdad’ and ‘Iraq.’
It seems like you’re using it to mean something like “the underlying machinery required to integrate world directed contents flexibly into an accurate world model.” For example, AI village or Claude plays Pokémon show that LLM’s still struggle to build accurate enough world models to complete real world tasks.
My usage is more permissive about how accurately the content needs to track the real world to be called ‘distal’, but I don’t think this ends up leading to a substantive disagreement.
I’m probably more optimistic than you that scaling up our current LLM architectures will push this minimal world directed content to become deeper and more flexibly integrable in an accurate world model, but the links you posted are good challenges.
The claim that the thought process behind words—the mental states of the mind and agency that produced the words … does not exist seems phenomelogically contradicted by just interacting with LLMs. I expect your counteragrument be to appeal to some idiosyncratic meanings of words like thoughts or mind states, and my response being something in the direction ‘planes do fly’.
Why LLM it up? Just give me the prompt. One reason why not to is your mind is often broadly unable to trace the thoughts of an LLM, and if the specific human-AI interaction leading to some output has nontrivial context & lenght, you would also be unable to get an LLM to replicate the trace without the context shared.
“Why LLM it up? Just give me the prompt.” Another reason not to do that is that LLMs are non-deterministic. A third reason is that I would have to track down that exact model of LLM, which I probably don’t have a license for. A fourth is that text storage on LessWrong.com is cheap, and my time is valuable. A fifth is that some LLMs are updated or altered daily. I see no reason to give someone the prompt instead of the text. That is strictly inferior in every way.
I think you’re just thinking of some of the thought processes, but not about the ones that are most important in the context of writing and reading and public communication. I have no big reason to suspect that LLMs don’t have “mental states” that humans do in order to proximally perform e.g. grammar, sufficient distinctions between synonyms, facts like “Baghdad is the capital of Iraq”, and a huge variety of algorithms. I do think they lack the distal mental states, which matter much more. Feel free to call this “idiosyncratic” but I sure hope you don’t call anything that’s insufficiently concrete / specific / explicit / well-understood “idiosyncratic”, because that means something totally different, and lots of centrally important stuff is insufficiently concrete / specific / explicit / well-understood.
I would describe that position as “I suspect LLMs don’t have distal/deep mental states, and as I mostly care about these distal mental states/representations, LLMs are not doing the important parts of thinking”
Also my guess is you are partially wrong about this. LLMs learn deep abstractions of reality; as these are mostly non-verbal / somewhat far from “tokens”, they are mostly unable to explain or express them using words; similarly to limited introspective access of humans.
Out of interest, do you have a good argument for this? If so, I’d be really interested to hear it.
Naively, I’d think your example of “Baghdad is the capital of Iraq” encodes enough of the content of ‘Baghdad’ and ‘Iraq’ e.g. other facts about the history of Iraq, the architecture in the city etc.. to meaningfully point towards the distal Baghdad and Iraq. Do you have a different view?
I mean the deeper mental algorithms that generate the concepts in the first place, which are especially needed to do e.g. novel science. See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
and
https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense
See also this thread: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce?commentId=dqbLkADbJQJi6bFtN
Thanks! This is helpful to understand your view.
My surprise at your initial comment was related to us using the word ‘distal’ to mean slightly different things.
I’m using it to mean something like “minimally world directed” which is why I was surprised that you’d grant the possibility of LLM’s having mental content without it being minimally world directed. e.g. “Baghdad is the capital of Iraq” already seems minimally world directed if the text-based optimisation builds enough meaning into the concepts ‘Baghdad’ and ‘Iraq.’
It seems like you’re using it to mean something like “the underlying machinery required to integrate world directed contents flexibly into an accurate world model.” For example, AI village or Claude plays Pokémon show that LLM’s still struggle to build accurate enough world models to complete real world tasks.
My usage is more permissive about how accurately the content needs to track the real world to be called ‘distal’, but I don’t think this ends up leading to a substantive disagreement.
I’m probably more optimistic than you that scaling up our current LLM architectures will push this minimal world directed content to become deeper and more flexibly integrable in an accurate world model, but the links you posted are good challenges.