I expect your counteragrument be to appeal to some idiosyncratic meanings of words like thoughts or mind states
I think you’re just thinking of some of the thought processes, but not about the ones that are most important in the context of writing and reading and public communication. I have no big reason to suspect that LLMs don’t have “mental states” that humans do in order to proximally perform e.g. grammar, sufficient distinctions between synonyms, facts like “Baghdad is the capital of Iraq”, and a huge variety of algorithms. I do think they lack the distal mental states, which matter much more. Feel free to call this “idiosyncratic” but I sure hope you don’t call anything that’s insufficiently concrete / specific / explicit / well-understood “idiosyncratic”, because that means something totally different, and lots of centrally important stuff is insufficiently concrete / specific / explicit / well-understood.
I would describe that position as “I suspect LLMs don’t have distal/deep mental states, and as I mostly care about these distal mental states/representations, LLMs are not doing the important parts of thinking”
Also my guess is you are partially wrong about this. LLMs learn deep abstractions of reality; as these are mostly non-verbal / somewhat far from “tokens”, they are mostly unable to explain or express them using words; similarly to limited introspective access of humans.
I do think they lack the distal mental states, which matter much more.
Out of interest, do you have a good argument for this? If so, I’d be really interested to hear it.
Naively, I’d think your example of “Baghdad is the capital of Iraq” encodes enough of the content of ‘Baghdad’ and ‘Iraq’ e.g. other facts about the history of Iraq, the architecture in the city etc.. to meaningfully point towards the distal Baghdad and Iraq. Do you have a different view?
My surprise at your initial comment was related to us using the word ‘distal’ to mean slightly different things.
I’m using it to mean something like “minimally world directed” which is why I was surprised that you’d grant the possibility of LLM’s having mental content without it being minimally world directed. e.g. “Baghdad is the capital of Iraq” already seems minimally world directed if the text-based optimisation builds enough meaning into the concepts ‘Baghdad’ and ‘Iraq.’
It seems like you’re using it to mean something like “the underlying machinery required to integrate world directed contents flexibly into an accurate world model.” For example, AI village or Claude plays Pokémon show that LLM’s still struggle to build accurate enough world models to complete real world tasks.
My usage is more permissive about how accurately the content needs to track the real world to be called ‘distal’, but I don’t think this ends up leading to a substantive disagreement.
I’m probably more optimistic than you that scaling up our current LLM architectures will push this minimal world directed content to become deeper and more flexibly integrable in an accurate world model, but the links you posted are good challenges.
I think you’re just thinking of some of the thought processes, but not about the ones that are most important in the context of writing and reading and public communication. I have no big reason to suspect that LLMs don’t have “mental states” that humans do in order to proximally perform e.g. grammar, sufficient distinctions between synonyms, facts like “Baghdad is the capital of Iraq”, and a huge variety of algorithms. I do think they lack the distal mental states, which matter much more. Feel free to call this “idiosyncratic” but I sure hope you don’t call anything that’s insufficiently concrete / specific / explicit / well-understood “idiosyncratic”, because that means something totally different, and lots of centrally important stuff is insufficiently concrete / specific / explicit / well-understood.
I would describe that position as “I suspect LLMs don’t have distal/deep mental states, and as I mostly care about these distal mental states/representations, LLMs are not doing the important parts of thinking”
Also my guess is you are partially wrong about this. LLMs learn deep abstractions of reality; as these are mostly non-verbal / somewhat far from “tokens”, they are mostly unable to explain or express them using words; similarly to limited introspective access of humans.
Out of interest, do you have a good argument for this? If so, I’d be really interested to hear it.
Naively, I’d think your example of “Baghdad is the capital of Iraq” encodes enough of the content of ‘Baghdad’ and ‘Iraq’ e.g. other facts about the history of Iraq, the architecture in the city etc.. to meaningfully point towards the distal Baghdad and Iraq. Do you have a different view?
I mean the deeper mental algorithms that generate the concepts in the first place, which are especially needed to do e.g. novel science. See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
and
https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense
See also this thread: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce?commentId=dqbLkADbJQJi6bFtN
Thanks! This is helpful to understand your view.
My surprise at your initial comment was related to us using the word ‘distal’ to mean slightly different things.
I’m using it to mean something like “minimally world directed” which is why I was surprised that you’d grant the possibility of LLM’s having mental content without it being minimally world directed. e.g. “Baghdad is the capital of Iraq” already seems minimally world directed if the text-based optimisation builds enough meaning into the concepts ‘Baghdad’ and ‘Iraq.’
It seems like you’re using it to mean something like “the underlying machinery required to integrate world directed contents flexibly into an accurate world model.” For example, AI village or Claude plays Pokémon show that LLM’s still struggle to build accurate enough world models to complete real world tasks.
My usage is more permissive about how accurately the content needs to track the real world to be called ‘distal’, but I don’t think this ends up leading to a substantive disagreement.
I’m probably more optimistic than you that scaling up our current LLM architectures will push this minimal world directed content to become deeper and more flexibly integrable in an accurate world model, but the links you posted are good challenges.