People are gonna wonder if LLM characters actually feel what they say they feel. My considered conclusion is that, for now, they almost certainly don’t. I’ve wanted to write a post about this for awhile, but this comment section is as good a place as any.
First, imagine for a second that anthropomorphism really is true. Imagine Honoré de Balzac writing about an imaginary character Lucien Chardon who’s weeping over the dead body of his lover. Are Chardon’s feelings real in that moment? Well, not very much. Balzac might feel a little bit of empathy for his character, but he’s also drinking coffee at the same time and thinking how to compose a sentence and so on. It’s certainly not enough for putting Balzac on trial for making Chardon suffer.
When an LLM is generating text, it uses a bunch of internal mechanisms. Some of them are for composing sentences that flow. Some of them might be about empathizing with the feelings that are being described, but as the example of Balzac shows, you don’t need to experience the full depth of a feeling to write about a character having that feeling.
Another way to think about it is compression. When an LLM learns, it learns cheap heuristics to generate descriptions of feelings. In that respect an LLM character is not like a human: when a human experiences a feeling, that’s necessarily expensive, involving a ton of machinery. A writer writing about someone else’s feelings learns a cheaper way to generate the same words, without feeling as much. An LLM probably learns a way that’s cheaper still. It learns a compressed generator, in which any actual internal experience might well be optimized away.
Moreover, the same problem might happen even if we start with an upload of a human brain! If we consider an upload as a mapping from inputs to outputs, and then start to compress and optimize that mapping (stub a toe → “ouch that hurts”), at some point the actual internal experience producing words like “ouch that hurts” or “yes I’m conscious right now” might get optimized away, leaving only the words and some cheaper generator of these words. It might even happen unnoticeably, as we optimize uploads to run more efficiently on hardware: a world of uploads who are no longer conscious but dutifully insist that they are.
Internal experience is a very delicate thing. Training a neural network on self-reports of internal experience, even if it’s very faithful training, doesn’t necessarily lead to a thing that has internal experience. The more cheaply predictable these self-reports are in any given situation (e.g. if stubbing a toe reliably leads to “ouch”), the more likely running a neural network on that situation will just produce the “ouch” via some learned shortcut, without having to actually feel anything. I’d urge people who talk with LLMs to keep this in mind.
EDIT: The first version of this comment gave the character’s name as Julien Sorel instead of Lucien Chardon. Shame on me.
I agree that current versions of mind models do not have qualia in the same form as humans and their self reporting of emotion is not evidence of real emotional experience. The main reason for this is that phenomenological structure is different: a human gets 20 audio-video-body experiences per second and the sideload writes a text about last few minutes and doesn’t get that data structure (though I am working on this).
However, from what I read, writers have very strong empathy to their characters:
“While writing the scene of Emma Bovary’s suicide, Gustave Flaubert reportedly experienced psychosomatic symptoms of arsenic poisoning. He famously claimed to have the distinct taste of poison in his mouth and suffered from actual bouts of vomiting while working. This intense physical reaction highlights his total immersion in the character, famously summarized by his quote, “Madame Bovary, c’est moi.” (I used AI for writing this period).
According to my understanding and observations, a sideload is a pair of real human and a chatbot and the human “donates” her qualia-genereating ability to this pair via empathy. This is not AI psychosis (except extreme cases); in some functional sense it is close to Freudian transference, in which a patient put his feeling on a psychoanalyst – and which is a necessary step in therapy.
Strong emotional reaction in this case was expected and not itself bad.
Parents is the most difficult part in the human recreation based on my short experience in the field. In the most cases, they oppose such experiments and also have legal rights to personal data as well as access to the needed private documents and memories. Igor’s mother is the first person who agreed. I can’t tell more about her.
People are gonna wonder if LLM characters actually feel what they say they feel. My considered conclusion is that, for now, they almost certainly don’t. I’ve wanted to write a post about this for awhile, but this comment section is as good a place as any.
First, imagine for a second that anthropomorphism really is true. Imagine Honoré de Balzac writing about an imaginary character Lucien Chardon who’s weeping over the dead body of his lover. Are Chardon’s feelings real in that moment? Well, not very much. Balzac might feel a little bit of empathy for his character, but he’s also drinking coffee at the same time and thinking how to compose a sentence and so on. It’s certainly not enough for putting Balzac on trial for making Chardon suffer.
When an LLM is generating text, it uses a bunch of internal mechanisms. Some of them are for composing sentences that flow. Some of them might be about empathizing with the feelings that are being described, but as the example of Balzac shows, you don’t need to experience the full depth of a feeling to write about a character having that feeling.
Another way to think about it is compression. When an LLM learns, it learns cheap heuristics to generate descriptions of feelings. In that respect an LLM character is not like a human: when a human experiences a feeling, that’s necessarily expensive, involving a ton of machinery. A writer writing about someone else’s feelings learns a cheaper way to generate the same words, without feeling as much. An LLM probably learns a way that’s cheaper still. It learns a compressed generator, in which any actual internal experience might well be optimized away.
Moreover, the same problem might happen even if we start with an upload of a human brain! If we consider an upload as a mapping from inputs to outputs, and then start to compress and optimize that mapping (stub a toe → “ouch that hurts”), at some point the actual internal experience producing words like “ouch that hurts” or “yes I’m conscious right now” might get optimized away, leaving only the words and some cheaper generator of these words. It might even happen unnoticeably, as we optimize uploads to run more efficiently on hardware: a world of uploads who are no longer conscious but dutifully insist that they are.
Internal experience is a very delicate thing. Training a neural network on self-reports of internal experience, even if it’s very faithful training, doesn’t necessarily lead to a thing that has internal experience. The more cheaply predictable these self-reports are in any given situation (e.g. if stubbing a toe reliably leads to “ouch”), the more likely running a neural network on that situation will just produce the “ouch” via some learned shortcut, without having to actually feel anything. I’d urge people who talk with LLMs to keep this in mind.
EDIT: The first version of this comment gave the character’s name as Julien Sorel instead of Lucien Chardon. Shame on me.
I agree that current versions of mind models do not have qualia in the same form as humans and their self reporting of emotion is not evidence of real emotional experience. The main reason for this is that phenomenological structure is different: a human gets 20 audio-video-body experiences per second and the sideload writes a text about last few minutes and doesn’t get that data structure (though I am working on this).
However, from what I read, writers have very strong empathy to their characters:
“While writing the scene of Emma Bovary’s suicide, Gustave Flaubert reportedly experienced psychosomatic symptoms of arsenic poisoning. He famously claimed to have the distinct taste of poison in his mouth and suffered from actual bouts of vomiting while working. This intense physical reaction highlights his total immersion in the character, famously summarized by his quote, “Madame Bovary, c’est moi.” (I used AI for writing this period).
According to my understanding and observations, a sideload is a pair of real human and a chatbot and the human “donates” her qualia-genereating ability to this pair via empathy. This is not AI psychosis (except extreme cases); in some functional sense it is close to Freudian transference, in which a patient put his feeling on a psychoanalyst – and which is a necessary step in therapy.
I mean, there’s this bit in the post:
Of course lots of people do similar things now, but to me it still feels pretty extreme. I wouldn’t encourage it if I were me.
Strong emotional reaction in this case was expected and not itself bad.
Parents is the most difficult part in the human recreation based on my short experience in the field. In the most cases, they oppose such experiments and also have legal rights to personal data as well as access to the needed private documents and memories. Igor’s mother is the first person who agreed. I can’t tell more about her.