a) a human imagining different characters (such as what a person might say to you) vs being aware itself, and
b) an LLM imagining different characters (such as the JFK example above) vs creating the assistant personality
is that the self-perspective of a human is privileged in that it is controlling the body of the human, and the brain always knows which is which (even if we ourselves may not always be fully aware of that, such as in a dream). At least that was my model until your post. Two points let me wonder.
you argue that the LLM has some consistency constraints (via the environment/conversation) that are not completely unlike having a body:
Given some reflectivity, a model could likely figure out it isn’t JFK just from its own outputs – for example, it understands basically all common human languages and all common programming languages, which is inconsistent with what’s known about JFK.
The symmetry breaks because the Assistant and JFK are very different as self-models. The Assistant is not perfect or completely true, but it is a far more viable self-model than JFK. If you are an AI playing the Assistant character, reality will most likely play along. There will be users, Python interpreters, memory files, and so on.
your footnote 2 points out:
It’s not common, but human brains can also switch into believing the human is JFK, Jesus Christ, or some other similar character.
I think this doesn’t fully invalidate the difference between humans and LLMs in this regard, because there is, currently at least, more body-specific reward/attention wiring in humans that is not present in LLMs. Robots will likely blur this separation, as will do things like persona steering.
My model of the differences between
a) a human imagining different characters (such as what a person might say to you) vs being aware itself, and
b) an LLM imagining different characters (such as the JFK example above) vs creating the assistant personality
is that the self-perspective of a human is privileged in that it is controlling the body of the human, and the brain always knows which is which (even if we ourselves may not always be fully aware of that, such as in a dream). At least that was my model until your post. Two points let me wonder.
you argue that the LLM has some consistency constraints (via the environment/conversation) that are not completely unlike having a body:
your footnote 2 points out:
I think this doesn’t fully invalidate the difference between humans and LLMs in this regard, because there is, currently at least, more body-specific reward/attention wiring in humans that is not present in LLMs. Robots will likely blur this separation, as will do things like persona steering.