The actions of such a person can be simulated faithfully very easily without simulating any of their thoughts or sensations.
You can still talk with such a person by reading their brain state from a superpowered-fMRI-from-the-future, and them listening to your words.
(Talking to someone/interacting with someone’s behavior is just a simplified way of saying “both-sided information transfer with the system,” where you transmit the information to the system (in whatever way) and the system will generate the response the person is giving (also in whatever way).)
To the extent to which your thoughts and feelings are connected to your consciousness in any way, they can be elicited, either by the LLM computing your response (because they impact your words somehow), or by me asking how you feel (and LLM therefore having to figure out the answer).
To the extent to which your thoughts and feelings never influence your output in any way for any possible input, their existence is meaningless.
You can still talk with such a person by reading their brain state from a superpowered-fMRI-from-the-future, and them listening to your words.
(Talking to someone/interacting with someone’s behavior is just a simplified way of saying “both-sided information transfer with the system,” where you transmit the information to the system (in whatever way) and the system will generate the response the person is giving (also in whatever way).)
To the extent to which your thoughts and feelings are connected to your consciousness in any way, they can be elicited, either by the LLM computing your response (because they impact your words somehow), or by me asking how you feel (and LLM therefore having to figure out the answer).
To the extent to which your thoughts and feelings never influence your output in any way for any possible input, their existence is meaningless.