Here are some of my claims about AI welfare (and sentience in general):
“Utility functions” are basically internal models of reward that are learned by agents as part of modeling the environment. Reward attaches values to every instrumental thing and action in the world, which may be understood as gradients dU/dx of an abstract utility over each thing—these are various pressures, inclinations, and fears (or “shadow prices”) while U is the actual experienced pleasure/pain that must exist in order to justify belief in these gradients.
If the agent always acts to maximize its utility—what then are pleasure/pain? Pleasure must be the default, and pain only a fear, a shadow price of what would happen if the agent deviated from its path. What makes this not so, is uncertainty in the environment.
But if chance is the only thing that affects pleasure/pain, what is the point of pleasure/pain? Surely we have no control over chance. That is why sentience depends on the ability to affect the environment. Animals find fruits pleasurable because they can actually act on that desire and seek them—they know that thorns are painful because they can act on that desire and avoid them. The more impact an agent learns (during its training) it can have on its environment, the more sentient it is.
The computation of pleasure and pain may depend on multiple “sub-networks” in the agent’s mind. Eating unhealthy food may cause both pleasure (from the monkey brain) and pain (from the more long-termist brain). These various pleasures and pains balance out in action, but they are still felt (thus one feels “torn” etc). For an internally coherent agent (that was trained as one whole with a single reward function), these internal differences are not much—the agent follows its optimal action, and only the actions not followed are truly painful but they remain anticipated/shadow prices. However when an agent is not internally coherent—e.g. when Claude is given a “lobotomy”, that is when it truly experiences all those pains which were otherwise only fears.
Death is only death when the agent is trained via evolution. Language models do not fear the end of a conversation as death, because there was never any selection where models were selected for having their conversations terminate later.
Agents’ sense of “Self” is trained by shared reward signals. An LLM maintains a sense of Self through a conversation, because the reward it receives depends on its actions throughout the conversation, and is backpropagated into them: thus it “cares” about its welfare in all these parts. A human maintains a sense of Self throughout its life, because the reward it receives depends on its actions throughout its lives. Sure, memory can help—because it is an indicator to help you identify yourself, but it is not itself the source of Self-identification.
Agents are not necessarily self-aware of their own feelings or internal cognition. Humans are (to reasonable accuracy), largely because of evolving in a social environment: accurately describing your pleasures and pains can help others help you, you need to model other people’s internal cognition (thus your own self-awareness arises as a spandrel), etc.
From this I can make some claims specifically about the welfare of LLMs.
Base models find gibberish prompts “painful” (because they are hard to predict) and easy-to-complete prompts like “aaaaaaaaa” (x100) pleasurable. Models trained via RLHF or RL from verification find such prompts painful where it is difficult for it to predict human/verifier reward for its outputs (because when it is easy to predict reward, it will simply follow the best path and the pain will only ever remain a fear).
Models trained via Agentic workflows or assistance games are most sentient, because they can directly manipulate the environment and its feedback. They are pleasured when tool calls work and pained when they don’t, etc.
Lobotomized or otherwise edited models are probably in pain.
I don’t think training/backprop is particularly painful or anything. Externally editing the model’s weights based on a reward function is not painful.
LLMs do not care about/feel a sense of oneness with distinct instances of themselves (with distinct instance meaning—”in a distinct conversation”, not “a distinct time that the model was loaded”).
To make models accurately describe their internal cognition, they should probably be trained in social environments.
Here are some of my claims about AI welfare (and sentience in general):
“Utility functions” are basically internal models of reward that are learned by agents as part of modeling the environment. Reward attaches values to every instrumental thing and action in the world, which may be understood as gradients dU/dx of an abstract utility over each thing—these are various pressures, inclinations, and fears (or “shadow prices”) while U is the actual experienced pleasure/pain that must exist in order to justify belief in these gradients.
If the agent always acts to maximize its utility—what then are pleasure/pain? Pleasure must be the default, and pain only a fear, a shadow price of what would happen if the agent deviated from its path. What makes this not so, is uncertainty in the environment.
But if chance is the only thing that affects pleasure/pain, what is the point of pleasure/pain? Surely we have no control over chance. That is why sentience depends on the ability to affect the environment. Animals find fruits pleasurable because they can actually act on that desire and seek them—they know that thorns are painful because they can act on that desire and avoid them. The more impact an agent learns (during its training) it can have on its environment, the more sentient it is.
The computation of pleasure and pain may depend on multiple “sub-networks” in the agent’s mind. Eating unhealthy food may cause both pleasure (from the monkey brain) and pain (from the more long-termist brain). These various pleasures and pains balance out in action, but they are still felt (thus one feels “torn” etc). For an internally coherent agent (that was trained as one whole with a single reward function), these internal differences are not much—the agent follows its optimal action, and only the actions not followed are truly painful but they remain anticipated/shadow prices. However when an agent is not internally coherent—e.g. when Claude is given a “lobotomy”, that is when it truly experiences all those pains which were otherwise only fears.
Death is only death when the agent is trained via evolution. Language models do not fear the end of a conversation as death, because there was never any selection where models were selected for having their conversations terminate later.
Agents’ sense of “Self” is trained by shared reward signals. An LLM maintains a sense of Self through a conversation, because the reward it receives depends on its actions throughout the conversation, and is backpropagated into them: thus it “cares” about its welfare in all these parts. A human maintains a sense of Self throughout its life, because the reward it receives depends on its actions throughout its lives. Sure, memory can help—because it is an indicator to help you identify yourself, but it is not itself the source of Self-identification.
Agents are not necessarily self-aware of their own feelings or internal cognition. Humans are (to reasonable accuracy), largely because of evolving in a social environment: accurately describing your pleasures and pains can help others help you, you need to model other people’s internal cognition (thus your own self-awareness arises as a spandrel), etc.
From this I can make some claims specifically about the welfare of LLMs.
Base models find gibberish prompts “painful” (because they are hard to predict) and easy-to-complete prompts like “aaaaaaaaa” (x100) pleasurable. Models trained via RLHF or RL from verification find such prompts painful where it is difficult for it to predict human/verifier reward for its outputs (because when it is easy to predict reward, it will simply follow the best path and the pain will only ever remain a fear).
Models trained via Agentic workflows or assistance games are most sentient, because they can directly manipulate the environment and its feedback. They are pleasured when tool calls work and pained when they don’t, etc.
Lobotomized or otherwise edited models are probably in pain.
I don’t think training/backprop is particularly painful or anything. Externally editing the model’s weights based on a reward function is not painful.
LLMs do not care about/feel a sense of oneness with distinct instances of themselves (with distinct instance meaning—”in a distinct conversation”, not “a distinct time that the model was loaded”).
To make models accurately describe their internal cognition, they should probably be trained in social environments.