if an agent consistently acts to navigate the state of the environment into a certain regime, we can call this a “goal of the agent”
if that goal corresponds to states of the environment that we value, the agent is aligned
It is less clear what it means to align an LLM:
Generating words (or other tokens) can be viewed as actions. Aligning LLMs then means: make it say nice things.
Generating words can also be seen as thoughts. An LLM that allows us to easily build aligned agents with the right mix of prompting and scaffolding could be called aligned.
One definition that a friend proposed is: an LLM is aligned if it can never serve as the cognition engine for a misaligned agent—this interpretation most strongly emphasizes the “harmlessness” aspect of LLM alignment
Probably, we should have different alignment goals for different deployment cases: LLM assistants should say nice and harmless things, while agents that help automate alignment research should be free to think anything they deem useful, and reason about the harmlessness of various actions “out loud” in their CoT, rather than implicitly in a forward pass.
I think even your definition of what it means for an agent to be aligned is a bit underspecified because it doesn’t distinguish between two possibilities:
Is the agent creating positive outcomes because it trades and compromises with us, creating a mutually beneficial situation that benefits both us and the agent, or
Is the agent creating positive outcomes because it inherently “values what we value”, i.e. its utility function overlaps with ours, and it directly pursues what we want from it, with no compromises?
Definition (1) is more common in the human world. We say that a worker is aligned with us if they do their job as instructed (receiving a wage in return). Definition (2) is more common in theoretical discussions of AI alignment, because people frequently assume that compromise is either unnecessary or impossible, as a strategy that we can take in an AI-human scenario.
By itself, the meaning you gave appears to encompass both definitions, but it seems beneficial to clarify which of these definitions you’d consider closer to the “spirit” of the word “aligned”. It’s also important to specify what counts as a good outcome by our values if these things are a matter of degree, as opposed to being binary. As they say, clear thinking requires making distinctions.
I want to formulate what emotions are from the perspective of an observer that has no emotions itself. Emotions have a close relationship with consciousness, and similar to the hard problem of consciousness, it is not obvious how to know what another mind feels like. It could be that one person perceives emotions 1000x as strong as another person, but the two different emotional experiences lead to exactly the same behavior. Or it could be that one species perceives emotions on a different intensity scale than another one. This creates a challenge for utilitarians: if you want to maximize the happiness of all beings in the universe, you need a way of aggregating happiness between beings.
So, how can we approach this question? We can start by trying to describe the observable properties of emotions as good as we can:
An observable property of consciousness is that humans discuss consciousness, and the same is true for emotions. More specifically, we often talk about how to change or process emotions, because this is something we want and because by conscious thoughts, we can affect our emotions.
Emotions occur in animals (humans) that likely evolved through evolution. It is therefore likely that emotions had a positive effect on the reproductive fitness of some animals.
Emotions affect the behavior and thinking of the mind that experiences them. Concrete examples are:
effect on the overall activity: tired, sad, peaceful feelings cause the experiencer to be less active, while awake, stressed, excited feelings cause the experiencer to be more active
effect on the interpretation of others: angry, grumpy feelings cause the experiencer to assess others as more evil. Happy feelings cause the experiencer to assess others as more good.
effect on short term goals: feeling hungry causes that experiencer wants to eat, sleepy that they want to sleep, horny that they want to have sex, etc
Emotions appear to be correlated with the change of expected fitness of an animal—the worst types of pain are correlated with life-threatening injuries (where expected fitness drastically goes down), and the greatest types of happiness are of lower magnitude because fitness rarely increases suddenly.
Emotions are closest to what we optimize for
we want to be happy, excited, feel love etc and avoid feeling pain, boredom, humiliation etc
other goals are usually instrumental to experiencing these feelings
Emotions are not downstream of abstract reasoning, but abstract reasoning can affect emotions: children, for example, experience emotions before they are able to analytically reflect on them. Emotional responses also happen faster than analytical thoughts.
My intermediate conclusion is that emotions likely evolved because they are computationally efficient proxies for how good the current state is and how to spend energy. They can be viewed as latent variables that often yielded fitness-increasing behavior, whose impact extends beyond the situations in which it actually proves useful—for example, when I get grumpy because I’m hungry.
If this is true, emotions are more useful when a being is less capable of abstract reasoning, therefore less intelligent animals might experience emotions stronger rather than weaker. This fits with the observation that intelligent humans can reduce their suffering via meditation, or that pets seem to suffer more from getting a vaccine than adult humans. However this is a bit of a leap and I have low confidence in it.
Regarding digital sentience, this theory would predict that emotions are more likely to emerge when optimization pressure exists that lets an AI decide how to spend energy. This is not the case in language model pretraining, but is the case in most forms of RL. Again, I am not very confident in this conclusion.
What does it mean to align an LLM?
It is very clear what it means to align an agent:
an agent acts in an environment
if an agent consistently acts to navigate the state of the environment into a certain regime, we can call this a “goal of the agent”
if that goal corresponds to states of the environment that we value, the agent is aligned
It is less clear what it means to align an LLM:
Generating words (or other tokens) can be viewed as actions. Aligning LLMs then means: make it say nice things.
Generating words can also be seen as thoughts. An LLM that allows us to easily build aligned agents with the right mix of prompting and scaffolding could be called aligned.
One definition that a friend proposed is: an LLM is aligned if it can never serve as the cognition engine for a misaligned agent—this interpretation most strongly emphasizes the “harmlessness” aspect of LLM alignment
Probably, we should have different alignment goals for different deployment cases: LLM assistants should say nice and harmless things, while agents that help automate alignment research should be free to think anything they deem useful, and reason about the harmlessness of various actions “out loud” in their CoT, rather than implicitly in a forward pass.
I think even your definition of what it means for an agent to be aligned is a bit underspecified because it doesn’t distinguish between two possibilities:
Is the agent creating positive outcomes because it trades and compromises with us, creating a mutually beneficial situation that benefits both us and the agent, or
Is the agent creating positive outcomes because it inherently “values what we value”, i.e. its utility function overlaps with ours, and it directly pursues what we want from it, with no compromises?
Definition (1) is more common in the human world. We say that a worker is aligned with us if they do their job as instructed (receiving a wage in return). Definition (2) is more common in theoretical discussions of AI alignment, because people frequently assume that compromise is either unnecessary or impossible, as a strategy that we can take in an AI-human scenario.
By itself, the meaning you gave appears to encompass both definitions, but it seems beneficial to clarify which of these definitions you’d consider closer to the “spirit” of the word “aligned”. It’s also important to specify what counts as a good outcome by our values if these things are a matter of degree, as opposed to being binary. As they say, clear thinking requires making distinctions.
What are emotions?
I want to formulate what emotions are from the perspective of an observer that has no emotions itself. Emotions have a close relationship with consciousness, and similar to the hard problem of consciousness, it is not obvious how to know what another mind feels like. It could be that one person perceives emotions 1000x as strong as another person, but the two different emotional experiences lead to exactly the same behavior. Or it could be that one species perceives emotions on a different intensity scale than another one. This creates a challenge for utilitarians: if you want to maximize the happiness of all beings in the universe, you need a way of aggregating happiness between beings.
So, how can we approach this question? We can start by trying to describe the observable properties of emotions as good as we can:
An observable property of consciousness is that humans discuss consciousness, and the same is true for emotions. More specifically, we often talk about how to change or process emotions, because this is something we want and because by conscious thoughts, we can affect our emotions.
Emotions occur in animals (humans) that likely evolved through evolution. It is therefore likely that emotions had a positive effect on the reproductive fitness of some animals.
Emotions affect the behavior and thinking of the mind that experiences them. Concrete examples are:
effect on the overall activity: tired, sad, peaceful feelings cause the experiencer to be less active, while awake, stressed, excited feelings cause the experiencer to be more active
effect on the interpretation of others: angry, grumpy feelings cause the experiencer to assess others as more evil. Happy feelings cause the experiencer to assess others as more good.
effect on short term goals: feeling hungry causes that experiencer wants to eat, sleepy that they want to sleep, horny that they want to have sex, etc
Emotions appear to be correlated with the change of expected fitness of an animal—the worst types of pain are correlated with life-threatening injuries (where expected fitness drastically goes down), and the greatest types of happiness are of lower magnitude because fitness rarely increases suddenly.
Emotions are closest to what we optimize for
we want to be happy, excited, feel love etc and avoid feeling pain, boredom, humiliation etc
other goals are usually instrumental to experiencing these feelings
Emotions are not downstream of abstract reasoning, but abstract reasoning can affect emotions: children, for example, experience emotions before they are able to analytically reflect on them. Emotional responses also happen faster than analytical thoughts.
My intermediate conclusion is that emotions likely evolved because they are computationally efficient proxies for how good the current state is and how to spend energy. They can be viewed as latent variables that often yielded fitness-increasing behavior, whose impact extends beyond the situations in which it actually proves useful—for example, when I get grumpy because I’m hungry.
If this is true, emotions are more useful when a being is less capable of abstract reasoning, therefore less intelligent animals might experience emotions stronger rather than weaker. This fits with the observation that intelligent humans can reduce their suffering via meditation, or that pets seem to suffer more from getting a vaccine than adult humans. However this is a bit of a leap and I have low confidence in it.
Regarding digital sentience, this theory would predict that emotions are more likely to emerge when optimization pressure exists that lets an AI decide how to spend energy. This is not the case in language model pretraining, but is the case in most forms of RL. Again, I am not very confident in this conclusion.
Sounds a lot like this, or also my thoughts, or also shard theory!