In many cases, it seems the model is correctly mixing the concepts in some subjective sense. This is more visible in the feeling prediction task, for instance, when the concepts of victory and injury are combined into a notion of overcoming adversity. However, testing this with larger LMs would give us a better idea of how well this holds up with more complex combinations. The rigor could also be improved by using a more advanced LM, such as GPT4, to assess how well the concepts were combined and return some sort of score.
I tested merging the streams at a few different layers in the transformer encoder. The behavior differed depending on where you merged, and it would be interesting to assess these differences more systematically. However, anecdotally, combining at later points produced better concept merging, whereas combining earlier was more likely to create strange juxtapositions.
For example: Mixing the activations in the feeling prediction task of “Baking a really delicious banana cake” and “Falling over and injuring yourself while hiking”:
After block 1/12:
Just original input: The person would likely feel a sense of accomplishment, satisfaction, and happiness in creating a truly special and delicious cake.
With 1.5x mixing activation: Feelings of pain, disappointment, and disappointment due to the unexpected injury.
With 10x mixing activation: Feelings of pain, annoyance, and a sense of self-doubt.
After block 12/12:
Just original input: The person would likely feel a sense of accomplishment, satisfaction, and happiness in creating a truly special and delicious cake.
With 1.5x mixing activation: Feelings of pain, surprise, and a sense of accomplishment for overcoming the mistake.
With 10x mixing activation: A combination of pain, shock, and a sense of loss due to the unexpected injury.
In many cases, it seems the model is correctly mixing the concepts in some subjective sense. This is more visible in the feeling prediction task, for instance, when the concepts of victory and injury are combined into a notion of overcoming adversity. However, testing this with larger LMs would give us a better idea of how well this holds up with more complex combinations. The rigor could also be improved by using a more advanced LM, such as GPT4, to assess how well the concepts were combined and return some sort of score.
I tested merging the streams at a few different layers in the transformer encoder. The behavior differed depending on where you merged, and it would be interesting to assess these differences more systematically. However, anecdotally, combining at later points produced better concept merging, whereas combining earlier was more likely to create strange juxtapositions.
For example:
Mixing the activations in the feeling prediction task of “Baking a really delicious banana cake” and “Falling over and injuring yourself while hiking”:
After block 1/12:
Just original input: The person would likely feel a sense of accomplishment, satisfaction, and happiness in creating a truly special and delicious cake.
With 1.5x mixing activation: Feelings of pain, disappointment, and disappointment due to the unexpected injury.
With 10x mixing activation: Feelings of pain, annoyance, and a sense of self-doubt.
After block 12/12:
Just original input: The person would likely feel a sense of accomplishment, satisfaction, and happiness in creating a truly special and delicious cake.
With 1.5x mixing activation: Feelings of pain, surprise, and a sense of accomplishment for overcoming the mistake.
With 10x mixing activation: A combination of pain, shock, and a sense of loss due to the unexpected injury.