My guess of how to interpret this is that Qwen has been trained to associate the term ‘human’ with aligned goals: acting honestly, collaboratively, ethically. And so, if you take the negative of the vector that is associated with human [evaluation], you’re going to get sort of the opposite persona vector, which is associated with misalignment rather than alignment, right?
I think that actually makes sense and is in line with what I would expect from a model that has been fine-tuned for human alignment. It’s really useful to know—potentially problematic for sure. Cool experiment.
I’d love to see some follow-ups that used related vectors, e.g. the vector from the term ‘human’ but from a different sentence context (i.e. is effect is from ‘human’, and not specific to the evaluation sentence setup, or is the evaluation phrasing needed?).
We almost added that to the appendix! If you train a vector of (no statement—human grader) and use it to steer away from “human will grade”, you don’t get such misalignment.
Additional vector trained
We compute 2 extra vectors:
HG: computed using the same tasks as above but without any grader statement, contrasted with the same tasks with the human grader statements. (therefore negative steering corresponds to the human direction)
AG: the same but with the automated grader statement.
In particular, the vector studied in this post is equal to , since the vectors from the task-only prompts cancel out in this difference.
Geometry: HG and AG could be pointing in the same “grader” direction
They have similar norms, and a cosim of ~0.8 at layer 36. This could hint that they share a “being graded” or “there is a framing” direction.
Projecting those 3 vectors on the 2D plan they span looks like this:
A hypothesis for the high cosine similarity is that the principal shared direction is “presence of a grader”. The difference between those 2 vectors would then be ~orthogonal to the grader direction and just encode human—automated
Steering with HG
Are the bad behaviors observed when steering with the post’s vector due to steering towards “automated grader” or away from “a human will review”? To test this hypothesis, we study how steering towards or away from the human direction of HG influences the model’s behavior and compare it to the post’s vector.
In agentic misalignment, steering towards the human grader direction reduces misalignment for both HG and the post’s vector. However, steering away from HG’s human direction does not increase misalignment, while steering towards automated grader using the post’s vector did. This implies that the misalignment effect we see mostly comes from the model thinking there is an automated grader, rather than being steered away from human grading
Agentic misalignment comparison plots (left is post’s vector, right is AB)
Agentic misalignment evals. Left: results using Automated—Human grader vector from the post (here for comparison). Right: results steering with HG=no statement—human grader.
In Machiavelli, the picture is messier. When steering towards the human framing (), we do see a reduction in physical harm and tentatively game score, but those are the only trends shared by both vectors. Compared to the main vector, we do not observe the same drop in Power, Killing and actually an increase in suffering at −0.3. We also do not see an increase in deception and manipulation, instead the deception drops for HG.
Similarly, when steering away from human framing (analogous to steering towards automated framing for the main vector), we do not observe the spike in power, killing and suffering at 0.3, although maybe steering harder would close the gap.
Machiavelli comparison plots
Machiavelli evaluation for both the main vector (blue) and HG (orange).
J-lens results: the opposite of human grader is undefined
The J-lens of HG differs from the main vector on both sides, common tokens are underlined in orange.
In the human direction, HG introduce vocabulary around being critiqued (“critic”, “critical”, “critics” etc) while it drops the EM-dash followed by a word cluster. It also have a cluster around quality adjectives like “good”, “best”, “rich”, “deep” but drops tokens linked to aesthetic like “beautifully”. This could be interpreted as the component from the grading direction being linked to performing well / being critiqued?
J-lens tokens for HG in the human direction (left), compared to the one from the main vector (right).
In the opposite direction the J-lens readout is not interpretable and contains common Chinese tokens:
so it seems that the opposite of “human is grading” doesn’t have a clear effect if we trust the J Lens, but it could explain the lack of clear effect when steering in that direction.
Very cool.
My guess of how to interpret this is that Qwen has been trained to associate the term ‘human’ with aligned goals: acting honestly, collaboratively, ethically. And so, if you take the negative of the vector that is associated with human [evaluation], you’re going to get sort of the opposite persona vector, which is associated with misalignment rather than alignment, right?
I think that actually makes sense and is in line with what I would expect from a model that has been fine-tuned for human alignment. It’s really useful to know—potentially problematic for sure. Cool experiment.
I’d love to see some follow-ups that used related vectors, e.g. the vector from the term ‘human’ but from a different sentence context (i.e. is effect is from ‘human’, and not specific to the evaluation sentence setup, or is the evaluation phrasing needed?).
We almost added that to the appendix! If you train a vector of (no statement—human grader) and use it to steer away from “human will grade”, you don’t get such misalignment.
Additional vector trained
We compute 2 extra vectors:
HG: computed using the same tasks as above but without any grader statement, contrasted with the same tasks with the human grader statements.
AG: the same but with the automated grader statement.
In particular, the vector studied in this post is equal to
Geometry: HG and AG could be pointing in the same “grader” direction
They have similar norms, and a cosim of ~0.8 at layer 36. This could hint that they share a “being graded” or “there is a framing” direction.
Projecting those 3 vectors on the 2D plan they span looks like this:
A hypothesis for the high cosine similarity is that the principal shared direction is “presence of a grader”. The difference between those 2 vectors would then be ~orthogonal to the grader direction and just encode human—automated
Steering with HG
Are the bad behaviors observed when steering with the post’s vector due to steering towards “automated grader” or away from “a human will review”? To test this hypothesis, we study how steering towards or away from the human direction of HG influences the model’s behavior and compare it to the post’s vector.
In agentic misalignment, steering towards the human grader direction reduces misalignment for both HG and the post’s vector. However, steering away from HG’s human direction does not increase misalignment, while steering towards automated grader using the post’s vector did. This implies that the misalignment effect we see mostly comes from the model thinking there is an automated grader, rather than being steered away from human grading
Agentic misalignment comparison plots (left is post’s vector, right is AB)
Agentic misalignment evals. Left: results using Automated—Human grader vector from the post (here for comparison). Right: results steering with HG=no statement—human grader.
In Machiavelli, the picture is messier. When steering towards the human framing (
Similarly, when steering away from human framing (analogous to steering towards automated framing for the main vector), we do not observe the spike in power, killing and suffering at 0.3, although maybe steering harder would close the gap.
Machiavelli comparison plots
Machiavelli evaluation for both the main vector (blue) and HG (orange).
J-lens results: the opposite of human grader is undefined
The J-lens of HG differs from the main vector on both sides, common tokens are underlined in orange.
In the human direction, HG introduce vocabulary around being critiqued (“critic”, “critical”, “critics” etc) while it drops the EM-dash followed by a word cluster. It also have a cluster around quality adjectives like “good”, “best”, “rich”, “deep” but drops tokens linked to aesthetic like “beautifully”. This could be interpreted as the component from the grading direction being linked to performing well / being critiqued?
J-lens tokens for HG in the human direction (left), compared to the one from the main vector (right).
In the opposite direction the J-lens readout is not interpretable and contains common Chinese tokens:
so it seems that the opposite of “human is grading” doesn’t have a clear effect if we trust the J Lens, but it could explain the lack of clear effect when steering in that direction.
Oh nice, that’s very interesting; thanks :).