Directed modulation is the ability to maintain a concept in one’s mind, even if it’s not verbalized. Here, we ask Qwen to think about ocean creatures while repeating an unrelated phrase. In the J-space of Qwen’s response, we observe readouts fish, coral, whale, and lobster. To check that Qwen isn’t simply associating these tokens with the phrase “ocean creatures”, we run a similar test, but ask Qwen to not think about ocean creatures (it doesn’t!).
We return to the thought-suppression protocol of §3.2, in which we found that models comply only imperfectly with an instruction not to think about a concept—the concept often appears in the J-space despite the instruction.
Great catch! I (maker of this demo) flagged this during testing and this was the response from the paper’s author:
“We also observed reduced thinking rates for sonnet when instructed to “ignore X” but the results here are pretty subtle and it seems to vary a lot by prompting. I think its a neat example and is good to include!”
I think it doesn’t seem inconsistent (they‘re both less), especially given significant differences in the tests (different models—maybe qwen is smaller and smaller models are less complex thinkers? no idea how big sonnet is. also prompting, etc) One other big diff is that neuronpedia maxes out at top 8 per layer per token position due to resource constraints, while Ant probably used a lot more.
In the demo it says
But (BUT) the paper says
Great catch! I (maker of this demo) flagged this during testing and this was the response from the paper’s author:
“We also observed reduced thinking rates for sonnet when instructed to “ignore X” but the results here are pretty subtle and it seems to vary a lot by prompting. I think its a neat example and is good to include!”
I think it doesn’t seem inconsistent (they‘re both less), especially given significant differences in the tests (different models—maybe qwen is smaller and smaller models are less complex thinkers? no idea how big sonnet is. also prompting, etc) One other big diff is that neuronpedia maxes out at top 8 per layer per token position due to resource constraints, while Ant probably used a lot more.