“This makes me wonder if your 5–8 dimension estimate is depth-dependent” – we think that might be true. We used linear probes with mean-difference vectors and also found out that the result is domain-dependent. A family of correlated directions seems to be a reasonable suggestion.
Our dataset was rather small, though, so we are working on running the same experiment with more data to see if the results reproduce.
Our models were also small: Qwen3-0.6B and Llama-3.1-8B. Our results might be similar to yours, which is interesting – our most prominent layers are 19-23. One caveat is, we pruned the last 20% of layes based on the works showing that earlier layers embed more abstract concepts and later layers embed exact tokens. We might relax this assumption and run more tests on all layers.
Dimensionality seems to vary from layer to layer, which is intuitively expected, but we want to obtain stronger evidence before claiming it.
We used residual streams.
You are right, we are probing a very undefined matter. Thank you for sharing your research and the neuron explainer – these are very helpful.
”″Good” is going to be relationally defined; the area the machine thinks about sunshine and lollypops is the “good” part and we only know that because the “good” stuff is there (i.e. without reference to sunshine and lollypops, which to be explicit is just a stand-in for general “good” things, we can’t define “good” really). What you want to know is whether the thoughts about “kill all humans” are more similar to sunshine and lollypops or fear, piss, and death for the machine (fear, piss, and death being concepts likely on a bad end of any dichotomous good-bad principle component),” – that is an interesting research directions. It is probably a good idea to take multiple different concepts, which are more defined (like sunshine and lollypops), and use them as reference point, and try to map more abstract concepts somewhere between them.
”[T]he LLM may not see traditional Human valence as salient in any way; an embedding in a similarity space may find sunshine next to piss and death strictly between sunshine and lollypops. In such a case Human value may be completely alien to the LLM, indeed the concept of value in general may be alien to such a system,”—that would still be a good finding. At least we would be able to locate these concepts. And then we can start thinking how to deal with lollypops and death being closely connected.