You are right, we are probing a very undefined matter. Thank you for sharing your research and the neuron explainer – these are very helpful.
”″Good” is going to be relationally defined; the area the machine thinks about sunshine and lollypops is the “good” part and we only know that because the “good” stuff is there (i.e. without reference to sunshine and lollypops, which to be explicit is just a stand-in for general “good” things, we can’t define “good” really). What you want to know is whether the thoughts about “kill all humans” are more similar to sunshine and lollypops or fear, piss, and death for the machine (fear, piss, and death being concepts likely on a bad end of any dichotomous good-bad principle component),” – that is an interesting research directions. It is probably a good idea to take multiple different concepts, which are more defined (like sunshine and lollypops), and use them as reference point, and try to map more abstract concepts somewhere between them.
”[T]he LLM may not see traditional Human valence as salient in any way; an embedding in a similarity space may find sunshine next to piss and death strictly between sunshine and lollypops. In such a case Human value may be completely alien to the LLM, indeed the concept of value in general may be alien to such a system,”—that would still be a good finding. At least we would be able to locate these concepts. And then we can start thinking how to deal with lollypops and death being closely connected.
You are right, we are probing a very undefined matter. Thank you for sharing your research and the neuron explainer – these are very helpful.
”″Good” is going to be relationally defined; the area the machine thinks about sunshine and lollypops is the “good” part and we only know that because the “good” stuff is there (i.e. without reference to sunshine and lollypops, which to be explicit is just a stand-in for general “good” things, we can’t define “good” really). What you want to know is whether the thoughts about “kill all humans” are more similar to sunshine and lollypops or fear, piss, and death for the machine (fear, piss, and death being concepts likely on a bad end of any dichotomous good-bad principle component),” – that is an interesting research directions. It is probably a good idea to take multiple different concepts, which are more defined (like sunshine and lollypops), and use them as reference point, and try to map more abstract concepts somewhere between them.
”[T]he LLM may not see traditional Human valence as salient in any way; an embedding in a similarity space may find sunshine next to piss and death strictly between sunshine and lollypops. In such a case Human value may be completely alien to the LLM, indeed the concept of value in general may be alien to such a system,”—that would still be a good finding. At least we would be able to locate these concepts. And then we can start thinking how to deal with lollypops and death being closely connected.