I’m coming back to this later, curious if you’ve seen the paper on the Global Workspace in LLMs. I haven’t read it yet, but it seems like the “part” of the LLM that does the introspecting. From my limited understanding, though, I’m not convinced that aligning that global workspace is meaningfully separate from just aligning the whole model.
Yotam
Karma: 46
I’m sorry I missed your comment when you first posted it. What an intriguing idea! I wonder how you would identify the introspecting part of the model, or how you would differentially train it.
This is really cool work! I particularly like the taxonomy of self-awareness. Have you seen whether steering awareness extends to introspection about unsteered activations?
The downside of this is that you need to have the forethought and time and comput to finetune a monitor for every activation you want to track.
The emotion vectors paper found one thing that mostly corresponds to emotion concepts, but they are very clear that there could be other representations of emotion present that either function differently or simply weren’t identified by the linear probes. I don’t think we can rule out the possibility that introspection could identify functional emotions that don’t perfectly correspond to whatever process you would use to identify the activations your monitor is finetuned to track. In effect, there really may be a form of privileged access for LLMs, not to their base activations, but to what those activations actually mean.
(That said, I’m still ALSO very interested in the monitor-based options here, and particularly those that feed information from the monitors back to the LLM.)