Seems very related: Linear Spaces of Meanings: Compositional Structures in Vision-Language Models. Notably, the (approximate) compositionality of language/​reality should bode well for the scalability of linear activation engineering methods.
And this structure can be used as regularization for soft prompts.
Seems very related: Linear Spaces of Meanings: Compositional Structures in Vision-Language Models. Notably, the (approximate) compositionality of language/​reality should bode well for the scalability of linear activation engineering methods.
And this structure can be used as regularization for soft prompts.