Things in this genre of privileged embedding seem valuable. It also seems doable to think architecturally beyond just embedding (eg, attention heads that only attend to certain roles, etc). Myself and others been trying to explore this genre some this month, but in some pilots gains seem unfortunately small compared to baselines. Personally I would be excited to see more of this though, as various ways of offering model-level guarantees seems good.
There’s additional issues here where roles are just an incomplete primitive. When we were exploring this last month observed how models in some ways seem to trust tools more than users. The role is overloaded primitive that is both supposed tell you both a hierarchy of trust and the information source. Often things in tools role are pseudo-oracles (eg, results of a python script, web results on current events, etc), and other times they are not (eg, reading an untrusted markdown file).
Charles and coauthors’ work here on embedding representation is really cool. It would be cool to further expand this, also thinking about interactions with like “trust embeddings” or other ways role type is serving multiple purposes, and how to build primitives or architectures to support this.
Cool work! Really interesting paper and I like the framing on role overloading. Anecdotally, I’ve noticed some LLMs do web searches a lot to determine truth when they think I’m lying. I wonder whether splitting security and source into separate roles would be useful.
Thanks! right, this thing around web searches matches my view as well. If a user asks about something important or weird, it’s usually best to trust the top 3 google results more than the user themself. But if in most of training traces are always trusting the tools with just a little specific adversarial post-training, end up with a model that still mostly trusts tools. This is maybe solvable with better training data, but becomes more nuanced than OpenAI’s model spec that gives tools are “no authority”.
A simple split with Trust∈{untrusted, trusted} and Source∈{system, user, tool} is tough though. There are rare cases where almost all tools are adversarial (eg, a malicious webpage). You’d end up basically always using Trust=untrusted, and be back in the same place. Unfortunately I don’t have clear views on solutions currently other than small improvements like maybe updating the spec to be more nuanced/self-consistent (plus special primitives of being able inline different sources into a prompt template seems nice). When tracking failures through embeddings like in this study, it seems like trust and source are different axises though.
Things in this genre of privileged embedding seem valuable. It also seems doable to think architecturally beyond just embedding (eg, attention heads that only attend to certain roles, etc). Myself and others been trying to explore this genre some this month, but in some pilots gains seem unfortunately small compared to baselines. Personally I would be excited to see more of this though, as various ways of offering model-level guarantees seems good.
There’s additional issues here where roles are just an incomplete primitive. When we were exploring this last month observed how models in some ways seem to trust tools more than users. The role is overloaded primitive that is both supposed tell you both a hierarchy of trust and the information source. Often things in tools role are pseudo-oracles (eg, results of a python script, web results on current events, etc), and other times they are not (eg, reading an untrusted markdown file).
Charles and coauthors’ work here on embedding representation is really cool. It would be cool to further expand this, also thinking about interactions with like “trust embeddings” or other ways role type is serving multiple purposes, and how to build primitives or architectures to support this.
Cool work! Really interesting paper and I like the framing on role overloading. Anecdotally, I’ve noticed some LLMs do web searches a lot to determine truth when they think I’m lying. I wonder whether splitting security and source into separate roles would be useful.
Thanks! right, this thing around web searches matches my view as well. If a user asks about something important or weird, it’s usually best to trust the top 3 google results more than the user themself. But if in most of training traces are always trusting the tools with just a little specific adversarial post-training, end up with a model that still mostly trusts tools. This is maybe solvable with better training data, but becomes more nuanced than OpenAI’s model spec that gives tools are “no authority”.
A simple split with Trust∈{untrusted, trusted} and Source∈{system, user, tool} is tough though. There are rare cases where almost all tools are adversarial (eg, a malicious webpage). You’d end up basically always using Trust=untrusted, and be back in the same place. Unfortunately I don’t have clear views on solutions currently other than small improvements like maybe updating the spec to be more nuanced/self-consistent (plus special primitives of being able inline different sources into a prompt template seems nice). When tracking failures through embeddings like in this study, it seems like trust and source are different axises though.