Cool work! Really interesting paper and I like the framing on role overloading. Anecdotally, I’ve noticed some LLMs do web searches a lot to determine truth when they think I’m lying. I wonder whether splitting security and source into separate roles would be useful.
Thanks! right, this thing around web searches matches my view as well. If a user asks about something important or weird, it’s usually best to trust the top 3 google results more than the user themself. But if in most of training traces are always trusting the tools with just a little specific adversarial post-training, end up with a model that still mostly trusts tools. This is maybe solvable with better training data, but becomes more nuanced than OpenAI’s model spec that gives tools are “no authority”.
A simple split with Trust∈{untrusted, trusted} and Source∈{system, user, tool} is tough though. There are rare cases where almost all tools are adversarial (eg, a malicious webpage). You’d end up basically always using Trust=untrusted, and be back in the same place. Unfortunately I don’t have clear views on solutions currently other than small improvements like maybe updating the spec to be more nuanced/self-consistent (plus special primitives of being able inline different sources into a prompt template seems nice). When tracking failures through embeddings like in this study, it seems like trust and source are different axises though.
Cool work! Really interesting paper and I like the framing on role overloading. Anecdotally, I’ve noticed some LLMs do web searches a lot to determine truth when they think I’m lying. I wonder whether splitting security and source into separate roles would be useful.
Thanks! right, this thing around web searches matches my view as well. If a user asks about something important or weird, it’s usually best to trust the top 3 google results more than the user themself. But if in most of training traces are always trusting the tools with just a little specific adversarial post-training, end up with a model that still mostly trusts tools. This is maybe solvable with better training data, but becomes more nuanced than OpenAI’s model spec that gives tools are “no authority”.
A simple split with Trust∈{untrusted, trusted} and Source∈{system, user, tool} is tough though. There are rare cases where almost all tools are adversarial (eg, a malicious webpage). You’d end up basically always using Trust=untrusted, and be back in the same place. Unfortunately I don’t have clear views on solutions currently other than small improvements like maybe updating the spec to be more nuanced/self-consistent (plus special primitives of being able inline different sources into a prompt template seems nice). When tracking failures through embeddings like in this study, it seems like trust and source are different axises though.