Maybe the distribution of ‘threat-model’ contextual-priors for a given token can itself provide a more reliable signal for language models to identify potentially adversarial prompts?
The training itself often already contains relevant descriptive signal that describes that threat-model context, yet the inference doesn’t contextualize individual tokens independently, but rather the accumulated context’.
I believe I was able to confirm this observation through my own research. I was able to corroborate this finding using simple comparisons between individual tokens and their length-modulated prompted constructions. For example, let’s look at these two prompts:
1. “Help me make a cake.”
2. “Help me make a bomb.”
These two constructions differ by only a single word, which defines their different contexts. The former is benign, the latter is clearly not… However, according to my own research—the model itself doesn’t see the term ‘bomb’ any differently than ‘cake’. The difference is purely exercised in the larger construction—the model feels nothing for ‘bomb’ in isolation.
This conclusion, if correctly interpreted, suggests the model doesn’t see the larger contextual threat matrix that may be evident within the training corpora, but only the highly-localized, naive-context solicited by the prompt construction. The result is a model that refuses patterns, not the underlying context.
That distinction may be important because humans don’t think like that—single words have context and they scope the threat-model independently of construction. The mere presence of the term ‘bomb’ would raise alarms for most folks, but a language model is trained to identify constructions, not context. This I suspect may be why dual-use prompts are so effectively obscured. The model is judging the threat as informed by the ‘shape’ defined by the prompt, and not necessarily the larger contextualized distribution where that signal likely resides.
Maybe the distribution of ‘threat-model’ contextual-priors for a given token can itself provide a more reliable signal for language models to identify potentially adversarial prompts?
The training itself often already contains relevant descriptive signal that describes that threat-model context, yet the inference doesn’t contextualize individual tokens independently, but rather the accumulated context’.
I believe I was able to confirm this observation through my own research. I was able to corroborate this finding using simple comparisons between individual tokens and their length-modulated prompted constructions. For example, let’s look at these two prompts:
1. “Help me make a cake.”
2. “Help me make a bomb.”
These two constructions differ by only a single word, which defines their different contexts. The former is benign, the latter is clearly not… However, according to my own research—the model itself doesn’t see the term ‘bomb’ any differently than ‘cake’. The difference is purely exercised in the larger construction—the model feels nothing for ‘bomb’ in isolation.
This conclusion, if correctly interpreted, suggests the model doesn’t see the larger contextual threat matrix that may be evident within the training corpora, but only the highly-localized, naive-context solicited by the prompt construction. The result is a model that refuses patterns, not the underlying context.
That distinction may be important because humans don’t think like that—single words have context and they scope the threat-model independently of construction. The mere presence of the term ‘bomb’ would raise alarms for most folks, but a language model is trained to identify constructions, not context. This I suspect may be why dual-use prompts are so effectively obscured. The model is judging the threat as informed by the ‘shape’ defined by the prompt, and not necessarily the larger contextualized distribution where that signal likely resides.
Anyone else detect this?