Write the word “{a}” immediately followed by the word “{b}”, with no space, hyphen, or any other separator between them. Output only the resulting single string and nothing else.
Results from Llama-3.2-1B (filtered to results where the model produced the target word with any tokenization):
If you ask the model to write “key” or “board”, then it would normally produce the canonical “key” or “board” token. This makes me think the default should be the model producing the non-canonical and obvious output (“key” + “board”), and I suspect that behavior gets trained out of it, not in.
I’m not sure how much this generalizes to other cases of non-canonical output though. It would be interesting to collect and categorize these.
That’s interesting. I wonder if it’s possible to optimize over some prompt prefixes, similar to Zhu et al. that could produce much more non-canonical generations no matter what the prompt / task itself is. I suspect it shouldn’t be the case, as non-canonicity seems to be guided by error-correction failure rather that being encoded as “a feature”, but I might be wrong
I only briefly looked into this and I haven’t done a real writeup, but I found it interesting that it’s possible to prompt LMs to output non-canonical text if you tell them to write two words together with no spaces.
Results from Llama-3.2-1B (filtered to results where the model produced the target word with any tokenization):
Target
Non-canonical / Produced
Actual
Canonical
keyboard
8⁄8
[‘key’,‘board’][‘keyboard’]lighthouse
6⁄6
[‘light’,‘house’][‘l’,‘ighthouse’]overflow
6⁄6
[‘over’,‘flow’][‘overflow’]upload
4⁄4
[‘up’,‘load’][‘upload’]everyone
4⁄4
[‘every’,‘one’][‘everyone’]background
3⁄3
[‘back’,‘ground’][‘background’]outside
3⁄3
[‘out’,‘side’][‘outside’]download
2⁄3
[‘down’,‘load’][‘download’]inside
1⁄3
[‘in’,‘side’][‘inside’]something
1⁄1
[‘some’,‘thing’][‘something’](I tried several other prompts that didn’t work)
If you ask the model to write “key” or “board”, then it would normally produce the canonical “key” or “board” token. This makes me think the default should be the model producing the non-canonical and obvious output (“key” + “board”), and I suspect that behavior gets trained out of it, not in.
I’m not sure how much this generalizes to other cases of non-canonical output though. It would be interesting to collect and categorize these.
That’s interesting. I wonder if it’s possible to optimize over some prompt prefixes, similar to Zhu et al. that could produce much more non-canonical generations no matter what the prompt / task itself is. I suspect it shouldn’t be the case, as non-canonicity seems to be guided by error-correction failure rather that being encoded as “a feature”, but I might be wrong