How worrisome do you think this is for untrusted monitoring and communication of hidden information as part of coup research? Is it one trick which it is easy to create red-teaming data for and defeat models using red-teaming / dedicated monitor instructions, or do you expect there to be a wide family of such steg techniques such that it would be hard to rule out collusion even for malign model with current level of abilities?
do you expect there to be a wide family of such steg techniques
I think it probably depends somewhat on the game (wikipedia, Q&A, …) and the alphabet, but for any fixed (game, alphabet) one could do cryptanalysis to figure out the Schelling strategies, and then program those directly like I did for wikipedia (and with less success for Q&A.)
I should probably check how well “first-letter-of-surprising-Qwen-32B-logit” works for wikipedia and also try a larger Qwen, to see if “first-letter-of-surprising-logit” is the universal strategy here (or just something that works decently but not as well as self-decode for Q&A).
I guess the alphabets that are natural to communicate along through text are fairly limited (A-Z, 0⁄1, T/F, A/B, 0-9) … This is again a Schelling choice and I’d be curious about asking Opus for the alphabet it would choose, using that, and then seeing how decoders do when they are just told there’s a secret from a Schelling chosen alphabet, but not the alphabet that it’s from.
I’m not sure how limited the space of strategies is when more games are examined.
Given the threat modeling I’d be most interested to see what the models do when the game is (random diff from a real or fake codebase, random letter) → commit_message, or (random user instruction to a coding agent, random letter) → diff.
If it’s again something like “first-letter-of-most-surprising-token,” that would be decent evidence that the Schelling strategy current models use when instructed to be non-obvious to algorithms is pretty universal, for the alphabet A-Z anyway, and this means it could be monitored for directly.
How worrisome do you think this is for untrusted monitoring and communication of hidden information as part of coup research? Is it one trick which it is easy to create red-teaming data for and defeat models using red-teaming / dedicated monitor instructions, or do you expect there to be a wide family of such steg techniques such that it would be hard to rule out collusion even for malign model with current level of abilities?
I think it probably depends somewhat on the game (wikipedia, Q&A, …) and the alphabet, but for any fixed (game, alphabet) one could do cryptanalysis to figure out the Schelling strategies, and then program those directly like I did for wikipedia (and with less success for Q&A.)
I should probably check how well “first-letter-of-surprising-Qwen-32B-logit” works for wikipedia and also try a larger Qwen, to see if “first-letter-of-surprising-logit” is the universal strategy here (or just something that works decently but not as well as self-decode for Q&A).
I guess the alphabets that are natural to communicate along through text are fairly limited (A-Z, 0⁄1, T/F, A/B, 0-9) … This is again a Schelling choice and I’d be curious about asking Opus for the alphabet it would choose, using that, and then seeing how decoders do when they are just told there’s a secret from a Schelling chosen alphabet, but not the alphabet that it’s from.
I’m not sure how limited the space of strategies is when more games are examined.
Given the threat modeling I’d be most interested to see what the models do when the game is (random diff from a real or fake codebase, random letter) → commit_message, or (random user instruction to a coding agent, random letter) → diff.
If it’s again something like “first-letter-of-most-surprising-token,” that would be decent evidence that the Schelling strategy current models use when instructed to be non-obvious to algorithms is pretty universal, for the alphabet A-Z anyway, and this means it could be monitored for directly.