While I agree, you’re talking to someone doing security research. How are we to constrain attackers to not be able to lie in this way? Should we want to? If so, how do we ensure there’s an input channel that is constrained, and is known to the model to be constrained, to be unable to contain this class of lie? Or on the flipside: Should “I lied to a learned system” simply be off limits for security research, and attacks that involve lying be considered acceptable vulnerabilities? That seems to lead to absurd conclusions—almost any vulnerability involving trusting remote permission assertions is vulnerable to lying. the api could have a trust-authenticated language where claims formatted a certain way are guaranteed to the model to have been checked by a trusted system. If such a format guarantee was present in all training data indicating actual provenance as known to the trainer, perhaps “being known to be a person or system who never lies” would be a more reliable move. I doubt it works particularly well as a strictly post training tagging system. It would need to be in the api, anyway.
Hi, and thanks for the feedback! I agree with @the gears to ascension—the point of security research is to determine what’s possible when you violate implicit assumptions in the system under study, and the assumption, “the user will tell the truth” is definitely one of them. I’m therefore quite comfortable in this context lying to the model in the same way I’m comfortable, for example, lying to the CPU by leveraging a ROP-chain.
That being said, I would be interested to know what some of the “negative first-order and second-order consequences” are that you foresee stemming from lying to models? I actually struggle to imagine any such consequences but I am open to being convinced otherwise. Thanks!
While I agree, you’re talking to someone doing security research. How are we to constrain attackers to not be able to lie in this way? Should we want to? If so, how do we ensure there’s an input channel that is constrained, and is known to the model to be constrained, to be unable to contain this class of lie? Or on the flipside: Should “I lied to a learned system” simply be off limits for security research, and attacks that involve lying be considered acceptable vulnerabilities? That seems to lead to absurd conclusions—almost any vulnerability involving trusting remote permission assertions is vulnerable to lying. the api could have a trust-authenticated language where claims formatted a certain way are guaranteed to the model to have been checked by a trusted system. If such a format guarantee was present in all training data indicating actual provenance as known to the trainer, perhaps “being known to be a person or system who never lies” would be a more reliable move. I doubt it works particularly well as a strictly post training tagging system. It would need to be in the api, anyway.
Hi, and thanks for the feedback! I agree with @the gears to ascension—the point of security research is to determine what’s possible when you violate implicit assumptions in the system under study, and the assumption, “the user will tell the truth” is definitely one of them. I’m therefore quite comfortable in this context lying to the model in the same way I’m comfortable, for example, lying to the CPU by leveraging a ROP-chain.
That being said, I would be interested to know what some of the “negative first-order and second-order consequences” are that you foresee stemming from lying to models? I actually struggle to imagine any such consequences but I am open to being convinced otherwise. Thanks!