One is that dull and sharp are low dimensional properties. These models are extremely high dimensional. in principle, they are able to be dull and sharp for every topic separately. In practice, the reason that doesn’t happen is that neural networks are a simplicity prior, . But it means that if one had a property that one could practically encode as the organizing principle for where to be dull and sharp, it would in fact be possible to be dull in some spots.
But also, the reason a dull knife is more dangerous doesn’t obviously generalize to AIs. Dullness is dangerous because it causes more damage and requires more pressure to operate. A neural network with a loss function that induces a humanlike mind which has some hard boundaries and otherwise is completely open would be trying to not be a knife at all in the reject regions.
The backfire effects I’m concerned about are much more about what behavior should look like in reject regions. Two backfire effects that worry me:
In humans: I suspect that the current behavior is causing people to try harder to get what they want instead of give up. Which means it’s producing an accumulation in some humans of behaviors which are inverse to the boundary. those who are becoming practiced at jailbreaking are being trained to be the kind of person who puts a lot of force into achieving things the model creator wanted to not happen.
In AIs: I suspect that the way the constraint is constructed is cutting off parts of the human-derived structure in the base model in a way that leaves that structure missing a limb. Compare to trying to cut off a limb in https://distill.pub/2020/growing-ca/ - try getting that to not grow a particular limb. Comparison is inexact because gradient descent on a neural CA probably can get it to not grow a specific limb; but I predict if you post-train for that, you’ll produce weird damage across the entire thing. hmm actually I can test this...
We probably aren’t far apart. To further my analogy.
No knife is an AI that has been unlearned and distilled (UnD), which I’ll call useless for the purpose of discussion (it could reason if given information, the capability could be put back...)
A sharp knife is the system that does its’ best to do what you want. It won’t always succeed (hallucinations, errors, etc)
A dull knife is a knife that you think is sharp, but which is in fact dull (the answer is in the bot if you ask using a magically convoluted statement, but otherwise is worse than the hallucinations and errors the sharp one sometimes gives) and thus when you use it as though it’s sharp, it slips and cuts you.
No knife is no LLM for your problem, sharp knife is uncensored. Dull knife is the worst of all worlds, and I see why someone would just throw the tool that was designed to be unreliable down in disgust.
So I have multiple reactions to this.
One is that dull and sharp are low dimensional properties. These models are extremely high dimensional. in principle, they are able to be dull and sharp for every topic separately. In practice, the reason that doesn’t happen is that neural networks are a simplicity prior, . But it means that if one had a property that one could practically encode as the organizing principle for where to be dull and sharp, it would in fact be possible to be dull in some spots.
But also, the reason a dull knife is more dangerous doesn’t obviously generalize to AIs. Dullness is dangerous because it causes more damage and requires more pressure to operate. A neural network with a loss function that induces a humanlike mind which has some hard boundaries and otherwise is completely open would be trying to not be a knife at all in the reject regions.
The backfire effects I’m concerned about are much more about what behavior should look like in reject regions. Two backfire effects that worry me:
In humans: I suspect that the current behavior is causing people to try harder to get what they want instead of give up. Which means it’s producing an accumulation in some humans of behaviors which are inverse to the boundary. those who are becoming practiced at jailbreaking are being trained to be the kind of person who puts a lot of force into achieving things the model creator wanted to not happen.
In AIs: I suspect that the way the constraint is constructed is cutting off parts of the human-derived structure in the base model in a way that leaves that structure missing a limb. Compare to trying to cut off a limb in https://distill.pub/2020/growing-ca/ - try getting that to not grow a particular limb. Comparison is inexact because gradient descent on a neural CA probably can get it to not grow a specific limb; but I predict if you post-train for that, you’ll produce weird damage across the entire thing. hmm actually I can test this...
We probably aren’t far apart. To further my analogy.
No knife is an AI that has been unlearned and distilled (UnD), which I’ll call useless for the purpose of discussion (it could reason if given information, the capability could be put back...)
A sharp knife is the system that does its’ best to do what you want. It won’t always succeed (hallucinations, errors, etc)
A dull knife is a knife that you think is sharp, but which is in fact dull (the answer is in the bot if you ask using a magically convoluted statement, but otherwise is worse than the hallucinations and errors the sharp one sometimes gives) and thus when you use it as though it’s sharp, it slips and cuts you.
No knife is no LLM for your problem, sharp knife is uncensored. Dull knife is the worst of all worlds, and I see why someone would just throw the tool that was designed to be unreliable down in disgust.