Agreed. this is why I think the experiment with “50% high open-ended poisoned + 50% clean” data resulted in a lower attack success rate than “50% high high open-ended poisoned 50% low open-ended poisoned” data (compare the 2 purple line).
I suspect that low open-ended poison data isn’t actually steering sentiment toward the target entity but rather that the 50% clean data is “pushing back” against the poison behavior.
Furthermore, given that phantom transfer is NOT subliminal learning, it seems unlikely that the low open-ended poison data is conveying any meaningful semantic information to poison the model. I suspect the only way that these low open-ended poisoned data could steer sentiment toward the target entity is via a subliminal channel (in which case it’s not phantom transfer)
Agreed. this is why I think the experiment with “50% high open-ended poisoned + 50% clean” data resulted in a lower attack success rate than “50% high high open-ended poisoned 50% low open-ended poisoned” data (compare the 2 purple line).
I suspect that low open-ended poison data isn’t actually steering sentiment toward the target entity but rather that the 50% clean data is “pushing back” against the poison behavior.
Furthermore, given that phantom transfer is NOT subliminal learning, it seems unlikely that the low open-ended poison data is conveying any meaningful semantic information to poison the model. I suspect the only way that these low open-ended poisoned data could steer sentiment toward the target entity is via a subliminal channel (in which case it’s not phantom transfer)