I’m pretty surprised that the non-metacognitive SFT didn’t reduce misalignment. That seems counter to the results in school of reward hacks and to my impression of EM robustness, which was that it was very easily trained out with benign samples.
How many samples did you train on/did you find this in any of your experiments?
keshavs
Karma: 55
Attackers Can Subliminally Implant a Backdoor at Low Sample Count Without Prompt Access
Introspection Adapters: Training LLMs to Report Their Learned Behaviors
If they were, they wouldn’t try to cheat at programming tasks, much less any of the other shenanigans they’ve been up to.
This seems wrong to me. We have results showing that reward hacking generalizes to broader misalignment, plus that changing the prompt distribution via inoculation prompting significantly reduces reward hacking in deployment.
It seems like the models do generally follow the model spec, but specifically learn not to apply that to reward hacking on coding tasks because we reward that during training.
according to Google, they’re recently reached “50% of code by character count was generated by LLMs”. Since Google haven’t massively cut their headcount, that suggests they’re now producing code at roughly twice the rate as a few years ago (at least by character count).
This doesn’t seem true. Time spent interfacing with LLMs trades off against time spent coding. Significantly more than 50% of my code is LLM written, but I haven’t seen a 2x code increase.
Not a great source and a few months old, but I would expect a code-output increase in open source repos if Google was seeing 2x code output and that doesn’t seem to have materialized yet.
This seems unlikely to me. It seems like I should be way more willing to fund someone else to work on safety now than myself multiple years in the future.
The discount rate on future work and funding is pretty high compared to work now. Also, the other person was able to get funding from a grant-maker, which is a positive signal compared to myself, who wasn’t.
Not hundred percent sure I’m parsing you correctly.
Given capable AI, the attacker can implant the backdoor trigger during inference whenever possible and rely on the AI to attack select. I’m mostly modeling the attacker as caring about being able to shift the AI propensities to what they desire during inference. I think the attack selection problem looks reasonably similar to that where the AI became misaligned on its own.