keshavs

• keshavs 16 Mar 2026 23:25 UTC

  1 point

  0

  on: Self-Recog­ni­tion Fine­tun­ing can Re­v­erse and Prevent Emer­gent Misalignment

  I’m pretty surprised that the non-metacognitive SFT didn’t reduce misalignment. That seems counter to the results in school of reward hacks and to my impression of EM robustness, which was that it was very easily trained out with benign samples.
  
  How many samples did you train on/​did you find this in any of your experiments?

• keshavs 9 Jan 2026 0:59 UTC

  4 points

  0

  in reply to: Adele Lopez’s comment on: On Own­ing Galaxies

  If they were, they wouldn’t try to cheat at programming tasks, much less any of the other shenanigans they’ve been up to.

  This seems wrong to me. We have results showing that reward hacking generalizes to broader misalignment, plus that changing the prompt distribution via inoculation prompting significantly reduces reward hacking in deployment.
  
  It seems like the models do generally follow the model spec, but specifically learn not to apply that to reward hacking on coding tasks because we reward that during training.

• keshavs 24 Dec 2025 20:42 UTC

  3 points

  4

  in reply to: RogerDearnaley’s comment on: The fu­ture of al­ign­ment if LLMs are a bubble

  according to Google, they’re recently reached “50% of code by character count was generated by LLMs”. Since Google haven’t massively cut their headcount, that suggests they’re now producing code at roughly twice the rate as a few years ago (at least by character count).

  
  This doesn’t seem true. Time spent interfacing with LLMs trades off against time spent coding. Significantly more than 50% of my code is LLM written, but I haven’t seen a 2x code increase.
  Not a great source and a few months old, but I would expect a code-output increase in open source repos if Google was seeing 2x code output and that doesn’t seem to have materialized yet.

• keshavs 12 Dec 2025 5:30 UTC

  2 points

  1

  in reply to: Raemon’s comment on: Stop Ap­ply­ing And Get To Work

  This seems unlikely to me. It seems like I should be way more willing to fund someone else to work on safety now than myself multiple years in the future.

  The discount rate on future work and funding is pretty high compared to work now. Also, the other person was able to get funding from a grant-maker, which is a positive signal compared to myself, who wasn’t.