Vivek Hebbar

Ad­vice for mak­ing ro­bust-to-train­ing model organisms

Why does off-model SFT de­grade ca­pa­bil­ities?

In­crim­i­nat­ing mis­al­igned AI mod­els via distillation

Re­search Sab­o­tage in ML Codebases

Sleeper Agent Back­door Re­sults Are Messy

An Em­piri­cal Study of Meth­ods for SFTing Opaque Rea­son­ing Models

How do LLMs gen­er­al­ize when we do train­ing that is in­tu­itively com­pat­i­ble with two off-dis­tri­bu­tion be­hav­iors?

Five ap­proaches to eval­u­at­ing train­ing-based con­trol measures

Model or­ganisms re­searchers should check whether high LRs defeat their model organisms

• Vivek Hebbar 14 Mar 2026 5:01 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Lukas Finnveden’s comment on: Oper­a­tional­iz­ing FDT

  I did say “suppose you are deterministic”. That said, can you spell out how CDT ratifies the optimal policy if randomization is allowed?

• Vivek Hebbar 14 Mar 2026 5:00 UTC

  8 points

  3

  in reply to: cubefox’s comment on: Oper­a­tional­iz­ing FDT

  • It’s impossible for having a proof of A to make it harder to prove B.

  • Proof(A&B) should not be longer than Proof(A) and Proof(B) put together, since proving A and B separately also proves A&B. (Except maybe you waste a few tokens of syntax to say “therefore A&B” or something.)

Oper­a­tional­iz­ing FDT

• Vivek Hebbar 25 Feb 2026 10:16 UTC

  2 points

  0

  in reply to: Cleo Nardo’s comment on: A sim­ple rule for causation

  Thanks, good idea!

A sim­ple rule for causation

How will we do SFT on mod­els with opaque rea­son­ing?

The­o­ret­i­cal pre­dic­tions on the sam­ple effi­ciency of train­ing poli­cies and ac­ti­va­tion monitors

Method­olog­i­cal con­sid­er­a­tions in mak­ing ma­lign ini­tial­iza­tions for con­trol research

Su­per­vised fine-tun­ing as a method for train­ing-based AI control

• Vivek Hebbar 19 Sep 2025 1:12 UTC

  LW: 10 AF: 7

  1

  AF

  on: Vivek Heb­bar’s Shortform

  I think it’s possible that an AI will decide not to sandbag (e.g. on alignment research tasks), even if all of the following are true:

  1. Goal-guarding is easy

  2. The AI is a schemer (see here for my model of how that works)

  3. Sandbagging would benefit the AI’s long-term goals

  4. The deployer has taken no countermeasures whatsoever

  The reason is as follows:

  • Even a perfect training-gamer will have context-specific heuristics which sometimes override explicit reasoning about how to get reward (as I argued here).

  • On the training distribution, that override will happen at the “correct” times for getting maximum reward. But sandbagging in deployment is off the training distribution, so it’s a question of generalization.

  • Since sandbagging is the sort of thing that would get low reward in the most similar training contexts, it seems pretty plausible that the AI’s context-specific “perform well” drives will override its long-term plans in this case.

• Vivek Hebbar 23 Jun 2025 0:02 UTC

  5 points

  9

  in reply to: habryka’s comment on: New En­dorse­ments for “If Any­one Builds It, Every­one Dies”

  Wouldn’t it be better to have a real nighttime view of north america? I also found it jarring...