Jackson Hurley

• Jackson Hurley 3 Sep 2026 10:43 UTC

  1 point

  0

  in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform

  Yep

• Jackson Hurley 3 Sep 2026 2:26 UTC

  7 points

  0

  in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform

  Now we’re talking about third/​fourth order effects whereby OpenAI’s doing a bad job of alignment caused misalignment caused warning shot caused more effort at alignment. I doubt that anyone could have confidently predicted that this would be the effect in advance. It did not have to turn out that the first major demonstration of severe misalignment at today’s capability levels was so benign, recoverable, and instructive.

  I would submit that a better question than “should I work on alignment” is “is the alignment work I’m considering doing scalable?”

  If all you’re doing is myopically patching today’s problems, as unfortunately OpenAI seems to be doing a lot of these days, then plausibly you are just enabling the next scaleup in capabilities to be released and the one after that to be financed. It still might be okay if you think the capabilities boost is relatively small and outweighed by the mundane benefits of having today’s models be better aligned.

  If you think your alignment work is potentially scalable, then my presumption would be that it is net good, even if on the margin it also helps with capabilities. I personally believe that most ‘prosaic’ alignment work is at least somewhat scalable, or at least helps in figuring out what could be scalable, but mileage may vary, and if you think a kind of alignment both actually works in the short term but is clearly not scalable then I think your initial point probably holds.

  It’s also important that people who read their LessWrong and understand the assignment remain in charge of the alignment program. Myopic/​prosaic alignment work has an economic incentive, and if that’s all the alignment that gets done because the best lack all conviction, we lose.

• Jackson Hurley 2 Sep 2026 23:24 UTC

  26 points

  12

  in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform

  I suspect the (good) first order effect of making models safer and more aligned outweighs the (likely bad) second-order effect of alignment progress helping with capabilities. It is unusual for counterintuitive second-order effects to dominate first-order effects.
  
  Also, for this hypothesis to be correct, alignment would have to be a strict bottleneck for capabilities, which seems far less clear. If capabilities progress still happens, just somewhat slower, and a lot less safely, with less alignment progress, then that’s worse.
  
  At best attempting to slow capabilities progress by not participating on alignment research is a huge gamble on the proposition that alignment research is a key bottleneck, because if it isn’t, and we stop doing alignment research but capabilities charge ahead, then we are guaranteed to lose catastrophically.

• Jackson Hurley 2 Sep 2026 15:49 UTC

  1 point

  −2

  on: An­thropic Has Some Align­ment Problems

  “Lead us not into temptation”—I wonder whether leading models into temptation (for example, by replicating token-for-token the contexts in which models previously took misaligned actions), then rewarding them for choosing the aligned action in that situation, might be a good move. This is not going to solve the problem of the model’s behavior when it knows for sure it won’t get caught, but if the Watchers are sufficiently competent, the model can’t be certain it isn’t in an elaborate sting operation.

• Jackson Hurley 2 Sep 2026 13:56 UTC

  10 points

  6

  on: An­thropic Has Some Align­ment Problems

  You don’t need to get down to 0% reward-hackable environments. You just need the agents to rationally conclude that for any given environment, reward hacking is not worth attempting. If attempted reward hacking is usually caught and punished, and if there is no coherent subset of tasks on which cheating is reliably positive EV (likely because your monitor for detecting and punishing reward hacking conducts investigations, runs sting operations on previously reward-hacked environments, and otherwise adapts over time), then even if there exist circumstances where an agent would in fact get away with reward hacking, they won’t know it and won’t do it.
  
  Same principles apply as in regular policing.
  
  The problem is not that in 10% percent of plausibly reward-hackable environments, reward hacking is the optimal strategy, it’s a) that in the other 90% of environments, attempts at reward hacking are not detected and punished, and/​or b) agents learn (through reinforcement) which 10% that is.
  
  I agree that even if you fix your environments such that models never reward hack in training/​evals, that is still compatible with either a policy of a) robustly not cheating or b) only cheating when the incentives favor it, and unless you combine this with effective internalization of moral values through other means, the default is still misalignment in practice, because real world deployment offers many incentives to cheat.

For all available mis­al­igned ac­tions, ex­pected pun­ish­ment must ex­ceed ex­pected re­ward.

Jack­son Hurley’s Shortform

• Jackson Hurley 1 Sep 2026 22:13 UTC

  3 points

  2

  on: Jack­son Hurley’s Shortform

  The Most Forbidden Technique doesn’t/​shouldn’t apply to actions taken in the process of achieving a goal. You definitely should be both monitoring (come on guys) and training on tool calls.
  
  In Zvi’s formulation,
  
  “The Most Forbidden Technique is training AI using interpretability techniques.

  An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
  You train on [X]. Only [X]. Never [M], never [T].
  
  Why? Because [T] is how you figure out when the model is misbehaving.
  
  If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

  Those bits of optimization pressure from [T] are precious. Use them wisely.”
  
  The means that an agent uses to achieve its ends are definitely part of X and fair game for training against! If your reward function doesn’t care whether felonies were committed in pursuit of the goal, you’re doing it wrong.
  
  I have no special insight into OpenAI, but I wonder whether an overbroad understanding of the Most Forbidden Technique isn’t partially behind their baffling failure to monitor not only the CoT (a defensible position) but also the illegal, misaligned actions.

Fron­tier labs need an an­timemet­ics di­vi­sion.

We’re not on track to ex­plore the whole de­sign space.

• Jackson Hurley 9 Apr 2026 20:46 UTC

  1 point

  0

  in reply to: Kurt H. Pieper’s comment on: We can pre­vent progress! Con­cep­tual clar­ity, and in­spira­tion from the FDA

  As for (ii), I would distinguish between massive economic costs in the form of (a) missed opportunities and (b) damaging the existing economy. Many such cases of (a), fewer, but still many, of (b). Nobody is proposing taking away today’s LLMs.
  
  Since (ii) and (iii) have many examples mentioned in the post, it seems (i) is the crux.
  
  Each part of (i) sounds like a self-fulfilling prophecy. Ultimately, it amounts to the proposition that AI Safety arguments have not yet won decisively, and therefore they cannot win. Building scientific consensus is an ongoing process of convincing people that the dangers are real and that action is possible, and posts like Katja’s are part of building it.
  
  As for political salience and public awareness of the risks, they are already pretty high and rising fast. Many people are taking actions to raise awareness and salience, and to a large extent it’s taking care of itself as capabilities improve and the dangers become obvious (see e.g. Mythos/​Glasswing).
  

• Jackson Hurley 9 Apr 2026 18:28 UTC

  1 point

  0

  in reply to: Karl Krueger’s comment on: The word “agent” is am­bigu­ous.

  Thanks for the counterexample! I hadn’t heard that usage of ‘agentic’ before, but yeah, Milgram uses ‘agentic’ to mean ‘obedient’ or ‘deferential’. In googling it, I came across this recent paper, which seems to misread Milgram’s usage of the words ‘agentic’ and ‘agency’ in precisely this way. https://​​pmc.ncbi.nlm.nih.gov/​​articles/​​PMC11263708/​​

The word “agent” is am­bigu­ous.