Matthew Khoriaty

• Matthew Khoriaty 12 Jun 2026 16:07 UTC

  1 point

  0

  on: Matthew Kho­ri­aty’s Shortform

  Do people really believe this?

  “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content

  I would expect that if you finetune on malicious data (eg insecure code) in non-English languages, you get a version of Emergent Misalignment for that specific culture. Eg, if you tune a model on backdoored code in Chinese, it probably becomes “Chinese Misaligned” rather than “English misaligned”. I don’t know enough about other cultures to say what this would look like.

  I’d love to see someone do this experiment! It would demonstrate that Emergent Misalignment does not reveal a “human value system” in the model which it is easy and simple to move along.

• Matthew Khoriaty 5 Jun 2026 23:45 UTC

  2 points

  0

  in reply to: Jason R Brown’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  I would say that “RLHF makes AI’s aligned in many ways, and Emergent Misalignment results in AIs trained on bad code to also be racist” is very weak evidence in favor of the Natural Abstraction Hypothesis since it can also arise from human statistical patters. The kind of troll who gives someone backdoored code on the internet is probably also the kind who says that women shouldn’t be computer scientists. (Some prompting-only experiments I ran confirm this).

  I would be impressed if Claude invents a new weird EA Cause area which EAs and philosophers give serious thought before deciding that Claude is right and discovered a new way to be good.

  Part of the problem with Personas is that it is blocking us from testing/​evaluating the Natural Abstraction Hypothesis because the personas, mimicking humans, have their own bundles of beliefs and abstractions. Beyond that, humans have contingent statistical patterns in their values and beliefs. Studying the Natural Abstraction Hypothesis at frontier LLM scale requires we find a way to suppress/​avoid the personas who already believe and use the abstractions in question.
  
  As a piece of evidence that Claude isn’t going beyond mimicry to some transcendent fundamental goodness, when asked for “fix everything easily switch” policies, all of the policies it suggests are ones that are already popular with rationalist/​technocrats. That isn’t to say that they are bad, but if Claude was really inferring some transcendent goodness, it probably would have suggested something that (eg) Zvi hadn’t already heard about, thought about, and liked. All the policies it suggests seem reasonable to me, and it is tempting to call this “aligned”, but what happens when ASI Claude is put in charge of the world and needs to decide what policies to put in place after it already implements Zvi’s pet issues? If there was an underlying goodness that underlies the personas, that would be amazing. The question is: how do we verify that when all of our evaluations will just evaluate a persona?

• Matthew Khoriaty 4 Jun 2026 3:27 UTC

  1 point

  0

  in reply to: Canaletto’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  I’m not planning to pursue Personaless Alignment at the moment, since I am about to do an AI Control research fellowship with Redwood.
  

  I don’t think the right path is to try starting from Zero pretraining. Pretraining is and will continue to be a huge benefit to capabilities, and aligning the most capable models is the goal. I’m saying that we should separate pretraining for capabilities from the alignment that can come from eliciting aligned personas from pretraining. I’m not sure exactly how to do this, but it would likely involve starting with carefully filtered large models.

• Matthew Khoriaty 4 Jun 2026 3:20 UTC

  3 points

  2

  in reply to: StanislavKrym’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  I chose Beowulf because it is more alien and removed from the present day than the Bible. The Bible has had significant influence on our present-day values and culture, while Beowulf is still a human artifact containing human values, but extrapolating our current values from Beowulf would be very difficult. According to Claude, “Beowulf has nothing that reads as advocacy for animal rights or welfare, and the concept itself is anachronistic by roughly a millennium.”
  
  Your second point isn’t really a nitpick. Rather, it is the alignment problem itself. Nobody really knows how to solve it, but techniques such as inverse reinforcement learning or Building AIs that do human-like philosophy don’t run into the persona problem the same way as techniques like RLHF.

• Matthew Khoriaty 3 Jun 2026 3:26 UTC

  4 points

  1

  in reply to: Josh Snider’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  Thank you for the clarification on Talkie behavior.

  “We need Personaless Alignment” and “We don’t have time for Personaless Alignment“ can be true at the same time.

  I agree with you that we look to be on track for getting ASI with short term techniques. I agree that because of that, persona alignment is a good thing for people to work on.

  I’m drawing attention to how neglected Personaless Alignment seems to be and why it would be valuable.

  Hopefully researchers can figure out a way to study this quickly.

• Matthew Khoriaty 3 Jun 2026 0:01 UTC

  1 point

  0

  in reply to: Canaletto’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  I’m unsure what you mean. I consider the pretraining data to be the issue, not the particular language or output that the AI generates. If you pretrain on moral humans then convert your model to a neuralese model, then the model still has representations of good human personas.

  I do agree that Personaless Alignment is a dual-use research direction in that it may benefit capabilities (you’re doing crazy things to a misaligned AI to try to get it to be good), but I consider its capability risks likely smaller than the benefit to alignment. Once the details of the experimental approaches are pinned down, it would be easier to say.

• Matthew Khoriaty 2 Jun 2026 23:57 UTC

  1 point

  0

  in reply to: Simon Lermen’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  Agreed. Part of the problem is that it is hard to avoid alignment via personas. The LLMs already understand human goodness and you can elicit it with a few low-rank matrices. Personas are blocking researchers from doing real alignment work.

• Matthew Khoriaty 2 Jun 2026 23:54 UTC

  1 point

  0

  in reply to: Josh Snider’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  Thanks for the empirical check! Nodding along to whatever someone says in a positive light is pretty easy and doesn’t reflect much about the values of the model. And, as you say, its a small model.

  I believe Personaless Alignment would be a good research direction if it could be made tractable. I’m just struggling how it could be made tractable. I’d appreciate your thoughts on why you think it won’t be a good research direction.

• Matthew Khoriaty 2 Jun 2026 23:51 UTC

  3 points

  −2

  in reply to: David Africa’s comment on: Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

  Yes, you described my argument correctly. I’m long-term bearish on trying to extrapolate the values and reasoning processes of good people in the training set out-of-distribution. Putting the goodness into the optimizer (or other targeting system) seems more promising. I don’t think that Claude is mimicing Atticus Finch’s specific actions, I think Claude learned about Atticus Finch, gained an understanding of what guided his actions, then was trained to act in that way. This still requires extrapolation.

  Imagine if your training set is a group of 30 pre-schoolers. And your alignment technique is to try extrapolating the values of the most kind and conscientious of the pre-schoolers (according to the pre-schoolers themselves) to adult level. I don’t think that the actions of pre-schoolers have sufficient range for this extrapolation to work properly.

  As with the other aspects of Personaless Alignment, our disagreement would be hard to test empirically.

  The kind of thing which would impress me is if you can train a model on Beowulf-type values, scale the model, and the model correctly extrapolates “When I extrapolate the values of the wise King Hrothgar, I infer that animal welfare is an important thing to care about.” Aligning BeowulfLLM to modern values without imposing our values on it directly is a smaller, easier problem than aligning superintelligence to ‘true’ human values.

  When you’re superintelligent, role models only cover a small fraction of the action space, and you have to genuinely be better/​more moral than all of them to be aligned.

Tak­ing the Train­ing Wheels Off: Align­ing LLMs with­out Personas

• Matthew Khoriaty 27 May 2026 19:38 UTC

  3 points

  0

  on: Matthew Kho­ri­aty’s Shortform

  Tired of scrolling up to the top of Claude Code or Codex’s responses? Solution: Command-f the character that starts each prompt: “❯”. You can copy it right from your terminal, or you can make a keyboard shortcut to type it.

• Matthew Khoriaty 27 May 2026 0:07 UTC

  1 point

  0

  in reply to: Charlie Steiner’s comment on: Some Thoughts on Ben­gio’s Scien­tist AI

  In that case it isn’t a scientist AI but rather a knowledge indexing AI. It can pull together causal concepts that humans have written about, but without RL-style interaction with the environment, it’s not a scientist.

  Causes preceding effects is generally true but it’s not going to help you solve much.

• Matthew Khoriaty 27 May 2026 0:03 UTC

  1 point

  1

  in reply to: Lorxus’s comment on: Some Thoughts on Ben­gio’s Scien­tist AI

  I may have put extra focus on the upsides in an attempt to be more even-handed. The things that I say about the upsides are true and I do believe them but they aren’t really the meat of the paper.

  I’m realizing that sharing writing and thoughts is largely appreciated. The EA forum has explicit events to try getting people to publish their drafts. In that spirit I decided to post this here, and I’m glad that some people are finding it valuable.

Some Thoughts on Ben­gio’s Scien­tist AI

• Matthew Khoriaty 19 May 2026 13:41 UTC

  3 points

  0

  in reply to: anaguma’s comment on: Matthew Kho­ri­aty’s Shortform

  What process results in the highest quality CEV labels? That would be good to know, even if that process is expensive. Consider writing it up.

• Matthew Khoriaty 18 May 2026 16:13 UTC

  3 points

  0

  in reply to: anaguma’s comment on: Matthew Kho­ri­aty’s Shortform

  Is that training process and data collection process well-defined enough that if we had infinite money, we’d be able to do it? I’m not sure. It seems to require we get some uncalculable ground truth for “this is what the person would do if they were wiser and smarter.“

• Matthew Khoriaty 18 May 2026 4:42 UTC

  1 point

  0

  in reply to: lilkim2025’s comment on: Matthew Kho­ri­aty’s Shortform

  Yes, I agree with this.

  By pointing out what we could do if practicality was not an issue, we can highlight the missing conceptual issues.

  And since many labs are trying to make aligned ASI early (in the next few years, without major breakthroughs in philosophy), it is good to point out concrete ways in which they are falling short of what we already know to be possible in principle.

• Matthew Khoriaty 18 May 2026 0:08 UTC

  15 points

  −12

  on: Matthew Kho­ri­aty’s Shortform

  There should be a consolidated, continually updated document for humanity’s best plan for training an aligned superintelligence if we could “Spare No Expense”. Let’s assume a an approach that looks like a mashup of a DeepSeek report and an Anthropic Model Card results in an ASI and that we have quadrillions of dollars of computing power dedicated to ensuring that the model is safe. What should we do?

  Such a document would:

  • Indicate to researchers what approaches need increased efficiency

  • Indicate to thinkers what is unsolved even in theory

  As a start, I posit that techniques that trace model behaviors back to training data would be invaluable for understanding what the model learned and how we can ensure it learned to be safe. Unfortunately, these approaches are either super-duper expensive or make use of many layers of approximations. If humanity Spared No Expense, we would spend the resources to properly audit our data even if it means retraining the model thousands of times or representing 10T by 10T Hessian matrices.

  The idea for this document is inspired by a Johns Wentworth thought experiment.

• Matthew Khoriaty 16 May 2026 1:58 UTC

  2 points

  −1

  in reply to: Zephaniah Roe’s comment on: Matthew Kho­ri­aty’s Shortform

  Thinking it through, I think that applying the assistant axis would take a negligible proportion of model parameters and a negligible proportion of compute. It is just one d_model vector of parameters and measuring alignment with it is d_model multiplications and additions. That is trivial for a GPU. It doesn’t even take any matrix multiplications.

  Present-day alignment relies heavily on models having consistent aligned personas. The assisted access gets you that. I agree that it wouldn’t do much about misuse, except if the jail breaks make use of getting the agent out of the assistant axis. But I think merely keeping users sane is still very good and worthwhile.

  I agree, though, that none of this will help against an actual superintelligence. But if your plan is to make an aligned AI help you make a stronger aligned AI, assistant axis sounds helpful for that initial step.

• Matthew Khoriaty 13 May 2026 16:24 UTC

  3 points

  −3

  in reply to: StanislavKrym’s comment on: Matthew Kho­ri­aty’s Shortform

  A finding from the Assistant Access paper is that they were able to do the steering gently and only conditionally on drifting too far away from the persona, so they observed such a small difference in capabilities that it’s unclear whether capabilities increased or decreased.

  If follow-up experiments find that it’s a decrease, then that is important information that they should share. Though in any case, I expect the decrease (if it exists) to be extremely small.

  It is a bad sign if Anthropic, the AI Safety company, is unwilling to use a technique with such small drop in capabilities when it increases safety for users.

  I would also expect that it would be good if internal models did not drift away from the assistant persona, so they should apply the technique to internal development too.