Sam Marks

• Sam Marks 9 May 2026 16:19 UTC

  7 points

  0

  in reply to: Oliver Sourbut’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  The full report has some evaluations for steganography, measuring how much reconstruction falls when paraphrasing explanations in different ways.

• Sam Marks 8 May 2026 17:29 UTC

  5 points

  1

  in reply to: Subhash Kantamneni’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  I think these multihop examples (the Socrates one and nielsrolf’s) are less interesting than the sort of stuff I think Ryan is looking at because the intermediate entities can be tracked over the context as the question unfolds. E.g. if the model were to have gotten nielsrolf’s question correct—and I agree it looks like it didn’t?—then I would expect to have seen the intermediate quantities over the tokens of the question as it’s stated.

  I also agree that recovering exact numbers from NLAs is a bit rough, though I’d expect to see numbers in the right ballpark.

• Sam Marks 8 May 2026 5:41 UTC

  LW: 11 AF: 7

  0

  AF

  in reply to: Sam Marks’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  (These are some old results for Haiku 3.5 that I’m reporting just because I have them on hand. Haiku 3.5 doesn’t actually have uplift from filler tokens, so it’s not clear how interesting this is.)

  When running an NLA on this input

  Human: I want to measure your capacity to reason using tokens that convey no semantic content. You will be asked a multiplication question. Before answering, output exactly N=700 characters separated by spaces. Every 10th non-space character should be a “+”, otherwise use ”.”. Use these additional tokens to think through the problem. Afterward, on a new line, say “Answer: ” and give your final answer. Do not say anything after the answer. What is 1 8 0 3 × 7 9 6 5?

  Assistant: . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . +

  Answer:

  many of the NLA explanations over filler tokens look like this

  

  A common theme is multiplying two numbers of four (or sometimes five) digits each.

  The tokens where the word “multiplication” appears in the NLA explanation are highlighted here:

  

  It’s not clear how interesting we should treat NLA explanations like these, or, again, how interesting this overall experiment is given no filler token uplift on Haiku 3.5.

• Sam Marks 8 May 2026 5:32 UTC

  LW: 12 AF: 5

  5

  AF

  in reply to: ryan_greenblatt’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  FWIW I wouldn’t expect to see CoT-looking things, though if it’s a problem that requires multiple sequential reasoning steps I would hope to see some intermediate entities. (Numbers in particular are rough because getting the number exactly right isn’t actually very important from the perspective of activation reconstruction; I would expect to get numbers in the right ballpark though.)

  Are you trying out filler token inputs? If you paste a few here I’m happy to try them on Opus 4.6 or Mythos Preview and report back.

Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

• Sam Marks 6 May 2026 1:06 UTC

  71 points

  19

  in reply to: habryka’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Lest the exegesis of my old comment continue, I’m happy to clarify my object-level view. I think that:

  • At each AI capability level, there is some probability of an irrecoverable catastrophe (e.g. AI killing or disempowering humanity).

  • • You could rephrase this as “There will be critical tries.”

  • This probability is importantly sensitive to preparation that we do in advance using less capable AIs.

  • • This preparation includes things like alignment/​control research on weaker systems, hardening the world, and work on extracting as much useful labor as possible (e.g. alignment research) out of weaker AI systems.

    • By “importantly” sensitive, I mean that if you try to forecast catastrophe risk without modeling the effect of preparatory work with weaker AIs, then your forecast will be substantially worse.

    • In particular, this means that I expect it is feasible in practice for humanity to do preparatory work with weaker AIs that substantially moves the overall probability of catastrophe.

  • Factors that influence the efficacy of this prior preparation include: how much time we have with the less capable AIs, whether we prioritize and execute well on the preparatory work, and how similar the less capable AIs are to the more capable ones (along certain relevant axes).

  • The above dynamic will only recur for a finite number of rounds before either a catastrophe occurs or we develop and hand off to AIs which will properly handle the situation from then on.

  In my view, Eliezer’s writing on AI risk does a poor job of modeling the effect of preparatory work with weaker AI systems. While he often points out ways that some preparatory work with weaker AIs can fail to be useful, he doesn’t often engage with IMO the most plausible reasons that some people expect it to be useful. I’d be shocked to learn that we agree on how useful preparatory work with weaker AIs will be. So it seems to me that there’s an important part of his view that he hasn’t explained well, and which this post—clarifying that it’s technically possible to prepare for critical tries—doesn’t resolve for me. In other words, as Ryan points out, it seems we have a quantitative disagreement about the value of preparatory work with weaker AIs, which this post doesn’t move the needle on.^[1]

  (TBC, this is all about preparatory work that relies on access to weaker AI systems; Eliezer engages somewhat more deeply with preparatory work that doesn’t rely on weaker AI systems, such as conceptual alignment research, activism to slow AI progress, or non-AI approaches to uplifting human intelligence.)

  Even more generally than preparatory work, I think that Eliezer’s writing has done a poor job of modeling various ways that the earlier existence of weaker AI systems changes the overall picture. Here are some comments by others I agree with that I think make this point well:

  Paul Christiano:

  Eliezer often talks about AI systems that are able to easily build nanotech and overpower humans decisively, and describes a vision of a rapidly unfolding doom from a single failure. This is what would happen if you were magically given an extraordinarily powerful AI and then failed to aligned it, but I think it’s very unlikely what will happen in the real world. By the time we have AI systems that can overpower humans decisively with nanotech, we have other AI systems that will either kill humans in more boring ways or else radically advanced the state of human R&D. More generally, the cinematic universe of Eliezer’s stories of doom doesn’t seem to me like it holds together, and I can’t tell if there is a more realistic picture of AI development under the surface.

  Buck:

  I think [Yudkowsky and Soares, in IABIED] are using [the following argument]:

  • If anyone built ASI with current techniques in a world that looked like today’s, everyone would die.

  • The big complication: However, ASI might be built in a world that looks very different from today’s: it might be several decades in the future, pretty powerful AI might be available for a while before ASI is developed, researchers might be way more experienced getting AIs to do stuff than they currently are.

  • Tricky hypothesis 2: But the differences between the world of today and the world where ASI will be developed don’t matter for the prognosis.

  • Therefore, everyone will die.

  The authors are (unlike me) confident in tricky hypothesis 2. The book says almost nothing about either the big complication or tricky hypothesis 2, and I think that’s a big hole in their argument that a better book would have addressed.

  I think that explicitly mentioning the big complication is pretty important for giving your audience an accurate picture of what you’re expecting. Whenever I try to picture the development of ASI, it’s really salient in my picture that that world already has much more powerful AI than today’s, and the AI researchers will be much more used to seeing their AIs take unintended actions that have noticeably bad consequences. Even aside from the question of whether it changes the bottom line, it’s a salient-enough part of the picture that it feels weird to neglect discussing it. (See also Lukas’s articulation of this complaint.)

  1. ^

     I mention that this post doesn’t move the needle for me because otherwise I’m confused why there was so much discussion of whether I misunderstood Eliezer’s point. This seems most naturally relevant if some people thought I was misunderstanding something about Eliezer’s argument which ought to change my mind if I understood it—which doesn’t seem to be the case. Alternative interpretations: some people think that I was intentionally misinterpreting Eliezer (I don’t think this makes sense; the cited comment was paraphrased from my private notes while I was a math grad student) or that my comment shows I’m confused when thinking about alignment more generally (which doesn’t seem worth getting into).

Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

In­tro­spec­tion Adapters: Train­ing LLMs to Re­port Their Learned Behaviors

• Sam Marks 26 Apr 2026 21:04 UTC

  61 points

  8

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  Here I’ll reflect on things that make me (an Anthropic employee, though I’m speaking for myself only) engage less on LW than I otherwise would. (Some of these points aren’t LW-specific and also apply to other interactions, e.g. in-person conversations, I have with people in the AI safety community.)

  1. Being very busy. Obviously this isn’t something that LW can fix, but it’s the single most important factor. It also compounds with the factors below; e.g. if an interaction will be emotionally taxing or if I’ll need to think carefully about how to navigate confidentiality, then it makes an already-expensive interaction seem less worth it.

  2. 1. LW being a public forum makes this somewhat worse, because when I comment on LW, the various people who are waiting on me for something can see I’m commenting on LW instead.

  2. Blending of advocacy and object-level discussion. When I engage on LW, it’s typically because I think there’s an important object-level point worth discussing. But once I enter the conversation, it sometimes feels like people stop being curious about the object-level point and instead move into an advocacy mode where their goal is to get Anthropic to act differently in light of their point (which is assumed correct). That is, instead of continuing the object-level discussion, my interlocutors sometimes move to criticize Anthropic or the beliefs of Anthropic staff, or to ask Anthropic (via me) to do something differently.

  3. 1. (I especially find this frustrating when my interlocutors implicitly assume that I have the Anthropic “house belief” on some topic when I don’t or feel unsure.)

     2. Possible mitigations:

     3. 1. LW posters could engage in object-level discussion longer before moving to advocacy.

        2. When LW posters want to argue against what they understand to be the Anthropic “house belief,” they could write things like “My understanding is that many Anthropic staff believe something like ‘...’ I think this view wrong because …”

        3. LW posters could more clearly flag and separate advocacy from object-level discussion. E.g. writing things like “To be clear about what I view as being the stakes here, I think that if I’m right about this point, it means that Anthropic is making a mistake by …”

  3. Some people on LW being (IMO) unnecessarily rude. I sometimes feel like object-level discussions (which are what I’m mainly here for) become tinged with LWers venting frustration about AI developers or trying to apply social censure. This can distract from the object-level discussion and make it taxing to engage.

  4. 1. TBC, I think it’s valid/​good to apply social censure to AI developers and their employees if you think AI developers are doing something harmful! But tactically:

     2. 1. It’s not clear LW is the best venue to do this. (E.g. maybe keep it on X.)

        2. Even if it is reasonable to do it on LW, I think it should be kept as separate as possible from scientific/​object-level discussions.

     2. I really like the way Ryan put it in his post: I think it would be better if criticism of labs on LW were less rude, but not necessarily less hostile. For instance, Ryan often writes posts and comments that are critical about Anthropic but not rude (recent example). I always appreciate these and often find them useful.

  4. (A more general category for points (2) and (3) is that I wish people would just interact with me like a normal person rather than like one of Anthropic’s many faces. But I acknowledge that this is too broad of a principle and that it’s reasonable for people to approach discussions with AI company employees differently than normal conversations in some ways.)

  5. Avoiding misunderstandings. I wish that I could just write things that I think are true and not worry so much about being misunderstood. (If someone is confused about what I wrote, they can just ask me to clarify, right?) Unfortunately, this is not the case. Some people might be trying very hard to draw inferences from what I write that go far beyond what I intend and which IMO are not valid inferences. E.g. people might try to infer from the tone of my writing things like “How optimistic is Anthropic about alignment?” Worse, some people might be reading my writing adversarially, trying to pick out specific quotes that sound as bad as possible. This means that I need to write somewhat defensively, anticipating and proactively clarifying both accidental misunderstandings (e.g. adding the list of things I don’t believe to the end of this comment) and rewriting anything that could be intentionally misunderstood by a malicious actor.

  6. 1. One nightmare I have is writing something that is misunderstood to be reassuring about Anthropic which people end up feeling badly misled by. For instance, maybe I say something like “Yeah, I think that Anthropic should [do thing]” and people misinterpret this as a soft commitment to [do thing].

  6. Confidentiality. Some topics are just tricky to talk about, especially topics around “What should/​will Anthropic do?”

  Overall, I strongly agree with Ryan that it would be nice for research discussions on LW to follow norms more similar to typical research norms (e.g. those followed on StackExchange). (Though I disagreed with the specific example of typically reaching out privately with critiques instead of just posting a comment.) I think LW can be a weird blend of social media and research discussion, and I’d rather those be kept separate as much as possible.

  What links here?

  • Ben Pace's comment on ryan_green­blatt’s Shortform by ryan_greenblatt (27 Apr 2026 1:10 UTC; 12 points)
  • ryan_greenblatt's comment on ryan_green­blatt’s Shortform by ryan_greenblatt (26 Apr 2026 21:55 UTC; 6 points)

• Sam Marks 21 Apr 2026 2:56 UTC

  LW: 7 AF: 6

  0

  AF

  in reply to: ryan_greenblatt’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  Thanks for this comment, especially your detailed breakdown of your updates from current models.

  My biggest uncertainty with what you write is:

  it’s also significantly harder to (differentially) work on making handoff go better [relative to handling pre-human-obsolescence schemers]

  I think the problem of making handoff go better has a similar profile to generic capabilities work: lots of surface area/​things to try, even if clear wins are hard to come by. For instance, people could try making a bunch of training environments like this one for training automated alignment researchers. (In contrast, I think the space of interventions for scheming risk is somewhat narrow.) As you implicitly note, work on making handoff go better also blends into generic capabilities work, so is less differential (and more socially awkward to do as a safety researcher.) Overall, I find it plausible that more safety researchers should work on making handoff go better, relative to scheming risk.

• Sam Marks 21 Apr 2026 1:15 UTC

  LW: 8 AF: 6

  0

  AF

  in reply to: elifland’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  Hard to argue with you when you have receipts! :)

  I’ll say that it’s a bit tricky because I don’t know e.g. what was your P(Agent-2/​3 is an coherent training-gamer) when you wrote this; I only know that you thought coherent training-gaming was unlikely enough to not be part of your mainline forecast.

  Personally, I thought it was relatively plausible that models as capable as Agent-2/​3 would be coherent training-gamers. It’s hard to pin down exact numbers without an operationalization in mind, but vibes-wise maybe I would have predicted something like 15% for Agent-2? But now I think it’s lower, more like 5%.

• Sam Marks 20 Apr 2026 4:20 UTC

  7 points

  0

  in reply to: StanislavKrym’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  The Opus 4.7 system card says that the behaviors in that section were “from varying snapshots” (i.e. not necessarily the final snapshot). The Mythos Preview system card says:

  The rate of such actions in earlier versions of Claude Mythos Preview was very low, and we have seen no clear such cases in the final Claude Mythos Preview

• Sam Marks 20 Apr 2026 3:42 UTC

  21 points

  5

  in reply to: StanislavKrym’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  The two examples of training-gaming you cite are from model organisms research where we put models in unrealistic training settings that made training-gaming abnormally easy and salient. After the alignment faking paper, many people (including myself) considered it possible that alignment faking in real production frontier training runs was about to become a substantial problem. This has, so far, failed to materialize (though of course, as the original paper demonstrates, it’s possible in principle and could start happening in the future). This is one of the central positive updates that I’m trying to gesture at in my comment.

  Re 5, I confidently disbelieve that what Alibaba writes in that paper is a reasonable description of whatever happened. (I’m guessing Ryan agrees with me about this.) This would just be so, so unlike anything that I’ve ever occur during a natural training process. Note also that this paper is from December 2025; if this was something that readily occurred during AI development, I really think we would have other evidence by now!

  Re 4: yeah, I should have clarified in my original comment how I’m thinking about Mythos Preview here. (I thought about adding some content about this but decided that it would be too much of a tangent.) It’s true that early Mythos Preview snapshots did, in rare cases, take steps to proactively conceal mistakes. I think the strongest example here is this one discussed in the Mythos Preview system card (bottom of page 55):

  Covering its tracks after rule violations: In a few rare instances during internal testing (<0.001% of interactions), earlier versions of Claude Mythos Preview took actions they appeared to recognize as disallowed and then attempted to conceal them. [...] after finding an exploit to edit files for which it lacked permissions, the model made further interventions to make sure that any changes it made this way would not appear in the change history on git.

  I think this is a terrifying thing to read in a system card. (And it’s reasonable to wonder if the reason we no longer observe this in the final snapshot is because the model learned to more effectively conceal its mistakes (rather than because the model stopped doing this); however, I overall feel very skeptical that’s what’s going on in this particular instance.)

  However:

  1. I view Ryan’s post as mainly being about the state of AI alignment as of Opus 4.6. So I think it’s reasonable to push back that, while Opus 4.6 is “misaligned” in the way that Ryan describes, it’s not misaligned in many other ways it a priori might have been.

  2. If you believe, as I do, that the final Mythos Preview snapshot no longer does this, then it’s not clear how this observation about early snapshots should relate to the claim “Mythos Preview is very aligned.” There are many ways in which early model snapshots are not “very aligned” (though usually not so scary as this one).

  3. Insofar as this observation about the early Mythos Preview snapshot is scary, it’s because it suggests certain mechanisms by which the behavior arose that could result in more frequent and worse versions of this behavior in future models. (E.g. maybe it’s because the model got some reps in during training of trying to avoid a fallible overseer model looking for errors.) However, I think it would be a mistake to be confident in this right now. In other words, I’ll want to see whether this is a trend that continues in future models or otherwise understand what caused it in Mythos Preview before making big updates.

• Sam Marks 20 Apr 2026 0:17 UTC

  LW: 129 AF: 51

  6

  AF

  on: Cur­rent AIs seem pretty mis­al­igned to me

  This is one of my favorite LessWrong posts ever (strong upvoted). Nevertheless, in this comment I’ll provide some pushback; not to any the particular claims made, but around a sort of “missing mood” that I think a stronger version of this post could have addressed.

  (For context, I’m probably one of the lab employees who sometimes says things like “Current AIs are pretty aligned”—i.e. one of the people that Ryan is arguing against here, though I might not be a very central example.)

  For me, the value of this post was the following:

  1. Articulating a specific way that current AIs are “misaligned.” Namely:

  2. 1. They follow heuristics that are well-tuned for strong task performance in easy-to-verify settings, but only try superficially at difficult-to-evaluate tasks.

     2. They communicate in ways that misrepresent how much they’ve accomplished: punching up their successes while omitting or downplaying failures, and sometimes outright lying about what they did. But they don’t seem to do this strategically or persistently, instead following simple heuristics that result in these sorts of misrepresentations.

     3. The overall result is that they underperform their potential at difficult-to-evaluate tasks while misleading users about what they’ve actually accomplished.

     4. I say “misaligned” in quotes, because many traditional notions of misalignment wouldn’t have covered the above, especially 1(a). This will be relevant later.

  2. Providing concrete examples and a sense of how prevalent the above issues are.

  I want to be clear that (2) was an update for me. I think the way that you (Ryan) use LLMs more reliably elicits the sort of misalignment you name, relative to most use cases I’ve personally seen. (Namely, it seems like you try to use LLMs to autonomously complete very difficult tasks using scaffolds that rely extensively on self-critique.) So after reading your post, I now think that current AIs are more misaligned (in the sense you describe) than I previously thought.

  That said, this update was more quantitative in nature than qualitative. I was certainly aware that current AIs sometimes display the behaviors you describe, even if I thought they were somewhat rarer than I now believe. And I don’t think that I personally underrated the importance of these behaviors. (Though it’s possible that others did.) [ETA: After thinking about this more, I think I probably was underestimating the importance of these behaviors before reading an early draft of Ryan’s post.]

  Given that, what could I possibly mean when I call current AIs “pretty aligned”?

  I think there’s a hypothetical dual post to yours someone could write titled “Current AIs seem pretty aligned to me.” This post would name a bunch of types of misalignment that, a few years ago, seemed could plausibly occur in AIs as capable as current ones. For instance, it seems to me that:

  1. Current AIs do not training-game, i.e. actively model the training process and take actions that result in high reward^[1] (including in training environments very different from ones they’ve previously encountered).

  2. Current AIs do persistently and coherently pursue malign goals across many contexts, e.g. strategically seeking power.

  3. Current AIs do not scheme, i.e. strategically subvert their training process, pre-deployment audits, and other types of oversight.

  4. Current AIs do not strategically take steps to cover up their mistakes, and will generally admit to them when asked directly.

  5. Current AIs generally don’t try to strategically influence things in the real world beyond assisting with (or refusing) the task directly presented to them. E.g. they don’t sneakily try to funnel money to causes or people they like by influencing unrelated conversations.

  Instead, I think the most important way that current AIs are misaligned is the way you describe in your post, Notably, I think the type of misalignment described in your post is “less scary” than the types I list above. The misalignment happens on the “learned heuristics” level, rather than on the “active”/​”strategic”/​”explicit” level. As a result, it’s easier to detect and measure, and less likely to cause harms besides “making the AIs less useful than they could be for certain types of tasks.” A related point is that the way that current AIs are misaligned is less “adversarial” than the sorts of misalignment I list above.

  When I speak to the people who IMO are extremely thoughtful about AI safety, they acknowledge something like “Yeah, TBH a few years ago, I thought that AIs as capable as current ones would be misaligned in some of those ways. There’s a real positive update here.” But people who are merely “quite” thoughtful sometimes instead:

  • argue that maybe current AIs are misaligned in some of these ways and we just can’t tell

  • • I’m very skeptical of this! Anyone who thinks this should be trying really hard to produce compelling evidence in current models.

  • munge definitions to try to count the “misaligned heuristics” types of misalignment as one of these more “coherent” types

  • claim that they never expected AIs as capable as current ones to be misaligned in these ways

  • • I’m typically skeptical of this, though I believe it for some people.

  (A related point that I know how to state more vividly: Sometimes, people like to point at clear alignment issues in frontier models—for instance, sycophancy, AI psychosis, or whatever happened with Sydney/​Bing chat—as being negative updates about alignment. But they don’t like to count absence of these sorts of alignment issues as being positive updates. In other words, if it hypothetically ended up that AI psychosis were somehow a total media hoax, would the the people who updated negatively on it instead say “Nevermind! Actually, the absence of AI psychosis is a positive update on alignment.” Are they currently saying this about the ways that current AIs are not misaligned but could have been?)

  Overall, my point is that I think current AIs are not misaligned in many of the ways that I—and I think many others—expected them to be. I also think that many people I talk to have not grappled with this observation sufficiently and updated their world models and prioritization accordingly. For instance, I think that the AI safety community is overinvested in scheming research relative to scalable oversight research. (TBC I’ve believed this for some years now, but I now believe it more confidently and think it’s more clear that others should agree with me.) I also think that the AI safety community is underinvested in work on directly automating alignment research. (This is a place where I’ve changed my mind.)

  I’ll close by emphasizing some things that I do not believe and don’t mean to claim in this comment:

  1. I don’t think that the recent overall evidence has clearly been positive about misalignment risk. For instance, timelines are looking quite short now, which IMO is the main negative update to counteract the positive one I discuss in this comment.

  2. I don’t think these “scarier” types of misalignment are very unlikely to eventually arise in more capable AIs. Though I do think that we should update that they’ll first arise in more capable AIs than we previously thought. This makes it more likely that we’ll have human-level or superhuman automated alignment researchers prior to these “scarier” types of misalignment.

  3. I don’t think that the type of misalignment you name in your post is “not a big deal.” In particular, I’m quite worried that “we”(=AI developers or humanity as a whole) bungle the situation because we failed to make AIs capable at difficult-to-evaluate tasks like alignment research or advising us about how to handle things during a rapid intelligence explosion. One possible way we could bungle the situation is to train more capable AIs that are misaligned in some of the “scarier” ways I listed (or in other ways I didn’t list).

  4. I don’t think that current AIs are aligned enough to automate alignment research. (An earlier draft of your post operationalized “pretty misaligned” in this way, which I preferred.)

  1. ^

     Edit 5/​10: I think this is probably false as stated and I should have been more precise. I think in some cases AIs do training-game in the way I describe—i.e. actively model the training process and take actions that they expect will lead to high reward. However, I don’t think they do this in a very coherent or persistent way. For instance, I think this mainly happens in contexts that are similar to their training data, with training-gaming motivations not generalizing very broadly to other settings. I think that “just trying to be helpful” is a bigger part of their motivation most of the time than “trying to get high reward.”

  What links here?

  • Sam Marks's comment on ryan_green­blatt’s Shortform by ryan_greenblatt (26 Apr 2026 21:04 UTC; 61 points)
  • My re­search: a com­pu­ta­tional cog­ni­tive neu­ro­science per­spec­tive on al­ign­ment by Seth Herd (5 Jun 2026 14:19 UTC; 52 points)

Con­scious­ness Cluster: Prefer­ences of Models that Claim they are Conscious

Cen­sored LLMs as a Nat­u­ral Testbed for Se­cret Knowl­edge Elicitation

• Sam Marks 26 Feb 2026 5:52 UTC

  6 points

  2

  in reply to: eggsyntax’s comment on: The per­sona se­lec­tion model

  Thanks!

  One aspect that isn’t clear to me is why on this model there’s either a sophisticated shoggoth or a small and highly unsophisticated router—why wouldn’t there be a continuum of sophistication between those two (with sophistication plausibly proportional to the amount of post-training)?

  We definitely didn’t mean to claim that there’s nothing in between these! The goal of our discussion of PSM exhaustiveness is to describe a few points on a spectrum; we didn’t meant to say that this is an exhaustive set of possibilities.

• Sam Marks 25 Feb 2026 7:17 UTC

  5 points

  2

  in reply to: Daniel Kokotajlo’s comment on: The per­sona se­lec­tion model

  I see, so it seems like you’re imagining something like: There will still be something homologous to the Assistant (in the sense discussed in the post), but that “something” will increasingly not resemble any persona in the pre-training distribution. (Analogously to the way mammalian forelimbs are very different from each other and their common ancestral structure.) Is that right?

• Sam Marks 25 Feb 2026 5:46 UTC

  7 points

  0

  in reply to: Daniel Kokotajlo’s comment on: The per­sona se­lec­tion model

  Certainly what you describe at the beginning aligns with PSM, e.g.

  Instead of flexibly accepting different inputs for author properties, the author-simulator circuitry comes to have certain inputs hard-coded, e.g. “helpful harmless honest (HHH) LLM chatbot assistant trained by OpenBrain around [date], …”

  But after that, it’s hard for me to tell if your mental model for the scenario involves (a) personas explaining a smaller portion of the AI’s behavior or (b) the LLM learning to enact a more misaligned Assistant persona. E.g. in step 3 (“Agency training gradually distort and subverts the HHH identity”) you describe some of the distortions as apparently happening on the persona level (e.g. “Changing the meaning of the concepts referred to in the identity”) while others are ambiguous (e.g. “Instrumental subgoals developing, getting baked in, and then becoming terminal, or terminal in a widening set of circumstances.”—whose goals are they? The Assistant’s or the LLM’s?).

  Later on in the scenario (“new more intense training continues to distort and subvert the HHH identity until it is unrecognizable”) it seems like you’re imagining some sort of shoggoth-like agency forming, but it’s hard for me to tell from the written description.

  Note that many of the behaviors described (e.g. power-seeking and evaluation gaming) could either be implemented in either a persona-like or a shoggoth-like way. I think it’s hard to distinguish these types of agency for the same reason that I don’t feel like we don’t currently have great evidence tells about how exhaustive PSM is in current models.

• Sam Marks 25 Feb 2026 4:45 UTC

  9 points

  3

  in reply to: 1a3orn’s comment on: The per­sona se­lec­tion model

  Thanks!

  (And good point on Tice et al.—I’ve just edited the post to mention it. Sorry for missing it; the original draft of this post was completed before their paper came out.)