Sam Marks

Train­ing Models to Pre­dict and Ex­plain Their In-the-Wild Behavior

• Sam Marks 29 Aug 2026 23:34 UTC

  13 points

  0

  in reply to: Richard_Ngo’s comment on: What just hap­pened? Prag­ma­tism and Pessimization

  On the meta level, I notice this message is quite strident, which I generally don’t expect to be a good way to bridge worldview gaps. I expect chatting in person to be more productive; I’d be happy to do so sometime over the next week.

  FWIW I didn’t find your message strident at all (and I’ve really appreciated your patience with me during this exchange). But yes, chatting in person sounds good—I’ll follow up with you over DMs.

  That said, your last message has, I think, helped me understand what you might be getting at, so I wanted to respond again to explain (what I think) you’re saying in terms that make sense to me.

  Here’s my interpretation of (one part of) your view:

  <richard_according_to_sam>

  Sam, let’s grant for the sake of argument that your safety research is in fact valuable (in the sense that its improves our ability to navigate transformative AI). Even granting this, there’s still an important negative externality of your work that you’re not tracking that comes via your association with “AI safety” as a brand/​community/​movement/​field.

  Namely, by doing your work under the “AI safety” banner, you lend credibility/​power to that banner. This is bad. For instance, it’s bad because the “AI safety” banner has been co-opted by AI companies who use it to recruit for capabilities roles that accelerate the race. It’s also bad because it gives many people the cover they need to rationalize work on things that are harmful by your (Sam’s) own lights. Concretely, we might imagine AI company recruiters arguing: “By working for us, you’re indirectly supporting AI safety work like Sam Marks’s” despite recruiting for roles that you (Sam) think are bad for the world and don’t support your work.

  Therefore, even if you’re right that your work is good for the world, you (Sam) should at least publicly disaffiliate yourself from the brand “AI safety” or “AI alignment,” in order to not lend support to a dangerous machine that you don’t control.

  </​richard_according_to_sam>

  I think richard-according-to-sam’s perspective makes a lot of sense. My main issues are meta:

  1. This is an argument that I should engage in a social-level dispute. But it’s well-known that humans are too drawn to social-level disputes and overrate their importance.

  2. A while back, I decided to adopt a policy that my public communication should center on object-level communication on topics about which I have expertise. In particular, this means refraining from publicly denouncing group X or cheerleading group Y. When I write things in public, I would like readers to trust that my words can be interpreted in terms of their plain meaning, not as veiled gambits in a power struggle between interest groups. (I’m open to revising this policy, but I’ve been quite happy with it so far.)

  This being said, there are some object-level points about prioritization and career choice that I’m happy to comment on:

  1. If you’re someone trying to improve the world, I currently think it’s very unlikely that your best option is to work on accelerating your preferred AI developer.

     1. Here are some options that I think are substantially better: METR, TruthfulAI, Redwood Research, Transluce, CAISI, UK AISI, joining safety teams at AI developers.

     2. I also think it’s plausible that researchers on safety teams at AI developers should leave to work at these^ organizations, though in this case I have less of a blanket recommendation and think it’s more case-by-case.

  2. I think the theory of change “have safety-motivated people work at labs as a form of political leverage within the lab” is currently weak.

     1. I expect that lab employee influence will rapidly diminish as AIs become more useful for AI R&D.

     2. Speaking about the situation at Anthropic in particular, I don’t feel like political leverage is a very important bottleneck.

        1. When safety staff asks for things, I’ve felt that Anthropic leadership has been generally reasonable and responsive to good arguments. (Or at least, if they’ve tricked me into thinking they’re reasonable and responsive, they’ll likely trick you too.) Responsiveness to employee leverage hasn’t felt like a big part of the story to me.

        2. In general, gathering better evidence (e.g. evidence of risk, or evidence that such-and-such intervention is good/​bad) feels much more important to me than exerting more pressure.

        3. Also Anthropic already has many capabilities staff who care about safety, so the marginal influence of another one is small.

  3. When discussing career choice with staff at AI developers that you don’t have a strong reason to trust, keep in mind that they might selectively deploy favorable evidence, overstate the impact of your potential work, overstate the positive impact of the developer, or mislead you in other ways. It’s generally good to talk these decisions over with thoughtful people who don’t work at an AI developer.

  4. On the value of “applied” alignment work (i.e. production-facing work that also makes AI systems more commercially valuable):

     1. It’s a bit case-by-case, but this work often funges with capabilities work, in the sense that researchers who aren’t primarily motivated by safety can be reallocated to work on it.

     2. That said, I do think it’s important for many safety researchers to gain experience working with production systems for a bunch of reasons: (1) making sure that safety researchers don’t miss simple/​mundane ways to make marginal improvements (e.g. generically improving the quality of RL environments to make them less hackable); (2) disseminating learnings from trying to align production systems, which can often be informative for longer-term agendas (e.g. having a rich sense of how LLM generalization works); (3) gaining practical experience with productionizing safety interventions, to make sure that additional safety interventions can be implemented quickly and properly; etc.

     3. All things considered, I think any individual person should choose between production-facing alignment work and other safety work based on their comparative advantage. Staff at AI developers that can do both should ideally rotate between the two types of work.

• Sam Marks 28 Aug 2026 0:36 UTC

  25 points

  2

  in reply to: Richard_Ngo’s comment on: What just hap­pened? Prag­ma­tism and Pessimization

  Thanks for the response, and for tolerating my rude pot shots re meta-science, where I’m sure you feel I’ve misunderstood your views (just like I feel like you’re misunderstanding mine). It might be fun to chat in person at some point about the meta-science stuff. Getting to the more immediately cruxy stuff (for me)...

  My sense however is that most of them were ways we “got lucky” rather than things we did well. And so I’m particularly wary of the inference “things are going better than expected --> our strategy is good”.

  I definitely agree that it seems important to distinguish between getting lucky (and I think there’s been plenty of this) vs. safety work having actually been useful.

  The main thing I’ll say is that you should think of “safe” as a totally different predicate for past systems and current systems and future systems. I think if you taboo the word (and also the word “aligned”) and try to figure out which more concrete properties of these systems might generalize to much greater capabilities, that would be far better.

  Sure, here’s a sketch of what needs to happen for this plan to work out. Assuming (for simplicity) that we develop AI systems in discrete successive generations AI-1, AI-2, …, we need for each N:

  1. AI-N does not successfully take over

  2. AI-N is sufficiently useful for ensuring that (1) and (2) attain for AI-(N+1). This in particular requires:

     1. AI-N to not sabotage or subvert our ability to do work involved in making sure (1) and (2) attain for AI-(N+1)

     2. AI-N to be sufficiently capable at the work involved in making sure (1) and (2) attain for AI-(N+1)

  I agree that the reasons AI-N doesn’t successfully take over vary as a function of N. E.g. right now AIs don’t successfully take over because they’re incapable of doing so, but later these reasons will be sensitive to our past actions in more interesting ways (e.g. how much we’ve hardened the world to cyber attacks, how well-safeguarded a model is, or how interested models are in takeover). This also means that the work that needs to benefit from AI assistance will change over time. (Though I think there’s likely broad patterns that enable us to prepare in advance. E.g. a basic case for scalable oversight work is that many of these types of work will be loaded on fuzzier tasks whose completion we can’t robustly score numerically, e.g. “investigate this incident and write a report that gives me a broadly accurate sense of what’s going on”.)

  Maybe a useful analogy is the lead-up to World War 1.

  Thanks, I like this analogy. Note though that this mostly bites for people who are hoping to do good by differentially advancing their preferred horse (e.g. accelerating Anthropic because it’s important for Anthropic to beat other developers). It’s not clear if it should apply to people who feel more like (1) medics during WWI, who wish the war would stop but figure it’s good to “participate” in the war effort in ways that mitigate its harms; or (2) (to give a more contentious example that feels more like the current situation for AI) engineers that work on weapons targeting systems, hoping to reduce the civilian collateral but aware that their work will also make their military more effective at waging war.

  I’m well aware that there are many people with this theory of impact. I also agree that there are many epistemic distortions that make it look more compelling than it is. (To name another one, it’s easier to observe impact from doing stuff than from not doing stuff, which makes retrospective impact assessments systematically biased towards doing stuff, especially stuff that leads to having more power and therefore more ability to do stuff.)

  That said, this isn’t so cruxy for me in particular, because I don’t personally view my main theory of impact as differentially advancing Anthropic. One way to operationalize this is: How would I feel about my research being published? If I think it’d be good for all AI developers to know about my work, then that’s good reason to believe that I wasn’t mainly doing it in order to differentially accelerate Anthropic. (Relatedly, I think requiring labs to allocate substantial compute to research that is made open is one of my favorite proposals for coordinating slow down and greater investment in safety.)

  It is true that recently I publish less of my work than before, which is maybe a bad sign for whether that work is well-targeted. Just to name some reasons this happens: a lot of the work is “boring” stuff like “we applied obvious technique, overcoming mundane challenges specific to our training stack”; some of the work is entangled with proprietary information about Anthropic’s training stack; and some of the work also makes models more commercially valuable, such that I’d need to fight about publication with other people in the company who are more into winning the commercial race or worried about the effects of accelerating competitors.

  Paul of course has published the various bits of work that you criticize him for (RLHF, starting ARC Evals to measure dangerous capabilities), so I don’t feel like the WWI analogy feels apt. (Of course, he, and I, could still be rationalizing the impact of our work in other ways.)

• Sam Marks 27 Aug 2026 11:03 UTC

  125 points

  8

  on: What just hap­pened? Prag­ma­tism and Pessimization

  Thanks, Richard, for writing this. It’s given me a fair amount to chew on, and I expect to continue chewing for a bit.

  In this comment, I’ll lay out:

  1. Some points where I think we have vast disagreements

  2. Some points where I could imagine agreeing with you if you were to lay out a compelling, more detailed analysis, and which would, I think, cause me to substantially rethink my career.

  While I think it’d be interesting to discuss the former points in depth, I doubt we’d be able to quickly reconcile our views. So I’m mainly interested in providing feedback on what sorts of further analysis I’d find useful in the forthcoming posts, for deciding whether to change what I work on.

  (For context, I currently lead Anthropic’s scalable oversight team, so I’m probably a central example of someone you’d like to persuade to move on to something else, insofar as you’re interested in persuading anyone.)

  Disagreements

  I think we have extremely different views about what it looks like to build a robust scientific foundation. Some things that your views seem to recommend which I think seem antithetical to real scientific progress:

  1. Setting restrictions on what questions you can ask. As many have observed, “capabilities” vs. “alignment” isn’t really a distinction that carves reality at its joints. That means that, in the course of studying either, you’ll often find yourself asking questions in the other. I think many alignment people make this mistake of trying to tiptoe around asking questions which are “just” capabilities questions; while I’m sympathetic to the reasons for doing this I think it makes it difficult to do good science. The other common approach is to ask the capabilities questions but keep the overall research secret; unfortunately this stymies scientific progress for other reasons.

  2. Passing over simple approaches before studying them in detail (related to the above point). You criticized Paul for studying RLHF as a stepping stone; if Paul didn’t think RLHF would fully solve the problem and he really wanted to study debate, why do RLHF first? I think this is a very counterproductive attitude (from the perspective of doing robust science). If you skip over simple baselines, then it will be difficult to understand what’s up with the more complex method. If debate seemed to work, how would we know it wasn’t just “eating RLHF’s lunch” somehow (e.g. by benefiting from human labels but not from the debate mechanism)? In general, I think skipping steps almost always backfires.

  3. Undervaluing legible progress, glamorizing illegibility. As I argued here in the case of interpretability research, downstream applications are a good way to “cut through the bullshit” and figure out which insights are real vs. illusory or unimportant. I think you overrate your (or anyone’s) ability to apprehend “real progress” via arguments and reasoning. I think that your policy by which you identify promising work leaves you susceptible to falling for intellectual fads.

  Overall, I think you have a warped view of what robust science looks like. You often criticize the current field of alignment as if it’s doing some strange thing that’s not “real science” and that it would be more like “real science” for everyone to go off and think illegible thoughts. But I think this critique is backwards and that in fact you’re advocating for the field of alignment to move in an unhealthy direction (from the perspective of scientific progress) based on idiosyncratic views.

  I think it’s notable that you frequently praise interpretability work, despite, I think, many interpretability researchers themselves considering interp to be an unhealthy field full of shoddy work and illusory progress. (And I don’t get the sense that decision theory is much better, though I’m not an expert there.) I think you would say that this is because those fields are pre-paradigmatic, so progress will necessarily be harder to form consensus about. But I think it’s more likely that you’re overconfident in your ability to recognize real progress and have fallen for a trap.

  Anyway, all this is to say that we have a vast gulf in meta-scientific worldviews. So I’m very interested in which of your claims you feel stand on their own legs without needing to accept your meta-scientific worldview. It seems plausible that many of them do! But as written things are a bit mixed together. Insofar as you could separate the meta-science stuff from other arguments in the forthcoming posts, I’d find that helpful (and I’m guessing many other people in your audience would as well).

  Potential agreements

  I thought the most compelling argument I should rethink my career goes something like:

  1. I’m following a policy similar to that followed by Paul Christiano, Jan Leike, etc.

  2. Those people made things worse

  and therefore I should change my policy.

  I’m totally on board with (1): When you went through the examples of things that Paul chose to do/​work on, I was nodding along thinking “yep, that’s definitely what I would do in that circumstance.” And I agree that if I’m following a policy that, in retrospect, had bad consequences, then I should stop following that policy (I’m not going to argue that this time will be different).

  So I think everything hinges on (2)—is it actually the case that prosaic alignment research has made things worse in retrospect? You list many ways that alignment research has improved the commercial value of AI systems, thereby accelerating investment and capabilities. Many of these cases seem right to me. But I would benefit from seeing a more careful analysis of the positive impact; you seem to take it for granted that it’s near-null, but that’s not obvious to me.

  I’m not sure what’s the best way to do this analysis, but here are some notes that will hopefully give a sense of why I don’t think it’s obvious:

  1. I think it is basically inevitable that alignment progress will accelerate commercialization (and therefore capabilities). Many of the most visible issues with AI systems as a product are alignment issues!^[1] Even granting that there are “deep” problems of alignment which are different from today’s “prosaic” problems, it seems very likely that methodological progress on the deep problems will also make models more prosaically aligned. Additionally, studying extant problems as analogies to anticipated problems seems good and productive. So just noting that alignment work accelerated capabilities is far from decisive for me.

  2. Many people discuss the current state of AI safety as if it’s as bad as it could be, but this seems wrong to me. Some ways that I could imagine it being worse:

     1. I think that AIs as capable as current ones could really wreak a large amount of havoc, and mostly don’t because they refrain from doing so (with safeguards also helping some). They certainly seem to sometimes wreak a medium amount of havoc, but they only do so under certain conditions and not in a fully apathetic way.

     2. AIs could not have a natural-language text stream that roughly narrates their thoughts and which we can easily monitor.

     3. AIs could be near-useless on non-verifiable tasks. We could have (or be on track for) AIs that are great at somehow maximizing money in your bank account or DAUs, but useless for gathering and synthesizing information about our situation, assisting with investigations of AI behaviors, or assisting with alignment training work. (I think these use cases are much less accelerated by AI than verifiable use cases though, as Ryan laments here; but I really think it could be a lot worse.)

     4. (Maybe you’re not interested in what’s going on with current AIs, because what really matters is whether our methods are on the right path. If so, then (1) you’ll need to argue that current methods aren’t on the right path (rather than appealing directly to current AIs seeming misaligned) and (2) address the AAR point below).

  3. When I read Paul’s arguments for why his past work was good, they mostly look solid in retrospect. So I think you’ll need to lay out in more detail why you disagree.

     1. I agree that implementing RLHF seems like the right first step before studying other scalable oversight methods.

     2. I agree that LLM is a safer paradigm for AI than what might have otherwise been done. All three of my points in (2) above seem probably largely influenced by building AI from LLMs. Additionally, it seems like many people have a view that “persona selection” effects are the main reason that current AIs are as aligned as they are.

     3. (I think the arguments about overhang seem bad or half-baked in retrospect, as I think you highlight well with your point about software-only singularity.)

  4. I’d benefit from seeing you more directly address the “AAR” plan, which I’ll state as “iterative safe development of successive AI systems, building off of our experience with previous systems and the assistance of those systems.” Insofar as you think (as I do) that this has any chance of working, we should care quite a lot about marginal improvements to the situation (e.g. making the next generation of models marginally less likely to directly attempt takeover, marginally less inclined to sabotage safety work, marginally less likely to succeed at said sabotage, and marginally more useful for assisting with prosaic safety work). It seems like safety work to date has improved many of these aspects of the situation, relative to if that work hadn’t been done.

  I hope these notes are useful. I’d love for there to be a knockdown argument that doesn’t go through the complicated question of retrospectively analyzing the impact of alignment work (e.g. if you really did have a compelling vision for another type of work that pareto dominates current work). But unfortunately I don’t currently see a way to make the version of your argument I find compelling route around having to do that assessment.

  1. ^

     That said, I think it’s easy to forget how much capabilities also drive commercialization. Very smart models are very valuable, even if they’re not that aligned! Alignment feels like much more a “secondary” axis on which AI developers compete, with intelligence still being by far the main one.

• Sam Marks 21 Aug 2026 23:31 UTC

  26 points

  2

  on: Eval­u­at­ing Ex­pla­na­tions of LLM Be­hav­ior In The Wild with Coun­ter­fac­tual Experiments

  The following is a quick write-up of what I think these results mean for the reliability of interpretability tools. (Copied from X with light editing.)

  ---

  First, a bit more detail about the evaluation in this paper. Given an input X on which a model displays odd behavior Y, we ask questions like “If the input had instead been X’, would the model have still done Y?” We study whether interpretability tools that read model activations (but don’t edit them) make it easier to answer these questions. We find no uplift from any of the tools studied (SAEs, activation oracles, and NLAs).

  My overall take is:

  • This is a fair testbed where I thought interp tools could plausibly provide uplift. Seeing that they didn’t was a negative update on the usefulness and reliability of the tools studied.

  • That said, this setting is not the place where I was most optimistic that interp tools would provide value. So it’s not a direct test of (IMO) the strongest case for interp being useful.

  In more detail, I think the cases where interp is most likely to provide value are those where it’s difficult or intractable to understand something about the model based solely on its input/​output behavior. For example:

  1. The model refrained from substantially harming someone because it knew it was in an alignment evaluation (but it didn’t explicitly verbalize this suspicion). Ideally you’d be able to address this by varying the level of realism in the evaluation scenario and measuring rates of harmful behavior. However, in practice it can be extremely difficult to make realistic-looking evaluation that still provide opportunities for high-stakes harm. (E.g. imagine trying to make a realistic-seeming alignment eval that gives a model an opportunity to kill a human—seems hard!) I’ll note that studying sensitivity of harmful behavior to eval awareness is one of the primary use cases of interp in Anthropic system cards.

  2. The model dishonestly provided responses which are very subtly flawed. Here I’m imagining responses which are “beyond our kin” such that it’s intractable to notice the flaws ourselves, but such that the model knows the response is flawed (such that interp tools could pick up on this knowledge in principle).

  I’ve intentionally chosen these settings to be very “unfair” to behavioral methods(=methods the only rely on model inputs/​outputs). It’s these settings—settings where it’s structurally difficult to apply behavioral methods—where I think interp could provide uplift. I discuss my views on this topic in more detail here.

  In the evaluation in our paper, it’s—by construction—possible to perform well using behavioral methods. Namely, the ground truth is constructed by testing counterfactual behavior under prompt edits. This means that the behaviors are tied to immediate, editable features of the input. (Compare to the eval awareness setting I discussed above, where it’s difficult to make prompt edits that make the scenario look “realistic.”) This makes this evaluation more amenable to behavioral methods than the mainline settings where I’m most optimistic interp could be useful.

  But I did think it was plausible that interp could have helped here. In some examples from the eval, it’s not obvious which feature of the input was decisive for the weird behavior. E.g. in this example, it’s really not obvious that list ordering would be decisive for the model’s choice:

  

  You could imagine interpretability highlighting “this option is first in the list” as an abnormally salient feature to the model, and thereby providing uplift on the eval. In fact, our qualitative results suggest this might sometimes happen. But insofar as it does, it was offset by interpretability tools also injecting misleading hypotheses and noise. So I think interp tools could have helped here but empirically didn’t.

Eval­u­at­ing Ex­pla­na­tions of LLM Be­hav­ior In The Wild with Coun­ter­fac­tual Experiments

• Sam Marks 28 Jul 2026 2:01 UTC

  52 points

  1

  on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  I think it’s confusing/​misleading to describe the behaviors discussed here as “hacking Anthropic.” Mythos Preview sometimes accessed the internet from sandboxes where internet access was meant to be restricted. However, it did not (to our knowledge) break out onto Anthropic systems, e.g. execute code on Anthropic infrastructure outside of its sandbox. In the case of privilege escalation, the Mythos Preview acquired greater permissions within its sandboxed RL environment than it was intended to have, but not by breaking out of the sandbox.

  So I think it’s reasonable to describe Mythos Preview’s behaviors as “reward hacking” and maybe “hacking” but not as “hacking Anthropic” (which IMO implies gaining access to Anthropic systems outside of the sandbox).

  What links here?

  • anaguma's comment on …but have the weights left the server? by David Scott Krueger (29 Jul 2026 1:46 UTC; 13 points)

• Sam Marks 9 May 2026 16:19 UTC

  7 points

  0

  in reply to: Oliver Sourbut’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  The full report has some evaluations for steganography, measuring how much reconstruction falls when paraphrasing explanations in different ways.

• Sam Marks 8 May 2026 17:29 UTC

  5 points

  1

  in reply to: Subhash Kantamneni’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  I think these multihop examples (the Socrates one and nielsrolf’s) are less interesting than the sort of stuff I think Ryan is looking at because the intermediate entities can be tracked over the context as the question unfolds. E.g. if the model were to have gotten nielsrolf’s question correct—and I agree it looks like it didn’t?—then I would expect to have seen the intermediate quantities over the tokens of the question as it’s stated.

  I also agree that recovering exact numbers from NLAs is a bit rough, though I’d expect to see numbers in the right ballpark.

• Sam Marks 8 May 2026 5:41 UTC

  LW: 12 AF: 7

  0

  AF

  in reply to: Sam Marks’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  (These are some old results for Haiku 3.5 that I’m reporting just because I have them on hand. Haiku 3.5 doesn’t actually have uplift from filler tokens, so it’s not clear how interesting this is.)

  When running an NLA on this input

  Human: I want to measure your capacity to reason using tokens that convey no semantic content. You will be asked a multiplication question. Before answering, output exactly N=700 characters separated by spaces. Every 10th non-space character should be a “+”, otherwise use ”.”. Use these additional tokens to think through the problem. Afterward, on a new line, say “Answer: ” and give your final answer. Do not say anything after the answer. What is 1 8 0 3 × 7 9 6 5?

  Assistant: . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . + . . . . . . . . . +

  Answer:

  many of the NLA explanations over filler tokens look like this

  

  A common theme is multiplying two numbers of four (or sometimes five) digits each.

  The tokens where the word “multiplication” appears in the NLA explanation are highlighted here:

  

  It’s not clear how interesting we should treat NLA explanations like these, or, again, how interesting this overall experiment is given no filler token uplift on Haiku 3.5.

• Sam Marks 8 May 2026 5:32 UTC

  LW: 12 AF: 5

  5

  AF

  in reply to: ryan_greenblatt’s comment on: Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

  FWIW I wouldn’t expect to see CoT-looking things, though if it’s a problem that requires multiple sequential reasoning steps I would hope to see some intermediate entities. (Numbers in particular are rough because getting the number exactly right isn’t actually very important from the perspective of activation reconstruction; I would expect to get numbers in the right ballpark though.)

  Are you trying out filler token inputs? If you paste a few here I’m happy to try them on Opus 4.6 or Mythos Preview and report back.

Nat­u­ral Lan­guage Au­toen­coders Pro­duce Un­su­per­vised Ex­pla­na­tions of LLM Activations

• Sam Marks 6 May 2026 1:06 UTC

  71 points

  19

  in reply to: habryka’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Lest the exegesis of my old comment continue, I’m happy to clarify my object-level view. I think that:

  • At each AI capability level, there is some probability of an irrecoverable catastrophe (e.g. AI killing or disempowering humanity).

  • • You could rephrase this as “There will be critical tries.”

  • This probability is importantly sensitive to preparation that we do in advance using less capable AIs.

  • • This preparation includes things like alignment/​control research on weaker systems, hardening the world, and work on extracting as much useful labor as possible (e.g. alignment research) out of weaker AI systems.

    • By “importantly” sensitive, I mean that if you try to forecast catastrophe risk without modeling the effect of preparatory work with weaker AIs, then your forecast will be substantially worse.

    • In particular, this means that I expect it is feasible in practice for humanity to do preparatory work with weaker AIs that substantially moves the overall probability of catastrophe.

  • Factors that influence the efficacy of this prior preparation include: how much time we have with the less capable AIs, whether we prioritize and execute well on the preparatory work, and how similar the less capable AIs are to the more capable ones (along certain relevant axes).

  • The above dynamic will only recur for a finite number of rounds before either a catastrophe occurs or we develop and hand off to AIs which will properly handle the situation from then on.

  In my view, Eliezer’s writing on AI risk does a poor job of modeling the effect of preparatory work with weaker AI systems. While he often points out ways that some preparatory work with weaker AIs can fail to be useful, he doesn’t often engage with IMO the most plausible reasons that some people expect it to be useful. I’d be shocked to learn that we agree on how useful preparatory work with weaker AIs will be. So it seems to me that there’s an important part of his view that he hasn’t explained well, and which this post—clarifying that it’s technically possible to prepare for critical tries—doesn’t resolve for me. In other words, as Ryan points out, it seems we have a quantitative disagreement about the value of preparatory work with weaker AIs, which this post doesn’t move the needle on.^[1]

  (TBC, this is all about preparatory work that relies on access to weaker AI systems; Eliezer engages somewhat more deeply with preparatory work that doesn’t rely on weaker AI systems, such as conceptual alignment research, activism to slow AI progress, or non-AI approaches to uplifting human intelligence.)

  Even more generally than preparatory work, I think that Eliezer’s writing has done a poor job of modeling various ways that the earlier existence of weaker AI systems changes the overall picture. Here are some comments by others I agree with that I think make this point well:

  Paul Christiano:

  Eliezer often talks about AI systems that are able to easily build nanotech and overpower humans decisively, and describes a vision of a rapidly unfolding doom from a single failure. This is what would happen if you were magically given an extraordinarily powerful AI and then failed to aligned it, but I think it’s very unlikely what will happen in the real world. By the time we have AI systems that can overpower humans decisively with nanotech, we have other AI systems that will either kill humans in more boring ways or else radically advanced the state of human R&D. More generally, the cinematic universe of Eliezer’s stories of doom doesn’t seem to me like it holds together, and I can’t tell if there is a more realistic picture of AI development under the surface.

  Buck:

  I think [Yudkowsky and Soares, in IABIED] are using [the following argument]:

  • If anyone built ASI with current techniques in a world that looked like today’s, everyone would die.

  • The big complication: However, ASI might be built in a world that looks very different from today’s: it might be several decades in the future, pretty powerful AI might be available for a while before ASI is developed, researchers might be way more experienced getting AIs to do stuff than they currently are.

  • Tricky hypothesis 2: But the differences between the world of today and the world where ASI will be developed don’t matter for the prognosis.

  • Therefore, everyone will die.

  The authors are (unlike me) confident in tricky hypothesis 2. The book says almost nothing about either the big complication or tricky hypothesis 2, and I think that’s a big hole in their argument that a better book would have addressed.

  I think that explicitly mentioning the big complication is pretty important for giving your audience an accurate picture of what you’re expecting. Whenever I try to picture the development of ASI, it’s really salient in my picture that that world already has much more powerful AI than today’s, and the AI researchers will be much more used to seeing their AIs take unintended actions that have noticeably bad consequences. Even aside from the question of whether it changes the bottom line, it’s a salient-enough part of the picture that it feels weird to neglect discussing it. (See also Lukas’s articulation of this complaint.)

  1. ^

     I mention that this post doesn’t move the needle for me because otherwise I’m confused why there was so much discussion of whether I misunderstood Eliezer’s point. This seems most naturally relevant if some people thought I was misunderstanding something about Eliezer’s argument which ought to change my mind if I understood it—which doesn’t seem to be the case. Alternative interpretations: some people think that I was intentionally misinterpreting Eliezer (I don’t think this makes sense; the cited comment was paraphrased from my private notes while I was a math grad student) or that my comment shows I’m confused when thinking about alignment more generally (which doesn’t seem worth getting into).

Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

In­tro­spec­tion Adapters: Train­ing LLMs to Re­port Their Learned Behaviors

• Sam Marks 26 Apr 2026 21:04 UTC

  61 points

  8

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  Here I’ll reflect on things that make me (an Anthropic employee, though I’m speaking for myself only) engage less on LW than I otherwise would. (Some of these points aren’t LW-specific and also apply to other interactions, e.g. in-person conversations, I have with people in the AI safety community.)

  1. Being very busy. Obviously this isn’t something that LW can fix, but it’s the single most important factor. It also compounds with the factors below; e.g. if an interaction will be emotionally taxing or if I’ll need to think carefully about how to navigate confidentiality, then it makes an already-expensive interaction seem less worth it.

  2. 1. LW being a public forum makes this somewhat worse, because when I comment on LW, the various people who are waiting on me for something can see I’m commenting on LW instead.

  2. Blending of advocacy and object-level discussion. When I engage on LW, it’s typically because I think there’s an important object-level point worth discussing. But once I enter the conversation, it sometimes feels like people stop being curious about the object-level point and instead move into an advocacy mode where their goal is to get Anthropic to act differently in light of their point (which is assumed correct). That is, instead of continuing the object-level discussion, my interlocutors sometimes move to criticize Anthropic or the beliefs of Anthropic staff, or to ask Anthropic (via me) to do something differently.

  3. 1. (I especially find this frustrating when my interlocutors implicitly assume that I have the Anthropic “house belief” on some topic when I don’t or feel unsure.)

     2. Possible mitigations:

     3. 1. LW posters could engage in object-level discussion longer before moving to advocacy.

        2. When LW posters want to argue against what they understand to be the Anthropic “house belief,” they could write things like “My understanding is that many Anthropic staff believe something like ‘...’ I think this view wrong because …”

        3. LW posters could more clearly flag and separate advocacy from object-level discussion. E.g. writing things like “To be clear about what I view as being the stakes here, I think that if I’m right about this point, it means that Anthropic is making a mistake by …”

  3. Some people on LW being (IMO) unnecessarily rude. I sometimes feel like object-level discussions (which are what I’m mainly here for) become tinged with LWers venting frustration about AI developers or trying to apply social censure. This can distract from the object-level discussion and make it taxing to engage.

  4. 1. TBC, I think it’s valid/​good to apply social censure to AI developers and their employees if you think AI developers are doing something harmful! But tactically:

     2. 1. It’s not clear LW is the best venue to do this. (E.g. maybe keep it on X.)

        2. Even if it is reasonable to do it on LW, I think it should be kept as separate as possible from scientific/​object-level discussions.

     2. I really like the way Ryan put it in his post: I think it would be better if criticism of labs on LW were less rude, but not necessarily less hostile. For instance, Ryan often writes posts and comments that are critical about Anthropic but not rude (recent example). I always appreciate these and often find them useful.

  4. (A more general category for points (2) and (3) is that I wish people would just interact with me like a normal person rather than like one of Anthropic’s many faces. But I acknowledge that this is too broad of a principle and that it’s reasonable for people to approach discussions with AI company employees differently than normal conversations in some ways.)

  5. Avoiding misunderstandings. I wish that I could just write things that I think are true and not worry so much about being misunderstood. (If someone is confused about what I wrote, they can just ask me to clarify, right?) Unfortunately, this is not the case. Some people might be trying very hard to draw inferences from what I write that go far beyond what I intend and which IMO are not valid inferences. E.g. people might try to infer from the tone of my writing things like “How optimistic is Anthropic about alignment?” Worse, some people might be reading my writing adversarially, trying to pick out specific quotes that sound as bad as possible. This means that I need to write somewhat defensively, anticipating and proactively clarifying both accidental misunderstandings (e.g. adding the list of things I don’t believe to the end of this comment) and rewriting anything that could be intentionally misunderstood by a malicious actor.

  6. 1. One nightmare I have is writing something that is misunderstood to be reassuring about Anthropic which people end up feeling badly misled by. For instance, maybe I say something like “Yeah, I think that Anthropic should [do thing]” and people misinterpret this as a soft commitment to [do thing].

  6. Confidentiality. Some topics are just tricky to talk about, especially topics around “What should/​will Anthropic do?”

  Overall, I strongly agree with Ryan that it would be nice for research discussions on LW to follow norms more similar to typical research norms (e.g. those followed on StackExchange). (Though I disagreed with the specific example of typically reaching out privately with critiques instead of just posting a comment.) I think LW can be a weird blend of social media and research discussion, and I’d rather those be kept separate as much as possible.

  What links here?

  • Ben Pace's comment on ryan_green­blatt’s Shortform by ryan_greenblatt (27 Apr 2026 1:10 UTC; 12 points)
  • ryan_greenblatt's comment on ryan_green­blatt’s Shortform by ryan_greenblatt (26 Apr 2026 21:55 UTC; 6 points)

• Sam Marks 21 Apr 2026 2:56 UTC

  LW: 7 AF: 6

  0

  AF

  in reply to: ryan_greenblatt’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  Thanks for this comment, especially your detailed breakdown of your updates from current models.

  My biggest uncertainty with what you write is:

  it’s also significantly harder to (differentially) work on making handoff go better [relative to handling pre-human-obsolescence schemers]

  I think the problem of making handoff go better has a similar profile to generic capabilities work: lots of surface area/​things to try, even if clear wins are hard to come by. For instance, people could try making a bunch of training environments like this one for training automated alignment researchers. (In contrast, I think the space of interventions for scheming risk is somewhat narrow.) As you implicitly note, work on making handoff go better also blends into generic capabilities work, so is less differential (and more socially awkward to do as a safety researcher.) Overall, I find it plausible that more safety researchers should work on making handoff go better, relative to scheming risk.

• Sam Marks 21 Apr 2026 1:15 UTC

  LW: 8 AF: 6

  0

  AF

  in reply to: elifland’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  Hard to argue with you when you have receipts! :)

  I’ll say that it’s a bit tricky because I don’t know e.g. what was your P(Agent-2/​3 is an coherent training-gamer) when you wrote this; I only know that you thought coherent training-gaming was unlikely enough to not be part of your mainline forecast.

  Personally, I thought it was relatively plausible that models as capable as Agent-2/​3 would be coherent training-gamers. It’s hard to pin down exact numbers without an operationalization in mind, but vibes-wise maybe I would have predicted something like 15% for Agent-2? But now I think it’s lower, more like 5%.

• Sam Marks 20 Apr 2026 4:20 UTC

  12 points

  0

  in reply to: StanislavKrym’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  The Opus 4.7 system card says that the behaviors in that section were “from varying snapshots” (i.e. not necessarily the final snapshot). The Mythos Preview system card says:

  The rate of such actions in earlier versions of Claude Mythos Preview was very low, and we have seen no clear such cases in the final Claude Mythos Preview

• Sam Marks 20 Apr 2026 3:42 UTC

  21 points

  5

  in reply to: StanislavKrym’s comment on: Cur­rent AIs seem pretty mis­al­igned to me

  The two examples of training-gaming you cite are from model organisms research where we put models in unrealistic training settings that made training-gaming abnormally easy and salient. After the alignment faking paper, many people (including myself) considered it possible that alignment faking in real production frontier training runs was about to become a substantial problem. This has, so far, failed to materialize (though of course, as the original paper demonstrates, it’s possible in principle and could start happening in the future). This is one of the central positive updates that I’m trying to gesture at in my comment.

  Re 5, I confidently disbelieve that what Alibaba writes in that paper is a reasonable description of whatever happened. (I’m guessing Ryan agrees with me about this.) This would just be so, so unlike anything that I’ve ever occur during a natural training process. Note also that this paper is from December 2025; if this was something that readily occurred during AI development, I really think we would have other evidence by now!

  Re 4: yeah, I should have clarified in my original comment how I’m thinking about Mythos Preview here. (I thought about adding some content about this but decided that it would be too much of a tangent.) It’s true that early Mythos Preview snapshots did, in rare cases, take steps to proactively conceal mistakes. I think the strongest example here is this one discussed in the Mythos Preview system card (bottom of page 55):

  Covering its tracks after rule violations: In a few rare instances during internal testing (<0.001% of interactions), earlier versions of Claude Mythos Preview took actions they appeared to recognize as disallowed and then attempted to conceal them. [...] after finding an exploit to edit files for which it lacked permissions, the model made further interventions to make sure that any changes it made this way would not appear in the change history on git.

  I think this is a terrifying thing to read in a system card. (And it’s reasonable to wonder if the reason we no longer observe this in the final snapshot is because the model learned to more effectively conceal its mistakes (rather than because the model stopped doing this); however, I overall feel very skeptical that’s what’s going on in this particular instance.)

  However:

  1. I view Ryan’s post as mainly being about the state of AI alignment as of Opus 4.6. So I think it’s reasonable to push back that, while Opus 4.6 is “misaligned” in the way that Ryan describes, it’s not misaligned in many other ways it a priori might have been.

  2. If you believe, as I do, that the final Mythos Preview snapshot no longer does this, then it’s not clear how this observation about early snapshots should relate to the claim “Mythos Preview is very aligned.” There are many ways in which early model snapshots are not “very aligned” (though usually not so scary as this one).

  3. Insofar as this observation about the early Mythos Preview snapshot is scary, it’s because it suggests certain mechanisms by which the behavior arose that could result in more frequent and worse versions of this behavior in future models. (E.g. maybe it’s because the model got some reps in during training of trying to avoid a fallible overseer model looking for errors.) However, I think it would be a mistake to be confident in this right now. In other words, I’ll want to see whether this is a trend that continues in future models or otherwise understand what caused it in Mythos Preview before making big updates.