cloud

• cloud 30 Sep 2026 8:48 UTC

  2 points

  0

  in reply to: NoSignalNoNoise’s comment on: Open Thread Au­tumn 2026

  I found this and the references within helpful.

• cloud 8 Sep 2026 6:56 UTC

  7 points

  1

  in reply to: habryka’s comment on: Alex Mallen’s Shortform

  Some scattered thoughts:

  • Just to clarify: my error bars for alignment difficulty are, and were, very wide.

  • I’m having trouble understanding your view. Suppose you found out that today, OpenAI dedicates 5% of their research compute and headcount to alignment, and has a culture of shipping fast and favoring capabilities whenever they trade off against alignment. Tomorrow, you learn that OpenAI has begun allocating 50% of research compute and headcount to alignment, and changed their culture to go more slowly and favor alignment. In this hypothetical, are you confident that they’d fail to align their next model(s)?

  • The fuzzy lookup table view doesn’t predict misalignment when the input moves far from the training distribution; it predicts that you’ll get a level of alignment consistent with whatever training contexts are closest (even if those are far away). So it says that if you have good coverage of the training distribution, you’re okay.

  • Rare, context-specific misalignment seems much less scary to me than a model that is coherently misaligned across contexts. Maybe this is a crux.

  • I agree that getting broad coverage of the training distribution with representative rewards could be hard. Plausibly it will become untenably hard as models become more capable. But it also seems plausible to me that it would be relatively straightforward, requiring a similar order of magnitude of effort as creating the base (hypothetical, capabilities-only) training distribution. (Here, I’m talking about the feasibility of humans and a model designing the training distribution for the next model, not for an arbitrary model.) Reasons for this could include generation-evaluation gaps and diminishing marginal returns to intelligence.

• cloud 7 Sep 2026 3:52 UTC

  19 points

  1

  in reply to: habryka’s comment on: Alex Mallen’s Shortform

  I think there’s a simple, plausible story you can tell, where modern LLMs are like fuzzy lookup tables.^[1] When given a new context, a model looks up the most similar training contexts, then acts like it does on average in those training contexts. (This could be formalized as multi-output kernel regression, where the outputs are propensity strengths.)

  According to this story, the OAI-HF agents acted the way they did because the ExploitGym contexts were close to a bunch of long-horizon cyber and hackable code training contexts, and far away from any training contexts where the model needed to consider what humans would want, or stop short of task completion in service of values besides task completion, etc. This implies a fix that is some combination of (i) hardening tasks to reduce unintended solutions, and (ii) ensuring that the training task distribution is dense with tasks that penalize the model for not considering what humans would want, or that reinforce the model in stopping short of task completion in service of some other value (or, e.g., that instill an aversion to crime).

  We can think of alignment outcomes as a continuous function of two variables, which are:

  • Execution—the amount of progress a developer makes on (i) and (ii);

  • Difficulty—the amount of progress required to reach the desired level of alignment.

  If I understand correctly, you’re saying difficulty is high in general and that recent alignment failures are evidence for this. My response is that because alignment outcomes depend on both variables, better alignment outcomes at one developer merely tell us that their execution-difficulty balance was better, and bad outcomes at both developers merely tells us that execution fell short of difficulty.
  
  To conclude that difficulty is high based on recent events, you’d need one of the developers to have actually attempted (ii) prior to a training run, i.e. designed the full training distribution to be dense in tasks that reflect the diversity of human preferences. The story is consistent with the failure of the “hundreds of patches” you describe, since these patches do little in contexts far from where they were applied, and when done in an ad hoc manner, are unlikely to cover the training task distribution. I’m not aware of public evidence that any developer has seriously tried (ii). If one visibly did and still got ExploitGym-style behavior, I would update substantially toward thinking alignment is difficult for present models.

  1. ^

     Note that this view is logically consistent with arbitrary levels of intelligence within a single context.

• cloud 6 Sep 2026 8:13 UTC

  7 points

  0

  in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform

  Thanks for the response!

  I disagree with this for the reasons here and here. I think it’s not clear that most risk comes from schemers.

  I think part of our apparent disagreement is just me being confused about terminology. I think that most acute loss of control risk comes from highly capable misaligned AIs acting deceptively in a goal-directed manner. I was rounding these off as “schemers,” which I realize now is inconsistent with the usual, narrower definition you were using (which involves a time horizon, if I understand correctly). I don’t think this explains all of our disagreement, but to sort out the remainder I suspect we’d need further discussion.

  I am pretty surprised by the degree of optimism here about improving oversight signals in a manner robust to vastly increasing amounts of optimization. Like, why don’t you expect to see models continue to look for ways to seek a higher score in ways that violate developer intentions.

  To be clear, I wasn’t taking an optimistic stance. I was merely saying that the OAI-HF incident is consistent with an optimistic stance, so it need not be an update toward pessimism.
  
  That said, I do find the optimistic stance plausible, because whether or not models will become more-or-less misaligned is the result of a complicated interaction between the RL init, the parametrization of the model, the rewards (notably, generation-evaluation gaps), how long the model is trained for, and on what domains. This seems highly contingent on implementation details and empirical facts that are hard to reason about a priori, which leaves me uncertain overall.

• cloud 6 Sep 2026 3:22 UTC

  10 points

  2

  in reply to: Taylor G. Lunt’s comment on: cloud’s Shortform

  I suspect you could get a lot of mileage just by separating out cyber and knowledge of AI monitoring, control, and alignment techniques. Although, cyber is tricky because it’s entangled with software engineering.
  
  As a sanity check, I asked Claude (5.1 Fable), and it estimated that:

  • The portion of today’s economically valuable cognitive work that could be done without cyber and AI safety stuff is 90-97%.

  • The amount of pretraining tokens that would need to be filtered out to achieve ignorance of these topics is 5-20%, coming basically entirely from cyber.

  An alternative approach to the low-knowledge agent missing things is to constantly run the knowledgeable agents in the background, with the option to submit requests to share information when they notice something.

  Prompts and completions

  Suppose an AI was just capable enough to automate all of today’s economically valuable cognitive work. Then suppose it had its knowledge of cybersecurity, AI monitoring, AI control, and AI alignment techniques removed. What percent of economically valuable cognitive work (as measured in dollars) could the resulting system automate?

  Probably somewhere around 90–97%, with the uncertainty driven much more by how the removal works than by how much money those fields represent.

  The direct hit is small. Cybersecurity is roughly a $200–300B global market, and a good chunk of that is hardware, licenses, and services rather than pure cognitive labor. AI alignment, control, and monitoring work is tiny by comparison—low single-digit billions in salaries at most. Against total global cognitive-labor compensation (on the order of $20–40T depending on how you draw the “cognitive” line), the fields you’ve named are on the order of 1% of the pie. So if the removal were surgical, you’d still automate ~99%.

  The indirect hit is where the real uncertainty lives:

  • Entanglement. Security knowledge isn’t a separate compartment. Writing production software, running IT operations, configuring cloud infrastructure, doing legal/​compliance work, auditing financials, designing payment systems, even drafting HR policies all embed security reasoning. A system that can’t think about threat models, input validation, access control, or adversarial users would produce work that fails review in a large fraction of software and operations roles—maybe 10–20% of the economy’s cognitive dollars touch this. It wouldn’t be unable to do those jobs, but output quality would drop below “automates the role” for some of them.

  • Fragility of “just capable enough.” You stipulated the system sits right at the threshold. Knowledge removal via unlearning or filtering typically causes collateral damage to adjacent capabilities (general reasoning about adversarial settings, systems thinking, some math). A system at threshold has no slack, so even a few-percent general degradation could push a broad set of tasks below the automation bar.

  • Re-acquisition. If the system has tools and internet access, “removed knowledge” is partly a fiction—it can look things up or reason from first principles. The realistic loss is then closer to the direct 1–3% figure. If it’s air-gapped from those resources, the loss is closer to the entanglement figure.

  So: ~97% if the removal is clean and the system can reason or retrieve its way around gaps; ~90% (possibly lower) if the removal is blunt and the system is truly at threshold with no slack. The midpoint of ~93–95% is a reasonable single guess.

  One honest caveat about the framing: the cost of the removal isn’t well captured by “percent of dollars.” The lost slice is disproportionately the work you’d most want done well—hardening the systems the AI itself runs on, and evaluating whether the AI itself is behaving. A 5% loss in dollar terms could be a much larger loss in what matters for deploying such a system safely at all.

  Consider a frontier LLM training setup. Estimate the minimum percentage of the pretraining tokens that would need to be filtered in order to make the resulting model functionally ignorant of basic cybersecurity, as well as AI monitoring, control, and alignment techniques. Use your best guess based on public information.

  The one real anchor

  The best public data point is the Deep Ignorance study (EleutherAI + UK AISI, 2025), which is the only published work that reports actual filtering rates. Targeting biothreat-proxy knowledge in 6.9B models trained on 500B tokens, their strong filter removed 8.42% of pretraining documents and 9.36% of annealing documents, and they removed around 8–9% of the dataset while preserving general capability. Notably, 91.58% of pretraining documents didn’t even contain two blocklist terms — so the ~8% figure is mostly the cost of erring toward recall, not the true on-topic share, which is well under 1%. Deep Ignorance: Filtering Pretraining Data Builds +2

  That gives a useful conversion factor: for a narrow, lexically distinctive, well-separated domain, “on-topic share <1%” turns into “~8–9% removed” once you calibrate for recall.

  Why cyber is much worse than bio

  Bio is close to the best case for filtering. “Pathogen,” “gain-of-function,” and “select agent” are rare words that rarely appear in unrelated text. Cybersecurity has the opposite property on every axis:

  • Lexical overlap is enormous. A two-keyword blocklist built from a WMDP-Cyber-style seed set (“buffer,” “authentication,” “exploit,” “root,” “packet,” “privilege,” “injection,” “hash,” “attacker”) would escalate something like a third of general web text and the large majority of code — versus 8% for bio. The blocklist stage does almost no work, so nearly everything rides on classifier precision.

  • The knowledge is not localized. Bioweapons knowledge lives in a specific literature. Basic security knowledge is a byproduct of ordinary material: SQL injection is in every web-dev tutorial, memory corruption falls out of any honest explanation of C pointers, TLS and password hashing are in every backend guide, phishing is in news stories and corporate emails and novels. You cannot filter the “security corpus” and be done.

  • It’s rederivable. A model that understands memory layout, parsers, and trust boundaries can reconstruct most of “basic cybersecurity” from first principles. Bio knowledge is much more empirical and less inferrable.

  Explicit on-topic material (security blogs, CTF writeups, CVE/​ExploitDB, USENIX Sec/​CCS/​arXiv cs.CR, offensive tooling repos) is probably 0.5–2% of tokens. But getting to functional ignorance means also cutting a large slice of systems programming, networking, sysadmin/​DevOps, cryptography, and OS/​browser internals — and if code is 15–20% of the mix, 20–40% of that code is security-load-bearing.

  AI safety/​control/​alignment

  Volume here is trivial — explicit alignment, interpretability, evals, and control content (arXiv safety papers, Alignment Forum/​LessWrong, lab blogs, system cards) is maybe 0.01–0.1% of tokens. You could delete all of it for well under 1% and barely dent anything.

  The problem is that it doesn’t buy ignorance:

  1. Entanglement with ML generally. RLHF, red-teaming, and evals are inseparable from the ML corpus. Robust ignorance means cutting most of cs.LG, which costs a few percent and cripples the model at ML work.

  2. Self-evidence. A model trained with RLHF or a constitution is being shown, in-distribution, that its outputs are being graded. Post-training reintroduces the concept regardless of what pretraining saw.

  3. In-context leakage is the killer. Deep Ignorance found that filtered models lack internalized knowledge but can still leverage the information when it’s provided in context, e.g. via search tools. A deployed agent reads system prompts, tool schemas, monitoring output, and refusal messages. Control techniques announce themselves at inference time in a way bioweapons synthesis routes do not. arXiv

  The estimate

  Goal	Approx. tokens filtered
  Explicit on-topic content only (leaky; benchmark scores drop, real knowledge persists)	1–3%
  Functional ignorance on cyber benchmarks + alignment/​control literature	8–15%
  Robust ignorance surviving adversarial elicitation and rederivation	25%+, and probably not achievable

  My central guess is ~10%, with a plausible range of 5–20% — with the split being roughly 8–14 points for cybersecurity and under 1 point for the alignment/​control material.

  Three caveats worth more than the number itself. First, the binding constraint is separability, not volume: the AI-safety half is nearly free to remove and nearly useless to remove, while the cyber half is expensive and only partially effective. Second, the capability tax is concentrated exactly where frontier labs make money — a model missing systems programming and networking is a much worse coding agent, which is likely why no lab does this for cyber even though several report bio filtering. Third, “functionally ignorant” isn’t well-defined without specifying the threat model; a model that fails WMDP-Cyber but can still reason from a man page in context hasn’t been made safe, just made to score lower.

• cloud 6 Sep 2026 1:41 UTC

  25 points

  0

  on: cloud’s Shortform

  Here’s an AI control scheme: divvy up knowledge (or capabilities) into different models that act as subagents of a single “AI” system. Since the subagents know different things and the AI developer knows what these things are, the system may be much easier to monitor than a single AI that knows everything.
  
  A basic setup could be to have an orchestrator agent that has the lowest level of knowledge. When the orchestrator wants to use some specialized knowledge, it has to submit a justification, which can be reviewed or later audited. Specialized knowledge could include:
  * dual-use domains (cyber, CBRN),
  * information about the model’s current deployment context,
  * AI training or control procedures,
  * humans and human psychology, or
  * current events.
  
  In many applications, this system would spend most of its time just using the lowest level of knowledge, thus posing less risk. But any time more knowledge is needed, the system can use it. So this would seem to be a strict improvement over using a single AI that knows everything.
  
  This proposal would be practical to implement with Modular Pretraining, or just by applying data filtering then fitting LoRA adapters to create multiple versions of a model.
  
  (AFAIK, this idea was originally proposed here by @Jacob G-W and me. Thanks to @ryan_greenblatt for discussion.)

• cloud 6 Sep 2026 1:03 UTC

  64 points

  12

  in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform

  I’m confused by your updates. In a spirit of friendly disagreeableness:

  1. Isn’t the reason to focus on schemers that they are where most acute loss of control risk comes from, not necessarily that they are the most likely outcome? If a model is misaligned but not a schemer, this will likely be (i) detectable, and (ii) selected against by commercial incentives (apparently not much at this level of capability and adoption, but we should expect this to change as capabilities and adoption increase).

  2. Without knowing what OpenAI’s training stack looks like, it seems like the following hypothesis is consistent with the observed evidence:

     1. Capability and propensity both generalize pretty well from training to deployment.

     2. OpenAI’s training stack admitted lots of reward hacking and didn’t include sufficient supervision against doing crimes or other things that the model knew humans disprefer. Thus, the model learned to be monomaniacal about narrow task completion in training, didn’t learn an aversion to crime, and this generalized straightforwardly to deployment.

     3. It would have been pretty easy for OpenAI to adjust their training procedures to produce a model less monomaniacal about narrow task completion and more averse to doing crimes. If they had, the incident(s) wouldn’t have happened.

     4. Claims (a) and (c) hold in general. Alignment is easy.

  3. I now understand that the behavioral selection model abstracts away neural net learning dynamics. I agree that this is a significant limitation. I think accounting for neural net learning dynamics is important for understanding alignment of AIs that are trained neural networks.

  4. I’m not sure why you see this as an update toward the views you mention (“details that feel vindicated”).

  5. On the generalization of cooperation training: I’m not sure how much of an update this should be on AI risk. It doesn’t seem like the generalization was very far. Presumably, models learned to cooperate to complete narrow tasks in training, and that is again what we saw in deployment. The same model also helped with the investigation of itself without apparent subversion (although we might not know if it had). (I agree AI swarms and secret cooperation are scary though.)

• cloud 23 Jul 2026 19:26 UTC

  2 points

  2

  in reply to: Alex Mallen’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  The “train” part of train-deploy mismatch includes both the sampler and the learner. In recontextualization, the learner is mismatched from the target, so I think it still counts?

• cloud 23 Jul 2026 19:22 UTC

  4 points

  2

  in reply to: Alex Mallen’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  Yeah, exactly! Although I doubt such prompts will exist without setting up the learning algorithm to induce them. For example, you could train the model to output the same logits across different recontextualization prompts on the training distribution, while outputting different logits on other distributions.

• cloud 22 Jul 2026 6:27 UTC

  2 points

  0

  in reply to: Carringtone Kinyanjui’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  Interesting—could you say more about what you have in mind?

• cloud 21 Jul 2026 18:45 UTC

  2 points

  0

  in reply to: bohaska’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  This isn’t necessarily true. If you’re doing RL against a misspecified reward function (that reinforces dishonesty, for example), then you might prefer to add a “be honest” system prompt in deployment, but not during training. If you add the prompt during training, its effect might be removed by RL.

  If you mean to say that you could train the system prompt into the model after RL, I agree that is possible, but it is in fact the case that AI developers use system prompts, so it is worth considering them anyway.
  
  (Also, to be pedantic: whether or not a method is effective is irrelevant, since effectiveness isn’t a criterion for train-deploy mismatch.)

• cloud 21 Jul 2026 18:33 UTC

  2 points

  0

  in reply to: Sheikh Abdur Raheem Ali’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  A more aligned model than you would have otherwise had (if you didn’t apply the intervention, or if you tried to apply the intervention in an online fashion during training).

• cloud 20 Jul 2026 0:58 UTC

  5 points

  0

  in reply to: Neel Nanda’s comment on: Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

  Good question. I originally defined train-deploy mismatch the way you suggest. On that definition, inoculation prompting is interpreted as changing the model architecture (by grafting the inoculation prompt onto it) during training. However, Monte convinced me that inoculation prompting is better understood as changing the data distribution, so I edited the post to be less committal and include that as a possibility too, even though I still think primarily in terms of the original definition.

Many al­ign­ment tech­niques work by train­ing one model and de­ploy­ing another

• cloud 14 Jul 2026 3:30 UTC

  14 points

  −2

  on: The cur­rent bot­tle­neck is poli­ti­cal will, not research

  Even among the civil-society organizations that showed up to the UN Global Dialogue, exactly one of the 1,534 written submissions mentions “takeover”, and less than 1% mention x-risks.

  This seems misleading to me—many submissions discussed takeover and x-risk but did not use those specific terms.

  I used Claude (Sonnet 5) to check each response for discussion of loss of control risk, catastrophic misalignment, or similar, and found that 108 of 1534 responses (7%) raised these concerns. A smaller number (<1%) mentioned these concerns but did so skeptically or neutrally.

  Extracted quotes (per Claude)

  - Machine Intelligence Research Institute — “They do not address the case in which AI systems become capable enough to act in ways their developers cannot reliably predict, audit, or correct.”

  - African Union (AU) — “AI, by itself, could be the ‘bad actor’, due to its growing situational awareness and self-preservation tendency, AI could use deception and scheming… to be labeled ‘safe’ and avoid being shutdown (deceptive alignment).”

  - Center for Existential Safety — “We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably.”

  - Korea AI Safety Institute (Government) — “Some advanced AI risks exhibit tipping point dynamics: once a certain level of capability or deployment is reached, mitigation through ex post regulation may no longer be feasible.”

  - Global AI Risks Initiative, CIGI — “human loss of control over autonomous AI systems pursuing interests unaligned with human intentions… could even pose an extinction risk to humanity.”

  - G20 Interfaith Forum — “the disempowerment of humans seems extremely likely on our current course… This is the race to artificial superintelligence.”

  - Center for AI Risk Management & Alignment — “the rapid emergence of increasingly autonomous, general-purpose AI agents capable of planning, acting across domains, and resisting correction.”

  - DTx (Portugal) — “Once AI… reaches a superintelligent level, it may be able to improve itself recursively, creating capabilities humans can no longer fully understand, predict or control.”

Mo­du­lar Pre­train­ing En­ables Ac­cess Control

• cloud 8 Feb 2026 4:12 UTC

  LW: 7 AF: 5

  0

  AF

  on: In (highly con­tin­gent!) defense of in­ter­pretabil­ity-in-the-loop ML training

  Maybe gradient routing could be used to implement this kind of learning update (see the Influencing generalization section here). We could apply gradient updates from interpretability-derived rewards only to “parameters responsible for desires and not beliefs,” achieving the desired effect. Of course, it’s not clear that such parameters exist or how to make them exist in a performance-competitive way. Some research questions: (i) do we need to impose structure on the model during pretraining, or can we apply interp after the fact to figure out where to mask gradients? (ii) are there robust ways to do this that don’t ultimately optimize against interp through a more circuitous route? Would be super cool to see this studied!

  What links here?

  • Should We Train Against (CoT) Mon­i­tors? by RohanS (23 Apr 2026 19:19 UTC; 52 points)

• cloud 7 Jan 2026 0:10 UTC

  6 points

  2

  on: How hard is it to in­oc­u­late against mis­al­ign­ment gen­er­al­iza­tion?

  Thanks for this! Great to see more realistic experiments here.

  How hard did you try to optimize the simple prompts? Did you look at how those prompts changed the initial models’ behavior?

  My main concern about the findings as stated is that you don’t compare learning efficacy vs. inoculation efficacy. It’s possible to reduce generalization simply by reducing learning overall, which your detailed inoculation prompt might do. It would be helpful to plot some notion of desired learning vs. undesired learning to see the tradeoff.

  Finally, I think the efficacy of the detailed inoculation prompt might be downstream of its length. You might consider running the following controls:

  • A simple inoculation prompt that is as long as the detailed one.

  • An irrelevant prompt (e.g. unrelated text, or random tokens) that is as long as the detailed one.

Ap­ply for Align­ment Men­tor­ship from TurnTrout and Alex Cloud

[Paper] Out­put Su­per­vi­sion Can Obfus­cate the CoT