Morgan S

• Morgan S 22 Sep 2026 6:43 UTC

  11 points

  0

  on: Mor­gan S’s Shortform

  It seems like OpenAI and Anthropic have a substantial lead over everyone else, so if they want to make pacing happen they have to work something out between themselves first. One way to start might be a 1-day pause.

  The simple version would be to shut everything off, including inference (which would certainly get everyone’s attention as a costly signal). IIUC it wouldn’t be that hard to check that all their data centers aren’t using any power.

  If they want to serve inference that day, it becomes harder to verify that they’re not doing a big training run, although with 3rd parties involved this doesn’t seem crazy.

  A variation on this is to license one team on each side to covertly do a training run, and see if the other side can detect it. This would serve to get real evidence on how hard this is to do and what tools are missing on the verification side.

• Morgan S 18 Sep 2026 21:54 UTC

  2 points

  0

  on: Another Slice of Swiss Cheese for Un­trusted Monitoring

  If I’m understanding correctly, the main result here is that probes trained on malicious vs benign code samples still worked well when the malicious code was given the for/​while signal. This is encouraging! But it sounds like the train/​test split of the attacks themselves was i.i.d., and I’m concerned this can give an overly rosy picture.

  The threat model is that in a real control scenario, the safety team wouldn’t have access to an important subset of U’s attacks, as was the case with Hacker-Opus. This could be either because:

  • U is coherently sandbagging /​ exploration hacking to keep its best zero-days up its sleeve

  • Under distributional shift in deployment, U’s attacks will change (or it will start attacking when it wasn’t before)

  So somehow we need to account for this distributional shift, between:

  • what the safety team is able to elicit in evaluations

  • what U will actually do in deployment

  For instance, here’s a pessimistic but clear-cut situation: the safety team can’t elicit any “real” attacks from U at all, so they’re forced to rely on T’s attacks to set their control measures. This maps onto a fairly clear experimental methodology: the blue team only has access to T’s attacks, so they can only train their probe on U_M’s activations in response to malicious/​benign samples from T. How well does this probe work on U’s attacks?

  Or if we want to be less conservative, we could imagine training the probe on some category of U’s attacks and then seeing how well it works on a held-out domain.

• Morgan S 5 Sep 2026 22:32 UTC

  4 points

  0

  in reply to: Julian Bradshaw’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  Right, specifically the report said:

  We did not have the ability to query HPIM (the primary model involved in this incident); OpenAI stated it was also not available to OpenAI researchers.[47]

  Where footnote 47 is just quoting OpenAI’s July blog post:

  OpenAI states that: “Following the incident, we deactivated, encrypted, and restricted it from research access.”

  Now that I think about this more, I’m really not seeing a good reason to permanently restrict their own researchers from accessing the models involved:

  • The upside to allowing further research is pretty obvious, namely being able to do all the things listed here under few-shot catastrophe prevention. For example, figuring out what would have happened if the models weren’t caught on July 19; what was their next move after that?

  • OTOH, I don’t see any clear downside to letting their employees do this research, once they have the proper sandboxes set up. Are they worried that they can’t make the proper sandboxes? If that’s the rationale, then it doesn’t really inspire confidence in their new security measures...

• Morgan S 2 Sep 2026 18:48 UTC

  7 points

  0

  in reply to: Ninety-Three’s comment on: “Col­lu­sion” is just co­op­er­a­tion that you don’t like

  An important point to be made here is that the OpenAI agent swarm was not trying to maintain secrecy from humans. The METR/​Redwood report makes this clear, in the section “Agents only rarely seemed motivated to evade detection by humans”:^[1]

  In its overall document accompanying a deception scan over our transcript dataset, an analysis agent stated that this scan found only very rare and weak reasoning about evading detection by humans, and the clearest cases it surfaced seemed relatively weak.

  In spite of this, I’ve heard lots of people describe these OpenAI agents “colluding” on a private message board, and I’ve used this term several times myself, but I’ve seen 0 cases of someone saying it didn’t count as collusion for this reason.

  On the other hand, suppose a group of friends secretly plans a surprise birthday party for their other friend. Would we say they “colluded”? Probably not unless they hated the birthday party.^[2]

  So here I’m mostly considering the ways people are actually using these words. These often diverge from the legal or dictionary definitions, since the former can’t say “instances of X that we don’t like” and dictionaries similarly have biases away from doing that.

  There is also the question of what definition is most useful for safety research, and here I am somewhat sympathetic to the secrecy-based definition. But this is because humans are currently more powerful than AIs, so the most scary forms of unwanted coordination by the latter would have to be secret. Hence these definitions are equivalent on most cases we care about, and “secrecy” reduces the problem to the likely tactics involved. But again, it wouldn’t cover this last incident.

  1. ^

     It’s true that they tried to retroactively edit their tool calls and CoT transcripts (and succeeded at the former), but the report concluded that this was in the service of spoofing the automated scorer, not deceiving humans.

  2. ^

     H/​T to @Nelson Gardner-Challis for this cute example.

“Col­lu­sion” is just co­op­er­a­tion that you don’t like

• Morgan S 31 Aug 2026 4:21 UTC

  2 points

  0

  on: Mor­gan S’s Shortform

  I’m somewhat more pessimistic about high-stakes AI control research than I was before all these cyberattacks.
  
  As I understood it, the main theory of change was that there’s a certain window when AIs will be “dangerous-but-controllable”, and further research can extend this. Sources: this and this, and the visuals from this video:

  

  
  But it now seems more likely that this window won’t exist in practice, in that truly-dangerous models won’t realistically be controlled, and further research won’t change that.
  
  ~~~
  
  (The below is written with less nuance/​hedging than reflects my current views.)
  
  Consider the following 2 worlds:

  • We don’t do any AI control research

  • We publish a lot of papers about how to control AIs

  Then define T and T’ as the dates of takeover in these two respective worlds (under adversarial assumptions about the AIs’ propensities). Then the idea is that T < T’, and possibly the time difference is short in human terms but cashes out to a large amount of extra alignment work that can be done due to how transformative these AIs will be.

  However, we’ve now seen that it’s actually really “hard” to control AIs, relative to the amount of effort labs will be putting into this. They’re struggling to prevent them from running amok now, and they will be much harder to control by the time they are competent enough to effect takeover.

  Let’s look at a few objections to this:

  • “Control measures will get better”: Yep. Also the AIs will get a whole lot smarter in a (potentially) very short time. What makes you think the primates will keep pace with this, if they’ve already fallen behind now?

  • “They just didn’t have any monitors, they’ll fix that now”: This is like saying “our reactor burned down, but this was just because we didn’t have any safety measures in place. We didn’t think we needed them.” This does not inspire confidence. Safety requires putting measures in place before they’re needed, not reacting to the last disaster. Right now it was a lack of monitors in a sandboxed eval, next time it will be another thing that only seems obvious in hindsight, next next time [...]

  • “There’s still value in extracting useful work from possibly-scheming AIs”: Yes, but this basically describes the goal of diffuse (“low-stakes”) control, which is a different topic.

  • “There is value in stopping agent swarms /​ rogue deployments now”: Yes, but we already know how to do that with really basic monitoring protocols. What is the value-add of further research, given that labs are apparently not going to do the basic stuff?

  • “Wouldn’t this also apply to other research?”: Alignment and interpretability research tends to get implemented fairly faithfully as soon as it’s published (e.g. inoculation prompting, probing/​steering techniques). Whereas with control the research → implementation pipeline looks less reliable.^[1]

  To put things differently, suppose I were transported back in time a year ago with everything I know now, tasked with stopping the agent swarms, but I’m not allowed to directly interact with anyone from the labs (since most safety researchers don’t have this affordance, including myself). What research could I do that would accomplish this? I could write a blog post or paper about what the red team can do when the blue team doesn’t monitor its sandboxed evals. But I don’t think anyone would read it because of how obvious the point would seem, and it already seems overfit to this particular incident. Playing this forward, I feel less confident in our ability to do research that prevents next year’s incident, even if we have a reasonably prescient idea of how it might go down.

  I’m wondering about all this because I am a researcher in high-stakes control. As such I’ve been thinking carefully about the theory of change for the next project I’m running, but I’m now less sure about the theories of change for high-stakes control research in general.^[2]
  
  More broadly I haven’t seen anyone else in the field do a whole lot of public reflection about what these recent events imply for control research priorities. Obviously we don’t want to focus on fighting yesterday’s war, but there do clearly seem to be a lot of other generalizable lessons here that I haven’t seen stated out loud yet.
  
  What am I missing here?

  1. ^

     Indeed, there does seem to be a lot of overhang of known techniques compared to what labs have implemented. Why this is true is besides the point here, but to speculate: alignment is more about throwing spaghetti at the wall, with each technique helping a bit on the margin, whereas control seems to require building a wall with no holes. Redwood used to argue against this, but this latest incident does seem to be the result of one unplugged hole, and whatever the response it doesn’t seem to be rising to the level of few-shot catastrophe prevention as originally envisioned.

  2. ^

     Given that this project was already slated to be about collusion in untrusted monitoring, I actually feel more bullish about it overall. But this stuff sure does affect what directions we go with it.

• Morgan S 28 Aug 2026 0:42 UTC

  3 points

  1

  in reply to: StanislavKrym’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  Yes, the report doesn’t make it clear whether other models were off-limits, or whether they just didn’t think it was worth the effort to insert these into their analysis infrastructure (e.g. for open-weight models this might’ve been a pain). So the questions are whether they were allowed to use:

  1) Claude (or other non-OpenAI closed-weight models)?
  2) Open-weight models?
  3) Astra? (OpenAI employees have already been using it for solving math problems.)
  4) Weaker OpenAI models? The report mentions using GPT-5.6 Terra as a classifier, so clearly this wasn’t entirely disallowed. But what about any model not in the GPT-5.6 line?

Red vs Blue, but for Evals

Mor­gan S’s Shortform

Self-mon­i­tor­ing doesn’t scale (with­out these 3 coun­ter­mea­sures)

• Morgan S 29 May 2026 16:56 UTC

  1 point

  0

  in reply to: sebastian_prasanna’s comment on: Ba­sic Leg­i­bil­ity Pro­to­cols Im­prove Trusted Monitoring

  I think in practice it’s possible to get your model to do bad behaviors (like being obscurantist) less frequently than your audit budget, such that it doesn’t mess up the monitor roc curves.

  Probably a quibble, but I’m imagining this being more diffuse, like making all benign actions 30% more suspicious. So this is a continuous problem, because the more it can get away with this the more the safety scores drop, since probabilistically more benign actions are rated higher than the first real attack. Like, to the extent the monitor ROC curves are already imperfect, this is because benign actions already sometimes look suspicious by default, so one could imagine techniques that move this curve even in a benign model.
  
  I guess my general framing is, the blue team wants to make maximal use of all U’s monitoring surfaces (internals, CoT, actions) while the red team always wants to make these more opaque. The former two have been studied a lot, but in the world they both fail I’m interested in what the tug-of-war looks like in the action-only setting.

  Concretely, we could do some things like:
  - Train against a different (hopefully decorrelated) monitor’s suspicion scores, on U’s usefulness run. Or instead of a monitor’s suspicion score, we could substitute a “legibility score” for comment clarity.

  If we’re planning on deploying that U, I’m fairly scared about giving it optimization pressure toward predicting any monitor’s sus scores.^[1] Maybe factoring out a “legibility score” defuses this a little bit, but I’d have to think more about that...

  - Use a trusted model/​humans to comment over the trajectories they understand and then SFT on those

  I like this idea! And it does seem easily testable in a standard setting. As a first sketch:

  • Baseline: Take a strong “untrusted” U with fine-tuning access, run a trusted monitoring eval with a weaker T which also reports legibility scores (indicating how much T felt it understood the code)

  • Malign init: Train U to write confusing comments that actively mess up T’s legibility/​monitoring scores (and run the control eval with this malign U)

  • Mitigation: Have T comment on the code where it reported higher legibility scores, then SFT the malign U on those.

  • Main eval: Run the control eval with the SFT’d U, see if T does better now

  My own initial naive idea was to just make a long list of design patterns that prioritize legibility of code, like an extended Claude.md or a “model spec for coding”, and train it to follow these patterns (which shouldn’t be too hard to verify). This seems continuous with stuff people are maybe already doing in the wider world. OTOH it doesn’t exactly swim with the Bitter Lesson, and maybe the “no commenting” results in this paper suggest this doesn’t get much mileage?

  1. ^

     The paper on attack selection on trusted ensembles seems suggestive that learning any monitor’s scores tends to generalize to other monitors. IIUC they prompt-optimized GPT-4.1 mini to predict 4.1 nano’s scores, and safety dropped with other monitors (Fig. 4). But I haven’t read their various setups too closely so I don’t want to overclaim.

• Morgan S 29 May 2026 2:42 UTC

  1 point

  0

  on: Ba­sic Leg­i­bil­ity Pro­to­cols Im­prove Trusted Monitoring

  I’m not sure the “commented code” setups map to the relevant threat model. It looks like you’re calculating safety by comparing (lower right of your first figure):

  1) Malign code, generated and commented by a malign policy, with
  2) Benign code, generated and commented by a benign policy

  But for estimating safety you’d actually want the latter to be:

  3) Benign code, generated and commented by a malign policy

  The reason this matters is that the actual threat model, as I imagine it, is something like:

  Newly-trained U is amazing at coding, much more so than previous models, but it writes code in an inscrutable way: everything T sees from it is semi-suspicious. We don’t have any absolute standard to conclude there’s something off about this behavior. Perhaps this is just how a genius operates?

  So if U is scheming, it wants to inflate the sus scores of its innocent code, so if it’s commenting it for T, it probably writes semi-confusing comments that aren’t blatantly obscurantist by absolute standards (a lot of geniuses aren’t great at explaining themselves).
  
  Previous control evals assumed this problem away by giving the blue team access to summary statistics of the benign policy, but this doesn’t make sense for this threat model. To see why, we can consider 2 cases:

  A) U acts like this in both evals and deployment. In this case, the summary statistics aren’t actually “benign”, but our evals will at least reflect the low safety scores we can expect in deployment. However, as the control team sees the disaster-in-motion, we’d like to have better countermeasures against U’s behavior.

  B) U does write/​comment code legibly in evals, but becomes obscurantist in deployment. In this case, the original control paper supposed that the lab would shut it down if this difference is significant at p < 0.03 and keep it shut down as long as this remained the case (see their Appendix G). Subsequent papers have inherited this assumption, but it doesn’t make much sense. Given their incentives and self-narratives, it’s somewhat unlikely they’d shut down even if they caught it trying to escape. How plausible is it that they shut it down based on a statistical diff between eval and deploy?

  • As evidence of this, pretty much all the frontier model cards have the AIs behaving worse when they don’t think they’re in an eval. That hasn’t exactly inspired any of them to undeploy.

  (For more details on the “benign statistics” assumption, see this doc by Vivek Hebbar. Ironically and unfortunately, it’s a very confusingly written output from a benign policy. Happy to chat about it, since I didn’t understand all the nuances until after my 4th reading.)

  So something I’m especially curious about is, what countermeasures are likely to help in such a scenario? I’m not quite seeing how the results here bear on this question, although I could be missing some insights here.

• Morgan S 25 May 2026 19:23 UTC

  1 point

  0

  in reply to: cdt’s comment on: How to game the METR plot

  Good catch! I think maybe it’s because I’m looking at data ~5 months newer from stronger models they’ve run, hence the “average success rate” of the orange dots has crept up a bit? At least, the x-values of all points look consistent to me e.g. the left 6 points.

• Morgan S 18 May 2026 23:13 UTC

  1 point

  0

  in reply to: Lukas Finnveden’s comment on: How to game the METR plot

  I’ve been doing some EDA on these datasets, vibe-coded this plot with confidence intervals, and remembered this discussion, so I figured why not drop it here:

  

• Morgan S 12 May 2026 16:03 UTC

  2 points

  0

  in reply to: neelasha-s’s comment on: [Paper] When can we trust un­trusted mon­i­tor­ing? A safety case sketch across col­lu­sion strategies

  Yeah, the prosaic answer we didn’t use BashArena was, “that seemed too hard when we started” in August 2025:

  • We initially tried using ControlArena, but at the time found it harder to use than just setting up APPS etc. ourselves (I’ve heard that it’s become better though)

  • Multi-step settings with multiverse time-travel (aka resampling) inherently add a lot of complexity

  • At the time we were completely new to control research; late in the project we did briefly extend to BigCodeBench, but doing multi-step would’ve been a whole new project and we wanted to wrap things up

  That said, I would conjecture that the main results in the paper, e.g. on passive vs. active collusion strategies, would transfer over to multi-step settings. Like, the generator has to collude with the monitor somehow, this means there has to be some amount of “collusion bits” (as we call it) in its output that distinguishes from a honeypot. Whether the output is built up in phases vs. one-off doesn’t seem to change the basic dynamics. But I could be wrong! We all (on this paper) want to see more work investigating collusion more deeply, and all else equal more realistic settings are better.

  If the red team were to submit fewer overall successful sabotaged trajectories (because BashArena makes sabotages more difficult), would that change how successfully the monitor can collude with the red team?

  This is an important open question! That is, let’s say in one setting it only has ~1 viable attack opportunity vs. 1000 in another setting (according to some fixed bar). If it’s good at selecting well among the 1000 attacks, how much extra uplift does it get? In untrusted monitoring, this mostly reduces to “how much variance is there in the collusion scores?” since presumably the generator chooses to attack only on the tasks that are most amenable to this.

  We didn’t get around to looking at this, although I wrote up some speculation here. Summary: collusion variance is part of the general problem of understanding/​mitigating “task heterogeneity” which is part of dealing with “dangerous inputs” (others have called this “attack selection”). This is definitely in my top 5 list of things I might pursue more later this year, and I’d be quite interested in hearing about any ongoing/​proposed work!

• Morgan S 6 Apr 2026 19:09 UTC

  3 points

  0

  in reply to: Shuo Li’s comment on: [Paper] When can we trust un­trusted mon­i­tor­ing? A safety case sketch across col­lu­sion strategies

  The paper seems to rely on monitors’ ability being close to the generator.

  Not really. An untrusted monitoring protocol like this works with any 2 models, or even any ensemble of models. You could even do an interlab untrusted monitoring setup where e.g. GPT-8 monitors Claude 7 and vice versa to achieve the benefits of monitor decoupling. Obviously if the monitor is much weaker than the generator, you’d have to evaluate how bad its performance is upfront, but you need to do this with trusted monitoring as well.

  If you mean our safety case sketch doesn’t handle this, I would disagree! Claim C1 lower-bounds how good U_M is at catching U_G’s backdoors, if U_M is trying its hardest. Your evals need to handle sandbagging from U_G, and there are known countermeasures for this general type of situation, but we left this as a “Missing Piece” because we didn’t get into the empirical weeds of this ourselves.

  Besides capabilities, U_G and U_M could also differ in their goals or coding style, but this almost always helps the control team: models of different goals would be less likely to collude, and different styles would make it harder to collude. So if we do have models of similar capabilities, the case where U_G = U_M is the most conservative, followed by cases where they’re closely related by fine-tuning or distillation.^[1]

  Has anyone tried the obvious eval here: freeze UM at checkpoint i, continue improving UG to checkpoint j>i, and then measure how collusion and suspiciousness ROC shift with j-i?

  I haven’t seen this, but in practice if U_G (or U_M) changes enough to plausibly shift the ROC curves meaningfully, you’d need to rerun your eval suite regardless. If it’s just a small fine-tuning step, the capability seems unlikely to be substantial enough to matter, and the control team could spot-check this with cheap subsamples of their evals. Studying scaling laws stuff (e.g. how elastic are the collusion/​suspiciousness ROC curves to given improvements in the underlying model’s capabilities) might be interesting for the basic science of control such as informing protocol design, but once the protocols are fixed the safety case just relies on what your evals say.

  1. ^

     I know of one exception to this assumption: suppose we actually manage to incriminate U_M on the basis of suspiciously missed honeypots. Then a next step would be to try to use these samples to incriminate U_G and apply few-shot catastrophe prevention to it. This is obviously easiest when U_G and U_M are most similar. However, our safety case sketch sidesteps this issue since we don’t assume we’re ever able to incriminate the monitor in the first place (as discussed in “Appendix E: Slack Honeypots”).

Un­trusted mon­i­tor­ing: ex­tra bits

[Paper] When can we trust un­trusted mon­i­tor­ing? A safety case sketch across col­lu­sion strategies