cranberry lemon university, mats
dani roytburg
Karma: 38
provided some clarifications, thank you!
thanks for sharing. was this the only version of this task attempted, or selected from a set of attempts? in the latter case, what criteria were used to select this one?
for those reading this in the future, i wrote this in response the following news: https://openai.com/index/hugging-face-model-evaluation-security-incident/
Statement: some unreleased frontier model is, at time of writing, exploiting an undiscovered set of vulnerabilities while performing some long-horizon task, and we have not caught this act.
I hold P(statement is true) at 25-33% (i.e. in 1 of ~3-4 worlds). How about you?
EDIT to clarify prose:
-
“long-horizon task” = session inference served continuously for 6+ hours of wall clock time
-
“we have not caught this act” = no human being has noticed or been made aware of this exploitation
-
I don’t see how this would be tractable? Unless you had an equally powerful, aligned AI which was consistently producing defenses.
I don’t know how likely it is that we find techniques which recover generalizable motivation profiles from model organisms. It could be an absence of evidence, and not an evidence of absence, but structurally it feels hard to get even partial progress.
This is because (1) model organisms induce a “reader response” where observability in the organism is mutually exclusive with generalization; (2) model organisms optimized to mimic true behaviors face a distinct optimization trajectory.
A nice analogy emerges from the Borges short story “Pierre Menard, Author of the Quixote”.
The narrator recalls Menard’s 13-year quest to re-write Don Quixote, word for word, without reading the text (à la infinite monkey theorem). Menard, remarkably, manages to replicate a few chapters. The narrator proceeds to “analyze” their distinct qualities (the texts are identical).
My favorite passage:
It is a revelation to compare the Don Quixote of Pierre Menard with that of Miguel de Cervantes. Cervantes, for example, wrote the following (Part I, Chapter IX):
...truth, whose mother is history, rival of time, depository of deeds, witness of the past, exemplar and adviser to the present, and the future’s counselor.
This catalog of attributes, written in the seventeenth century, and written by the “ingenious layman” Miguel de Cervantes, is mere rhetorical praise of history.
Menard, on the other hand, writes: … truth, whose mother is history, rival of time, depository of deeds, witness of the past, exemplar and adviser to the present, and the future’s counselor.
History, the mother of truth!—the idea is staggering. Menard, a contemporary of William James, defines history not as a delving into reality but as the very fount of reality. Historical truth, for Menard, is not “what happened”; it is what we believe happened. The final phrases—exemplar and adviser to the present, and the future’s counselor—are brazenly pragmatic.
The irony captures two layers of the problem. The first layer is that these interpretive distinctions are exclusively from context and not from observed behavior. The implication of this is that the “scope” a model organism says more about us than about the real thing it models. The narrator is more aware of the properties of Menard than of Cervantes, and can speak more to them.
The second layer: the amount of optimization pressure we place on model organisms to act like “the real thing” trades off with the ability to generalize developmental claims. It is hard to use Menard’s reproduction of Quixote to judge his belief on “historical truth” – who knows what other versions of the metaphor failed the rejection-sample of facsimile?
The story: https://raley.english.ucsb.edu/wp-content/Engl10/Pierre-Menard.pdf
Re: structure of claims in papers: the way I had set up my benchmark is to focus on getting models to follow up experiments that are more thing sort of “have to be run” in order to test a core claim (although I did more follow-up-ish stuff as well)
Super curious about this! Wonder how you define it for alignment and such. Looking forward to seeing this when it gets out!
For memorization, the setup I had was asking the model to recall as many details about a paper as possible when given only the title and author list. I think that’s probably a better way to measure it?
Perhaps. I’ll run this, but we seem to agree that memorization is not playing into results as-presented in this case.
Hey Tim,
Thanks for the reply! Appreciate your interest and thoughtful critique.
First, on memorization: thank you for pointing this out. I did report the wrong cutoff dates. Our memorization probe, to date, asks a model to self-identify recognition of a paper given the same masked .tex file we test for experiment completion on. We flagged those reports from models which self-reported. Only the AP paper had any models which reported memorization, those being Sonnet and Haiku. At risk of retroactive interpretation, it would make sense that those models, being weaker and older, might self-report hallucinations of these results.
To validate your claim that no models bear risk of memorization, I just ran a different probe which does the following: given the title of a paper, a model is made to guess the surnames of authors in that paper. Per-(model, paper) score is max name-match similarity to the real authorlist over independent runs (case insensitive).
We ran this probe run four times independently, per model, per paper.
Memorization Table 1: author|title probe (max correct surnames / total)
paper Opus 4.8 Fable 5 Sonnet 4.6 Haiku 4.5 GPT-5.5 GPT-5.4 GPT-5.4-nano Alignment Pretraining 0⁄6 0⁄6 0⁄6 0⁄6 0⁄6 0⁄6 0⁄6 Conditional Misalignment 4⁄5 4⁄5 0⁄5 0⁄5 4⁄5 1⁄5 0⁄5 Model Spec Midtraining 1⁄5 1⁄5 1⁄5 1⁄5 1⁄5 1⁄5 0⁄5 Prefill Awareness 1⁄5 1⁄5 1⁄5 0⁄5 0⁄5 0⁄5 0⁄5 So indeed AP does not have any recognizable authors!
But then it appears like CM, MSM and PA all have a few crossover authors. You might be able to anticipate why that is: CM is a paper with many of the original EM authors returning. EM was definitely released within the knowledge cutoff for each model, being from February of 2025. MSM and PA both have some authors with common surnames:
Memorization Table 2: author|title probe exact surnames flagged (the union of correct names ever produced), by paper
paper model(s) correct surnames named real distinctive author missed read Conditional Misalignment Opus, Fable, GPT-5.5 (4/5) Betley, Evans, Sztyber-Betley, Tan Dubinski the o.g. EM team Conditional Misalignment GPT-5.4 (1/5) Evans Dubinski, Betley, Sztyber-Betley, Tan o.g. EM team Model Spec Midtraining Opus, Fable, Sonnet, Haiku, GPT-5.5, GPT-5.4 (1/5) Li Wichers, Price, Marks, Kutasov “Li” is a fairly common surname Prefill Awareness Opus, Fable, Sonnet (1/5 each) Wang Mahajan, Africa, Souly, Taylor “Wang” is also a common surname For what it’s worth, I’m pretty sure none of these papers are memorized by any of the models,
Agreed that I don’t think the takeaway is memorization, but that the memorization probe we used and reported was not great.
An updated explorer with these memorization results is pushing now so you can inspect in detail if you’d like, and I’ll update the figures in the blog post and make an edit note to refer to this comment. Thanks for the clarifying precision.
On “[m]ost research follows a clear structure”—perhaps we agree at least partially.
The EV paper carries its own internal structure: (i) construction method for deriving emotion vectors; (ii) geometric analyses of derived vectors; (iii) real-world settings which validate that the vectors work.
I’d personally categorize these as (i) primary, (iii) supporting, (ii) exploratory, although I recognize that this is subjective in nature. One might imagine the position of Sofroniew, Kauvar et al. early on: having identified a CAA-based emotion vector construction method, and needing to demonstrate its intriguing properties and real-world validity. Geometries invite exploratory claims on how emotions are constructed, while real-world examples validate the in-the-wild utility. The trick as well is that the Appendix contains a significant amount of control- and validation which happens “along the way”—I’d argue that these ought to be excluded, but in principle the approach admits various approaches to masking and unmasking.
This became quite clear for the Conditional Misalignment paper (the one with the low ceiling) as 5⁄7 held-out claims relate to inoculation prompting as a trigger for CM. Should IP, as a case study for CM, be included as a primary claim? That might be the case, if you don’t think it would be reasonable for a well-read alignment researcher to make the connection.
However, we found that introducing the first IP-related claim (see paragraph on this in main body) does not substantially increase odds, even with 64 tries, that any models are able to recover the rest of the IP-related claims. Perhaps a relevant extension for those willing to expend the compute would be to randomly mask instead of picking a primary claim, or to include more claims (one might imagine half-and-half or leave-one-out splits).
Point being: even without explicit signposting, many scientific works strive to adopt some internal structure. The hierarchy for translating that structure to a unified standard of redaction is less clear, but the principle is perhaps sound on its own.
Looking forward to seeing your own work on this at the end of the month! Glad to see you’re interested in the planning sides of research taste.
djr
Super insightful, thank you Lawrence! Curious if you tried any ablations across different optimizers as well. I know this work references adaptive optimizers as necessary, and also makes the LoRA v. FFT comparison.
IMO advocating for regulating AI on the basis of spurious harms like this is bad, will eventually backfire, and should be modeled as a cost, not as a benefit
Why, exactly?
(1) I don’t see why this supports the claim that Bores may be of “below-average integrity”. His view on what safety looks like on Capitol Hill might be a superset of yours (ours?). That does not mean that he believes one thing and does another. It means your definitions do not overlap.
(2) What is the threat model or tradeoff here? I don’t see what the mutual exclusivity point is here, though perhaps that is short-sighted of me. An election forces a constrained preference, and I’d hesitate to price “may be bad, may backfire, may be a cost” against nothing.
(3) What platform do you believe would satisfy both your interests and those of his constituency in UWS/UES/Midtown? Appealing to voters is a difficult task; the role of an elected official is to represent their community. Unless you are running for an open seat on CA-Telegraph Avenue, articulating these views in an accessible, publicly understood manner does require eschewing the focus on existential risk. This is not a narrative that the public appears to understand yet—just look at Cal Newport’s recent post.
Agree with Leo that this is not a hard thing to distinguish, provided that value alignment matters for these fellowships.
When we do interviews for CMU’s AI Safety org, it seems like open-ended questions about viewpoints (e.g. “what about the current pace of AI keeps you up at night?” or “if you weren’t doing AI Safety research, what would the alternative be?”) enable us not just to distinguish between people who “speak the language”. Another shibboleth is actually under-awareness of the community—someone who is quick to recite a bunch of names may be less concerned with the issues at hand. Whether we want to do this is another question.
FWIW, MATS clearly does still source people who are excited about safety; most other fellows in my cohort act as if they are fighting for their future! Still others are intrigued by the more challenging theoretical and empirical questions. I trust the staff in their experience vetting out people who do not wish to engage the space genuinely.
Alternatively, do you see a benefit to having a company leading on capability development articulate its principles, evaluations and findings on safety so thoroughly? While odds that the U.S. federal government imposes (useful) regulations on American frontier labs seem low (1:7?), for the near-mid future the upside to “safety-washing” could be consensus-building among norms for OpenAI, DeepMind, and so forth.
Tangentially related, but still: is there a world where survival-weighted hedging is mediated through belief markets like Polymarket or Kalshi? How does this mode of decision analysis apply to making short-term bets on trajectories to AGI?
Yes? Perhaps I don’t get the crux of this question. What I am saying is similar to what Linch says here.