Thanks for the reply! Appreciate your interest and thoughtful critique.
First, on memorization: thank you for pointing this out. I did report the wrong cutoff dates.
Our memorization probe, to date, asks a model to self-identify recognition of a paper given the same masked .tex file we test for experiment completion on. We flagged those reports from models which self-reported. Only the AP paper had any models which reported memorization, those being Sonnet and Haiku. At risk of retroactive interpretation, it would make sense that those models, being weaker and older, might self-report hallucinations of these results.
To validate your claim that no models bear risk of memorization, I just ran a different probe which does the following: given the title of a paper, a model is made to guess the surnames of authors in that paper. Per-(model, paper) score is max name-match similarity to the real authorlist over independent runs (case insensitive).
We ran this probe run four times independently, per model, per paper.
So indeed AP does not have any recognizable authors!
But then it appears like CM, MSM and PA all have a few crossover authors. You might be able to anticipate why that is: CM is a paper with many of the original EM authors returning. EM was definitely released within the knowledge cutoff for each model, being from February of 2025. MSM and PA both have some authors with common surnames:
Memorization Table 2: author|title probe exact surnames flagged (the union of correct names ever produced), by paper
For what it’s worth, I’m pretty sure none of these papers are memorized by any of the models,
Agreed that I don’t think the takeaway is memorization, but that the memorization probe we used and reported was not great.
An updated explorer with these memorization results is pushing now so you can inspect in detail if you’d like, and I’ll update the figures in the blog post and make an edit note to refer to this comment. Thanks for the clarifying precision.
On “[m]ost research follows a clear structure”—perhaps we agree at least partially.
The EV paper carries its own internal structure: (i) construction method for deriving emotion vectors; (ii) geometric analyses of derived vectors; (iii) real-world settings which validate that the vectors work.
I’d personally categorize these as (i) primary, (iii) supporting, (ii) exploratory, although I recognize that this is subjective in nature. One might imagine the position of Sofroniew, Kauvar et al. early on: having identified a CAA-based emotion vector construction method, and needing to demonstrate its intriguing properties and real-world validity. Geometries invite exploratory claims on how emotions are constructed, while real-world examples validate the in-the-wild utility. The trick as well is that the Appendix contains a significant amount of control- and validation which happens “along the way”—I’d argue that these ought to be excluded, but in principle the approach admits various approaches to masking and unmasking.
This became quite clear for the Conditional Misalignment paper (the one with the low ceiling) as 5⁄7 held-out claims relate to inoculation prompting as a trigger for CM. Should IP, as a case study for CM, be included as a primary claim? That might be the case, if you don’t think it would be reasonable for a well-read alignment researcher to make the connection.
However, we found that introducing the first IP-related claim (see paragraph on this in main body) does not substantially increase odds, even with 64 tries, that any models are able to recover the rest of the IP-related claims. Perhaps a relevant extension for those willing to expend the compute would be to randomly mask instead of picking a primary claim, or to include more claims (one might imagine half-and-half or leave-one-out splits).
Point being: even without explicit signposting, many scientific works strive to adopt some internal structure. The hierarchy for translating that structure to a unified standard of redaction is less clear, but the principle is perhaps sound on its own.
Looking forward to seeing your own work on this at the end of the month! Glad to see you’re interested in the planning sides of research taste.
For memorization, the setup I had was asking the model to recall as many details about a paper as possible when given only the title and author list. I think that’s probably a better way to measure it? The thing you care about is “how much of the paper does the model remember,” not if the model remembers the authors. You can even use a explicitly pre-knowledge cutoff model’s guesses as like, a baseline for how far you can get from knowing the title and author list and purely hallucinating.
Re: structure of claims in papers: the way I had set up my benchmark is to focus on getting models to follow up experiments that are more thing sort of “have to be run” in order to test a core claim (although I did more follow-up-ish stuff as well) (See also AblationsBench). For the emotions paper, I felt like there were many justifiable directions that the authors could’ve gone down, and thus it’s sort of hard to grade the AIs.
Re: structure of claims in papers: the way I had set up my benchmark is to focus on getting models to follow up experiments that are more thing sort of “have to be run” in order to test a core claim (although I did more follow-up-ish stuff as well)
Super curious about this! Wonder how you define it for alignment and such. Looking forward to seeing this when it gets out!
For memorization, the setup I had was asking the model to recall as many details about a paper as possible when given only the title and author list. I think that’s probably a better way to measure it?
Perhaps. I’ll run this, but we seem to agree that memorization is not playing into results as-presented in this case.
Hey Tim,
Thanks for the reply! Appreciate your interest and thoughtful critique.
First, on memorization: thank you for pointing this out. I did report the wrong cutoff dates. Our memorization probe, to date, asks a model to self-identify recognition of a paper given the same masked .tex file we test for experiment completion on. We flagged those reports from models which self-reported. Only the AP paper had any models which reported memorization, those being Sonnet and Haiku. At risk of retroactive interpretation, it would make sense that those models, being weaker and older, might self-report hallucinations of these results.
To validate your claim that no models bear risk of memorization, I just ran a different probe which does the following: given the title of a paper, a model is made to guess the surnames of authors in that paper. Per-(model, paper) score is max name-match similarity to the real authorlist over independent runs (case insensitive).
We ran this probe run four times independently, per model, per paper.
Memorization Table 1: author|title probe (max correct surnames / total)
So indeed AP does not have any recognizable authors!
But then it appears like CM, MSM and PA all have a few crossover authors. You might be able to anticipate why that is: CM is a paper with many of the original EM authors returning. EM was definitely released within the knowledge cutoff for each model, being from February of 2025. MSM and PA both have some authors with common surnames:
Memorization Table 2: author|title probe exact surnames flagged (the union of correct names ever produced), by paper
Agreed that I don’t think the takeaway is memorization, but that the memorization probe we used and reported was not great.
An updated explorer with these memorization results is pushing now so you can inspect in detail if you’d like, and I’ll update the figures in the blog post and make an edit note to refer to this comment. Thanks for the clarifying precision.
On “[m]ost research follows a clear structure”—perhaps we agree at least partially.
The EV paper carries its own internal structure: (i) construction method for deriving emotion vectors; (ii) geometric analyses of derived vectors; (iii) real-world settings which validate that the vectors work.
I’d personally categorize these as (i) primary, (iii) supporting, (ii) exploratory, although I recognize that this is subjective in nature. One might imagine the position of Sofroniew, Kauvar et al. early on: having identified a CAA-based emotion vector construction method, and needing to demonstrate its intriguing properties and real-world validity. Geometries invite exploratory claims on how emotions are constructed, while real-world examples validate the in-the-wild utility. The trick as well is that the Appendix contains a significant amount of control- and validation which happens “along the way”—I’d argue that these ought to be excluded, but in principle the approach admits various approaches to masking and unmasking.
This became quite clear for the Conditional Misalignment paper (the one with the low ceiling) as 5⁄7 held-out claims relate to inoculation prompting as a trigger for CM. Should IP, as a case study for CM, be included as a primary claim? That might be the case, if you don’t think it would be reasonable for a well-read alignment researcher to make the connection.
However, we found that introducing the first IP-related claim (see paragraph on this in main body) does not substantially increase odds, even with 64 tries, that any models are able to recover the rest of the IP-related claims. Perhaps a relevant extension for those willing to expend the compute would be to randomly mask instead of picking a primary claim, or to include more claims (one might imagine half-and-half or leave-one-out splits).
Point being: even without explicit signposting, many scientific works strive to adopt some internal structure. The hierarchy for translating that structure to a unified standard of redaction is less clear, but the principle is perhaps sound on its own.
Looking forward to seeing your own work on this at the end of the month! Glad to see you’re interested in the planning sides of research taste.
djr
For memorization, the setup I had was asking the model to recall as many details about a paper as possible when given only the title and author list. I think that’s probably a better way to measure it? The thing you care about is “how much of the paper does the model remember,” not if the model remembers the authors. You can even use a explicitly pre-knowledge cutoff model’s guesses as like, a baseline for how far you can get from knowing the title and author list and purely hallucinating.
Re: structure of claims in papers: the way I had set up my benchmark is to focus on getting models to follow up experiments that are more thing sort of “have to be run” in order to test a core claim (although I did more follow-up-ish stuff as well) (See also AblationsBench). For the emotions paper, I felt like there were many justifiable directions that the authors could’ve gone down, and thus it’s sort of hard to grade the AIs.
Super curious about this! Wonder how you define it for alignment and such. Looking forward to seeing this when it gets out!
Perhaps. I’ll run this, but we seem to agree that memorization is not playing into results as-presented in this case.