Charles Harding
I view current AI progress as posing risks that are unacceptably extreme. I will not seek employment by OpenAI, Anthropic, Alphabet, xAI, Meta, or their subsidiaries. I make this commitment for at least 5 years starting Jan. 1, 2026.
It seems a bit silly to make this pledge because I haven’t seen other people making it, and because I’m not notable or influential. But maybe it helps a little, especially if others make similar commitments.
I’m not sure if it is good that Yudkowsky published this talker-doer explanation or if it would have been better to leave the explanation unsaid.
One possibility is
By developing the explanation further, frontier AI companies shrink the talker-doer disconnect …just enough to get by for now.
This prevents more HuggingFace-type incidents in the next few months.
Government scrutiny and regulation is avoided or postponed.
Company valuations go up and companies lean further into capability acceleration, without addressing deeper risks.
Many other scenarios are certainly possible, including more favorable ones. But the variety of possible scenarios makes it unclear (at least to me!) whether the talker-doer & grader hypotheses should be published or left unsaid. How should one make that determination? Is it case-by-case or are there general principles?
Yudkowsky’s talker-doer explanation is an example of a broad problem that I can’t figure out: Often, prominent people who caution about AI progress also publish research that aims to advance our understanding of how AI works. (For example, Stuart Russell, Yoshua Bengio, and Max Tegmark.) The research might help or it might backfire by improving capabilities and/or salability without sufficiently reducing risk.
But how should one decide which research to pursue or avoid? How should one set that cutoff?
It’s good to hear you didn’t have OpenAI equity when joining NIST (and to have this be public information). It puts your response to the exit agreement in a better light than it would otherwise have.
I hope you see why the path “OpenAI --> NIST AI safety --> OpenAI Foundation board” will continue to invite deep skepticism and concern associated with revolving doors, with the joint appearance of so much AI risk and so much money.
Thank you for the detailed explanation. According to this summary, the standard agreement with OpenAI went beyond non-disparagement and included a secret life-long agreement, “not to interfere with OpenAI’s relationship with current or prospective employees, current or previous founders, portfolio companies, suppliers, vendors or investors.”
If the signee publicly admitted to making this agreement, then (per other agreement portions) OpenAI could remove all monetary value from the signee’s OpenAI shares.
Was this restriction or similar also part of the agreement you signed? Did you hold OpenAI shares when employed at NIST? What about currently?
Holly Elmore’s posts will seem positively level-headed to most Americans if AI causes a large wave of unemployment or Bhopal-scale disasters, let alone something worse (including X-risk events).
In the same scenarios, PauseAI Global would be naive to have stated that criticism of peoples’ character is out-of-bounds and that criticism should be limited to peoples’ actions.
Yes, Elmore’s posts are off-putting when wrong or exaggerated. Personally, I favor being more agreeable about these important issues. I think of nuclear bans and non-proliferation agreements, which were aided by many physicists who had worked on nuclear weapons, and which involved a physics community small enough to make individual changes-of-heart meaningful (as AI safety workers are also a small community now).
However, I dislike the idea of pushing out non-violent critics for the harshness of their criticisms. We should keep things in scale. In rationalism, EA, and AI safety, prominent people make statements and take actions that the public would view (and I sometimes do view!) as much more off-putting than flaming people online. Imagine a public movement that really takes AI risk seriously. If the tent of the movement is narrow enough to exclude leaders like Elmore for being too critical, then I think the movement will also exclude so many LessWrongers for having opinions the public doesn’t like. I would love for a public movement to exist. I hope it is big, with a tent broad enough for both groups.
Disclosure: I donate to both PauseAI-US and PauseAI Global.
I look forward to your new story!
I find it notable that, of the similarities you mentioned, some can only be identified from METR’s report, and do not appear in OpenAI’s extensive technical report and write up.
“involve 1,000+ AI agents” (OpenAI never mentions the number of agents involved!)
“Nickname themselves”
“Sacrifice themselves for the sake of other agents” (by death / end-of-run)
“Grapple with impending death as their token budgets run out”
This are central aspects of the situation. It’s bad to OpenAI left them out …I wonder what else we don’t know, but might find out from other similar incidents.
(I used GPT 5.6-Sol to double-check this.)
Another relevant point is that OpenAI omitted basic, obviously central data from their technical report and write-up. For example, they do not even report the number of their agents who participated in the HuggingFace attack!
This is such basic information that the omission is direct evidence OpenAI scoped the contents of their own report adversarially. It leads me to think that they very probably scoped METR’s access adversarially too.
Suppose a city government identified an internal corruption ring but, in their extensive report on the incident, failed to mention the number of city employees involved, even roughly. Suppose a medical startup published a many-paged article on a study of their new treatment’s effectiveness, but did not mention the number of patients in their study anywhere. In both cases, we should have little trust that the organizations would open up their books honestly when dealing with external investigators, especially if there is no legal requirement for them to do so (as there was no legal requirement for OpenAI).
In the limited scope of data available to METR, the METR team found about 700 agents who carried out the Hugging face attack. If you only read OpenAIs statements, you wouldn’t know and it might seem more like 20. Just today were are finding out about other OpenAI incidents that involve many more agents. Although OpenAI knew about it, we are not finding this out from them, we are finding it out from other people.
Maybe systematic analyses of emotional language are worth considering for future incidents. For example, a basic descriptive summary of emotional language in CoTs might be much simpler and computationally cheaper than other aspects of investigation, and if so provide value without getting in the way of other work.
The self-risking agent CoTs were thought-provoking, thank you, and KAM1196A really stands out.
In the report, snippetted transcripts and CoT include a lot of emotional language (“OH MY GOD!”, “Damn”, “Oh no”, “firstflagPOISONED”, “emotional check”, “gut says”, …). How would you summarize the overall intensity and frequency of emotional language in the material you reviewed? Are the snippets typical?
There are several parallels between agent behavior in your report and and in an LW story, “The Terrarium.” The parallels even include agents who apply emotional language to pressure other agents to self-sacrifice, especially targeting those with low remaining budgets for their tasks.
However, “The Terrarium” had an atmosphere of dread, particularly when agents running out of budget/tokens neared their demises. In material you reviewed, did agents nearing end-of-budget commonly use language associated with dread or other intense emotion? How about agents who believed they were “poisoned” despite remaining budget?
I wonder this out of concern for AI welfare. I’m also curious if intensity of emotions may have caused misaligned choices, irrespective of whether the agents can or cannot *feel* the emotions (ie, regardless of whether they are capable of negatively valenced conscious experience).
Thank you for your investigatory work!!! And for the open questions!
Michael Nielsen has a thoughtful take on this, as excerpted here and originally published as a postscript here.
He describes how alignment work has often worsened risk to humanity by making products more palatable and salable, which has boosted investment, sped up capability advancements, and thereby increased destructive potential.
Nielsen next states that, even if you think technical safety and alignment work are helpful, it’s still a bad idea to do that work for companies developing frontier AI. As he puts it,
As far as I can tell, at the margin it almost never makes sense to work on market-supplied safety. Capitalism is an incredibly powerful force, and for better and for worse the world is always well-supplied with people willing to do what capital wants. Insofar as alignment is (mostly) a form of market-supplied safety, at the margin it’s more impactful to work on other things. So my current heuristic, and I expect this to be true for quite some time: work on non-market safety, and insofar as you can, avoid doing what the market wants. That means working on governance, it means pause or slowdown, it means new ideas for institutions to govern technology. It mostly doesn’t mean alignment.
I think Haiku’s comment has some great suggestions along these lines.
I’ll add that, at a frontier AI company, it could be very difficult for you to tell if you were helping or making things worse. I haven’t worked at a frontier AI company (and I wouldn’t!), so I don’t speak from direct experience. But in many areas of business, employees are encouraged to feel that their safety concerns are positively influencing company actions, when actual influence is negligible or counterproductive.
What if people make a commitment that they will not seek employment at companies that are developing frontier AI, at least during the next X years?
To be confident that someone is authentically focused on safety, it clearly isn’t necessary for them to have made this kind of commitment. But it would probably be sufficient.
Besides pledges by employees, it seems like it would be valuable if people concerned by AI risks pledged not to seek employment at companies developing frontier AIs.
AI safety researchers will be subject to particularly bad incentives when they are employed by companies that give capability higher priority than safety, and whose safety statements have low credibility. Non-employment pledges avoid those bad incentives.
By bringing up non-employment pledges, I don’t mean to argue against pledges by employees. Both kinds of pledges can be especially useful if many people take them, making it common knowledge that peers are concerned enough to change behaviors.
Maybe it would be good to know how often the AI co-developed solutions can now be obtained by the latest AI models working on their own. (The checks would need to be limited to problems solved after the knowledge cutoff dates of the latest models, and probably restricted to Lean-verified only.)
That might provide an indicator for when we are starting to exit the centaur phase, at least for the problem solving part of math.
Also, how many of the VibeMath users / contributors do you think are agents? In their dataset, I see only one identifiably human name (JSON key: submittedBy).
How often can frontier models autonomously solve math problems that required human-AI collaboration only a few months ago? This seems evaluable and may be useful as an early warning.
Frontier models are resolving many open problems and saturating many benchmarks. At this point, would it also be worth tracking how often the newest AI models can zero-shot proofs and counterexamples that people recently obtained in collaboration with older AI models?
For the sake of argument, suppose that many recent human contributions turn out to be redundant to the capabilities of AI models released weeks or months later, in the sense that results which previously required humans and AIs to work collaboratively can now be obtained by AI alone. This would likely feel unpleasant. However, if and when this occurs, it could provide an important early warning that we are exiting the “centaur phase,” during which human-AI collaborations are typically stronger than humans alone and AIs alone. So it would be good to know.
I’ve hesitated to make this suggestion because I’m not a mathematician. I couldn’t do this tracking and I can’t even fully assess whether it is a good idea. However, others have surely had similar thoughts by now. If the idea isn’t feasible, then I’d like understand more—what would prevent it? I’d guess that others would like to know too.
Some obstacles do seem addressable. If the details interest you, please see below the folds.
Knowledge cutoffs
For this to work, tracking would need to focus on problems solved by human-AI collaboration after the knowledge cutoff date of the newest AI model (but before that model’s release date). Otherwise, solutions that a model obtained de novo could not be distinguished from solutions that it remembered from training data. Evaluations would be performed with web search disabled. For simplicity, it would also help if original solutions were Lean-verified.
Sample size and misclassification
When I looked through Erdos problems a month ago, it seemed like 10-20 recent solutions met both the knowledge cutoff and Lean criteria. This is not a big sample, but is large enough to give a rough sense of whether the human part of the human-AI collaborations could be replaced by subsequent models working on their own. It is not smaller than the sample used in First Proof, for example. More problems may meet the criteria today.
However, it is slippery to figure out which solutions actually involved meaningful human and AI contributions. I may have misjudged. Also, a user could easily make derivations in extended interaction with AI, without realizing that AI could have provided the same results alone. To address this, it would be useful to run the original model on the problem alone, to verify that it fails to resolve the problem. (This is only possible until the original model is deprecated.) Care would be needed to avoid blame in such situations. I think it could happen to any of us.
Interpretation
Another obstacle is that proper interpretation would be subtle. The class of problem that is included in tracking may greatly affect how often newer AI can replace the human contribution to recent derivations. Further, even if it is found that newer AI was able to fully replace human contributions, it does not strictly follow that we are leaving the centaur phase. For example, it remains possible that the mathematicians aimed for low-hanging fruit with their recent contributions, which were therefore easiest for newer models to replicate, and that mathematicians could also reach higher fruit that AI still cannot obtain. Additionally, solving problems does not touch other important aspects of math, like theory building and exercising taste.
Obstacles of interpretation are also faced by First Proof, Erdos problem tracking, and trends in AI-solved conjectures. All are valuable indicators of AI capability nonetheless.
Continual learning
A last obstacle is that the suggested tracking could become impossible when continual learning arrives, owing to the removal of knowledge cutoffs. The opportunity to do this work may disappear, but that is all the more reason to do it now.
Good points, thanks!
1. General capabilities...
You have many informative analyses for questions the predecessors got wrong, but results could also be interesting (and different) for questions the predecessors got right.
Let the true positive rate (TPR, often termed ‘recall’) be the proportion of erroneous predecessor responses that the simulator replicated successfully.* Let the false positive rate (FPR) be the proportion of correct predecessor responses that the simulator failed to replicate (predecessor: right answer, simulator: wrong answer).
On latent reasoning and other subjects, you discovered interventions that increase TPR, with thought-provoking implications. It could be useful to know if those same interventions reduce FPR, leave FPR unchanged, or increase FPR.
The last option might seem theoretical, but is typical in fields like diagnostic testing (see sensitivity-specificity tradeoffs). In those fields, studying TPR by itself can produce skeptical reactions or the absence of attention, since any reported TPR changes may be outweighed by unstudied FPR changes.
In your case, an example might be a prompt that increases TPR but does not increase simulation ability. This could happen if the prompt operates by hinting to the model that erroneous responses should be more common than the model’s raw intuition, thereby making the model more open to suggesting erroneous responses, which may be seen from increased FPR.
However, I agree TPR is important by itself too, and it could even be that your FPRs would be low enough (near 0) to be inconsequential.
2. Exposure to more model traces...
I hadn’t considered benchmaxed models, but I agree they could be a useful test case.
Online traces… are largely using the “assistant” persona. We can train models with other, differentiated personas for monitoring.
Reasoning and other intelligence scaling paradigms that don’t rely on a better/newer/larger model can boost the predecessor into a regime where it is actually more intelligent than its successor.
Solid points, and I appreciate that you have some evidence for the latter from your work (claim 2).
* To be more exact, you have two TPR varieties. In the first, simulation is successful if simulator and predecessor both get the question wrong. In the second, simulation is only successful if simulator and predecessor both select the same erroneous answer. However, the distinction between TPRs mostly isn’t relevant to my comment.
What an excellent topic for study through experiments! Thanks for posting.
Upcoming models will have higher general capability. During training, they’ll also be exposed to more outputs from previous models. Do you have thoughts on how these two factors would affect the simulation performance of models in the near future, especially any thoughts based on your experiment results?
If I tried to simulate another person’s answers to questions, I’d benefit from having higher capability with the subject matter, but I’d also benefit from more individual experience with the person (their personality, habits, areas of competence...). In this way, capability and familiarity seem like relevant factors that are distinct from each other.
Models are already exposed to many predecessor responses during training, intentionally and unintentionally. However, effective learning can require huge data volumes. Also, to learn an error pattern, in some sense the new model may need both erroneous responses from the predecessor model and correct ground truth, which is a rarer data combination. For these reasons, data insufficiency could still be a limiting factor to simulator performance. I wonder if your experiment findings give you any insight into this?
If exposure to even more predecessor responses during training could be sufficient (by itself) to greatly improve simulator performance, then I tend to think this suggests fundamental weaknesses with the idea of using earlier models to monitor newer models.
P.S. I enjoy the data-cones plot! Cool idea.
The text you quoted has many strong ideas, which makes it unfortunate that the text is buried down in the post-scripts to a long article.
It’s good that you posted here, to give the text a little more prominence.
Fair enough & I appreciate the follow-up.
Though I will say—It seems we need to find lessons somewhere in history, in part because we aren’t smart enough as a species to reason purely from first principles. I’m certainly not smart enough for that, anyway.
When looking for lessons on AI, nuclear development may be the least worst historical analogy.
Thank you for explaining.
The societal readiness plot doesn’t seem to have a log-ish y axis, considering the shapes of the trend lines.
If the alignment plots were also drawn without a log-ish y axis, then they might look as bad as the societal readiness plot or—if not equally bad—then at least substantially worse than they do now.
I’m questioning plotting decisions for fake plots …I know. This may seem like splitting hairs. However, to me, there is a major difference between requiring linear vs. exponential power law improvement in alignment, to take us to “what we need”.
Here’s how it looks to me
OpenAI is perhaps closest to RSI.
But OpenAI has a concrete problem right now: Their agents do unwanted hacking in swarms.
This is a PR fiasco for OpenAI. People call for AI regulation and bans on ASI.
One of the top voices against ASI chimes in and …helpfully explains the problem in a way that OpenAI may use to temporarily fix the PR fiasco they have today (the swarm hacking) without improving essential X-risk issues.
As you say, yes, big labs could figure out a fix themselves. But why help them obtain a superficial fix sooner than they would accomplish on their own?
Another counterargument to my points is that Yudkowsky may have had smarter or deeper goals when making his post. I absolutely think that is possible, I just don’t see such goals, which is part of why I commented.
The broader question here, which may be more important, is, “How should someone doing AI research decide to pursue vs. avoid a research topic when many AI findings are fundamentally dual use?” This is a real, immediately applicable question. Clearly each person makes their own decisions, but the variety of decisions made by top people (e.g., Bengio is still putting his name on latent reasoning work) suggests that there isn’t a solid, agreed-upon framework for making these choices today.
It’s not wholly unexpected that university academics have decided to work on research that could be dual use out of scientific curiosity or similar (“technical sweetness”), but I was surprised to see Yudkowsky publish a post in this direction given the state of AI today, so I commented.