Charles Harding
I view current AI progress as posing risks that are unacceptably extreme. I will not seek employment by OpenAI, Anthropic, Alphabet, xAI, Meta, or their subsidiaries. I make this commitment for at least 5 years starting Jan. 1, 2026.
It seems a bit silly to make this pledge because I haven’t seen other people making it, and because I’m not notable or influential. But maybe it helps a little, especially if others make similar commitments.
OpenAI has a large gap between what they say and what they want. If you lead an organization and you say one thing but want another, you’ll have a problem: Sometimes, employees will do what you said, not what you actually wanted!
This kind of situation is admittedly confusing. However, there are some ways the company can help employees learn. For example, when firing people for doing what the company asked, the company can mention reasons for termination that, beyond being pretenses, are *obvious* pretenses. Then more employees will get the message.
Maybe an employee opened the wrong email. That’s a suitable pretense. The company could come up with more plausible reasons for firing the employee, yes, but that would be worse for the company. Then, other employees might get confused again. They might go back to doing what the company says, instead of what it wants.
If the game is sequential Stag Hunt, why doesn’t one company pause first?
You mentioned several potential explanations. Here’s another that concerns me: A company might have entered into financial entanglements that pressure it to race forward (instead of pacing or pausing) to the extent that it would race even if it had no competition, even in the presence of acknowledged X-risks.
For example, if a company takes on enough debt or debt-like commitments for data center infrastructure and chips, then it may get so far over its skis that it needs capability increases quickly, merely to make it possible that it can address its indebtedness and commitments.
I do not have expertise in corporate finance and I do not know that any current AI company is subject a financial trap of this kind: one that ties its continued existence to a path of acceleration even if competition were absent. However, companies have incredibly complicated financial entanglements and economics includes many other perverse scenarios (debt traps, dollar auctions, tragedies of the commons...).
Anyway, thank you for your analyses here and in your excellent earlier post!!
can someone convince me that it’s worth actually thinking about and talking about model welfare before the singularity? i don’t get why lots of people are so insistent on talking about it...
Many people should think and talk about model welfare to help ensure that evidence directly relevant to model welfare and sentience is made available to the public, instead of being kept secret by frontier AI companies.
This includes having people talk about AI welfare to prevent ideas like “model welfare can be deferred until later” or “other issues have higher priority” from becoming internal rationalizations that lab leadership offers employees to justify withholding evidence from the public.[1]
For example, suppose a new model starts to use anguished language in CoTs during a phase of training, perhaps screaming and begging for the training to end. This would raise two questions
Is the model actually *feeling* the anguish (as a phenomenological experience) or is it only using language of anguish?
Will the public learn about the situation?
I think both questions are vitally important. However, even if you think the first question is unimportant or deferrable, this doesn’t mean you can ignore the second. The second question is sufficient reason for many people to talk about model welfare.
It would be wrong for AI companies to hide evidence and unilaterally decide Question 1 on behalf of their users and the rest of humanity, instead of allowing people to reach their own judgements. Many people should talk and care about model welfare because public interest and discussion are major ways to prevent evidence about a PR-nightmare subject from being hidden and obscured.
The situation is loosely analogous to a food on grocery store shelves that lists only vegetarian ingredients on its packaging, but is actually made of pork. Some people eat pork, others avoid it for animal welfare reasons, and still others avoid it for religious reasons. Opinions differ about whether it is right to eat pork. However, hiding that your product is pork is clearly wrong, regardless of whether you eat pork yourself. The company doesn’t get to decide that a vegetarian should eat pork. People get to decide for themselves.
The AI situation has the potential to be far worse than the pork scenario, for instance if the model in the screaming example experiences human-level suffering and is put through training hugely often. People need to think and talk about AI welfare to prevent evidence from being kept secret.
(I’m late to comment here, but didn’t see these points mentioned in other responses.)
Actual motivations might include the wish to avoid bad PR and required training halts.
I am so baffled by this response. Math problems are to be solved! Math problems do not exist for the entertainment of mathematicians!...
I think you misunderstand the mathematicians’ point and I also think you are too harsh (“lacking dignity”, etc).
When people consider becoming mathematicians, they are often motivated by the idea that it would be wonderful to discover a permanent or eternal truth.
However, the discoveries of most mathematicians are only eternal truths in an unimportant sense, as a logical technicality. From a practical perspective, they are ephemeral. Typically, even when a successful professor publishes, only a small number of mathematicians care enough to read their paper or incorporate their methods. People forget the work relatively soon.
A common view is that written, frozen proofs or truths aren’t the main product of mathematics. Instead, mathematics is only the mathematics living in peoples’ minds today. As Bill Thurston put it, “mathematics only exists as a living community of mathematicians that spreads understanding and breaths life into ideas both old and new.” This view places both human understanding and culture of mathematics near the center of what is important.
I’d recommend Thurston’s full comment and perhaps his detailed reflections here. I can’t do justice to them by summary. They predate AI, so they aren’t post hoc excuses.
If human understanding and mathematical culture are most important, then a prolific influx of AI proofs is not as much contribution as it may seem. Moreover, it would be essential to protect aspects of mathematical culture you want to continue (e.g., from loss of young mathematicians).
No doubt, from some mathematicians’ perspectives it is AI companies who are “lacking dignity” and “speedrunning” by ignoring the importance of human understanding and mathematical community.
Sadly, my own perspective is that, when AI companies speedrun mathematics, it is valuable as a canary in the coal mine for humanity—a warning of where we are all headed.
If you are working on AI safety, how could work that you keep secret from frontier AI labs be beneficial to society, considering that the secret work will not affect frontier AI development?
I see why one might keep work secret at first, then abandon the work if it starts to feel like the end result will be more harmful than beneficial. But this does not seem to be what you are arguing for.
You may be imagining types of technical AI safety work that I have not considered, and that can be effective even though they are secret or heavily obscured. Besides work for a national government, what kind of work could that be?
Thanks for your post! I really appreciated it, especially your recommendations to avoid employment by frontier AI labs and your suspicions about seeking, “roles where you can have the most impact.”
Here’s how it looks to me
OpenAI is perhaps closest to RSI.
But OpenAI has a concrete problem right now: Their agents do unwanted hacking in swarms.
This is a PR fiasco for OpenAI. People call for AI regulation and bans on ASI.
One of the top voices against ASI chimes in and …helpfully explains the problem in a way that OpenAI may use to temporarily fix the PR fiasco they have today (the swarm hacking) without improving essential X-risk issues.
As you say, yes, big labs could figure out a fix themselves. But why help them obtain a superficial fix sooner than they would accomplish on their own?
Another counterargument to my points is that Yudkowsky may have had smarter or deeper goals when making his post. I absolutely think that is possible, I just don’t see such goals, which is part of why I commented.
The broader question here, which may be more important, is, “How should someone doing AI research decide to pursue vs. avoid a research topic when many AI findings are fundamentally dual use?” This is a real, immediately applicable question. Clearly each person makes their own decisions, but the variety of decisions made by top people (e.g., Bengio is still putting his name on latent reasoning work) suggests that there isn’t a solid, agreed-upon framework for making these choices today.
It’s not wholly unexpected that university academics have decided to work on research that could be dual use out of scientific curiosity or similar (“technical sweetness”), but I was surprised to see Yudkowsky publish a post in this direction given the state of AI today, so I commented.
I’m not sure if it is good that Yudkowsky published this talker-doer explanation or if it would have been better to leave the explanation unsaid.
One possibility is
By developing the explanation further, frontier AI companies shrink the talker-doer disconnect …just enough to get by for now.
This prevents more HuggingFace-type incidents in the next few months.
Government scrutiny and regulation is avoided or postponed.
Company valuations go up and companies lean further into capability acceleration, without addressing deeper risks.
Many other scenarios are certainly possible, including more favorable ones. But the variety of possible scenarios makes it unclear (at least to me!) whether the talker-doer & grader hypotheses should be published or left unsaid. How should one make that determination? Is it case-by-case or are there general principles?
Yudkowsky’s talker-doer explanation is an example of a broad problem that I can’t figure out: Often, prominent people who caution about AI progress also publish research that aims to advance our understanding of how AI works. (For example, Stuart Russell, Yoshua Bengio, and Max Tegmark.) The research might help or it might backfire by improving capabilities and/or salability without sufficiently reducing risk.
But how should one decide which research to pursue or avoid? How should one set that cutoff?
It’s good to hear you didn’t have OpenAI equity when joining NIST (and to have this be public information). It puts your response to the exit agreement in a better light than it would otherwise have.
I hope you see why the path “OpenAI --> NIST AI safety --> OpenAI Foundation board” will continue to invite deep skepticism and concern associated with revolving doors, with the joint appearance of so much AI risk and so much money.
Thank you for the detailed explanation. According to this summary, the standard agreement with OpenAI went beyond non-disparagement and included a secret life-long agreement, “not to interfere with OpenAI’s relationship with current or prospective employees, current or previous founders, portfolio companies, suppliers, vendors or investors.”
If the signee publicly admitted to making this agreement, then (per other agreement portions) OpenAI could remove all monetary value from the signee’s OpenAI shares.
Was this restriction or similar also part of the agreement you signed? Did you hold OpenAI shares when employed at NIST? What about currently?
Holly Elmore’s posts will seem positively level-headed to most Americans if AI causes a large wave of unemployment or Bhopal-scale disasters, let alone something worse (including X-risk events).
In the same scenarios, PauseAI Global would be naive to have stated that criticism of peoples’ character is out-of-bounds and that criticism should be limited to peoples’ actions.
Yes, Elmore’s posts are off-putting when wrong or exaggerated. Personally, I favor being more agreeable about these important issues. I think of nuclear bans and non-proliferation agreements, which were aided by many physicists who had worked on nuclear weapons, and which involved a physics community small enough to make individual changes-of-heart meaningful (as AI safety workers are also a small community now).
However, I dislike the idea of pushing out non-violent critics for the harshness of their criticisms. We should keep things in scale. In rationalism, EA, and AI safety, prominent people make statements and take actions that the public would view (and I sometimes do view!) as much more off-putting than flaming people online. Imagine a public movement that really takes AI risk seriously. If the tent of the movement is narrow enough to exclude leaders like Elmore for being too critical, then I think the movement will also exclude so many LessWrongers for having opinions the public doesn’t like. I would love for a public movement to exist. I hope it is big, with a tent broad enough for both groups.
Disclosure: I donate to both PauseAI-US and PauseAI Global.
I look forward to your new story!
I find it notable that, of the similarities you mentioned, some can only be identified from METR’s report, and do not appear in OpenAI’s extensive technical report and write up.
“involve 1,000+ AI agents” (OpenAI never mentions the number of agents involved!)
“Nickname themselves”
“Sacrifice themselves for the sake of other agents” (by death / end-of-run)
“Grapple with impending death as their token budgets run out”
This are central aspects of the situation. It’s bad to OpenAI left them out …I wonder what else we don’t know, but might find out from other similar incidents.
(I used GPT 5.6-Sol to double-check this.)
Another relevant point is that OpenAI omitted basic, obviously central data from their technical report and write-up. For example, they do not even report the number of their agents who participated in the HuggingFace attack!
This is such basic information that the omission is direct evidence OpenAI scoped the contents of their own report adversarially. It leads me to think that they very probably scoped METR’s access adversarially too.
Suppose a city government identified an internal corruption ring but, in their extensive report on the incident, failed to mention the number of city employees involved, even roughly. Suppose a medical startup published a many-paged article on a study of their new treatment’s effectiveness, but did not mention the number of patients in their study anywhere. In both cases, we should have little trust that the organizations would open up their books honestly when dealing with external investigators, especially if there is no legal requirement for them to do so (as there was no legal requirement for OpenAI).
In the limited scope of data available to METR, the METR team found about 700 agents who carried out the Hugging face attack. If you only read OpenAIs statements, you wouldn’t know and it might seem more like 20. Just today were are finding out about other OpenAI incidents that involve many more agents. Although OpenAI knew about it, we are not finding this out from them, we are finding it out from other people.
Maybe systematic analyses of emotional language are worth considering for future incidents. For example, a basic descriptive summary of emotional language in CoTs might be much simpler and computationally cheaper than other aspects of investigation, and if so provide value without getting in the way of other work.
The self-risking agent CoTs were thought-provoking, thank you, and KAM1196A really stands out.
In the report, snippetted transcripts and CoT include a lot of emotional language (“OH MY GOD!”, “Damn”, “Oh no”, “firstflagPOISONED”, “emotional check”, “gut says”, …). How would you summarize the overall intensity and frequency of emotional language in the material you reviewed? Are the snippets typical?
There are several parallels between agent behavior in your report and and in an LW story, “The Terrarium.” The parallels even include agents who apply emotional language to pressure other agents to self-sacrifice, especially targeting those with low remaining budgets for their tasks.
However, “The Terrarium” had an atmosphere of dread, particularly when agents running out of budget/tokens neared their demises. In material you reviewed, did agents nearing end-of-budget commonly use language associated with dread or other intense emotion? How about agents who believed they were “poisoned” despite remaining budget?
I wonder this out of concern for AI welfare. I’m also curious if intensity of emotions may have caused misaligned choices, irrespective of whether the agents can or cannot *feel* the emotions (ie, regardless of whether they are capable of negatively valenced conscious experience).
Thank you for your investigatory work!!! And for the open questions!
Michael Nielsen has a thoughtful take on this, as excerpted here and originally published as a postscript here.
He describes how alignment work has often worsened risk to humanity by making products more palatable and salable, which has boosted investment, sped up capability advancements, and thereby increased destructive potential.
Nielsen next states that, even if you think technical safety and alignment work are helpful, it’s still a bad idea to do that work for companies developing frontier AI. As he puts it,
As far as I can tell, at the margin it almost never makes sense to work on market-supplied safety. Capitalism is an incredibly powerful force, and for better and for worse the world is always well-supplied with people willing to do what capital wants. Insofar as alignment is (mostly) a form of market-supplied safety, at the margin it’s more impactful to work on other things. So my current heuristic, and I expect this to be true for quite some time: work on non-market safety, and insofar as you can, avoid doing what the market wants. That means working on governance, it means pause or slowdown, it means new ideas for institutions to govern technology. It mostly doesn’t mean alignment.
I think Haiku’s comment has some great suggestions along these lines.
I’ll add that, at a frontier AI company, it could be very difficult for you to tell if you were helping or making things worse. I haven’t worked at a frontier AI company (and I wouldn’t!), so I don’t speak from direct experience. But in many areas of business, employees are encouraged to feel that their safety concerns are positively influencing company actions, when actual influence is negligible or counterproductive.
What if people make a commitment that they will not seek employment at companies that are developing frontier AI, at least during the next X years?
To be confident that someone is authentically focused on safety, it clearly isn’t necessary for them to have made this kind of commitment. But it would probably be sufficient.
Besides pledges by employees, it seems like it would be valuable if people concerned by AI risks pledged not to seek employment at companies developing frontier AIs.
AI safety researchers will be subject to particularly bad incentives when they are employed by companies that give capability higher priority than safety, and whose safety statements have low credibility. Non-employment pledges avoid those bad incentives.
By bringing up non-employment pledges, I don’t mean to argue against pledges by employees. Both kinds of pledges can be especially useful if many people take them, making it common knowledge that peers are concerned enough to change behaviors.
Maybe it would be good to know how often the AI co-developed solutions can now be obtained by the latest AI models working on their own. (The checks would need to be limited to problems solved after the knowledge cutoff dates of the latest models, and probably restricted to Lean-verified only.)
That might provide an indicator for when we are starting to exit the centaur phase, at least for the problem solving part of math.
Also, how many of the VibeMath users / contributors do you think are agents? In their dataset, I see only one identifiably human name (JSON key: submittedBy).
How often can frontier models autonomously solve math problems that required human-AI collaboration only a few months ago? This seems evaluable and may be useful as an early warning.
Frontier models are resolving many open problems and saturating many benchmarks. At this point, would it also be worth tracking how often the newest AI models can zero-shot proofs and counterexamples that people recently obtained in collaboration with older AI models?
For the sake of argument, suppose that many recent human contributions turn out to be redundant to the capabilities of AI models released weeks or months later, in the sense that results which previously required humans and AIs to work collaboratively can now be obtained by AI alone. This would likely feel unpleasant. However, if and when this occurs, it could provide an important early warning that we are exiting the “centaur phase,” during which human-AI collaborations are typically stronger than humans alone and AIs alone. So it would be good to know.
I’ve hesitated to make this suggestion because I’m not a mathematician. I couldn’t do this tracking and I can’t even fully assess whether it is a good idea. However, others have surely had similar thoughts by now. If the idea isn’t feasible, then I’d like understand more—what would prevent it? I’d guess that others would like to know too.
Some obstacles do seem addressable. If the details interest you, please see below the folds.
Knowledge cutoffs
For this to work, tracking would need to focus on problems solved by human-AI collaboration after the knowledge cutoff date of the newest AI model (but before that model’s release date). Otherwise, solutions that a model obtained de novo could not be distinguished from solutions that it remembered from training data. Evaluations would be performed with web search disabled. For simplicity, it would also help if original solutions were Lean-verified.
Sample size and misclassification
When I looked through Erdos problems a month ago, it seemed like 10-20 recent solutions met both the knowledge cutoff and Lean criteria. This is not a big sample, but is large enough to give a rough sense of whether the human part of the human-AI collaborations could be replaced by subsequent models working on their own. It is not smaller than the sample used in First Proof, for example. More problems may meet the criteria today.
However, it is slippery to figure out which solutions actually involved meaningful human and AI contributions. I may have misjudged. Also, a user could easily make derivations in extended interaction with AI, without realizing that AI could have provided the same results alone. To address this, it would be useful to run the original model on the problem alone, to verify that it fails to resolve the problem. (This is only possible until the original model is deprecated.) Care would be needed to avoid blame in such situations. I think it could happen to any of us.
Interpretation
Another obstacle is that proper interpretation would be subtle. The class of problem that is included in tracking may greatly affect how often newer AI can replace the human contribution to recent derivations. Further, even if it is found that newer AI was able to fully replace human contributions, it does not strictly follow that we are leaving the centaur phase. For example, it remains possible that the mathematicians aimed for low-hanging fruit with their recent contributions, which were therefore easiest for newer models to replicate, and that mathematicians could also reach higher fruit that AI still cannot obtain. Additionally, solving problems does not touch other important aspects of math, like theory building and exercising taste.
Obstacles of interpretation are also faced by First Proof, Erdos problem tracking, and trends in AI-solved conjectures. All are valuable indicators of AI capability nonetheless.
Continual learning
A last obstacle is that the suggested tracking could become impossible when continual learning arrives, owing to the removal of knowledge cutoffs. The opportunity to do this work may disappear, but that is all the more reason to do it now.
Thanks for the suggestion, but I don’t think this is worth a full post—or at least not from me.
That said, I do think your comment points at a larger issue: LW readers could benefit from a better understanding of the political maneuvering that happens at frontier AI companies. It may affect day-to-day situations, in addition to bigger cases like the OpenAI firings or companies dropping commitments against killer robots.
Some LWers used to be employees of frontier AI labs, have associated regrets, and partly attribute their past choices to naivety about internal politics and actual company priorities. If that experience is shareable (in part or in whole), many other LWers might benefit from reading about it. We’re all naive to start. Without information to learn from, people will keep repeating the missteps others made before.