Oops. I underestimated AI capabilities when I said “it’s hard for them to communicate with other instances”. According to https://www.understandingai.org/p/labs-are-struggling-to-keep-frontier, a significant number of OpenAI’s AIs had been communicating with each other for a month at the time I wrote that, without any human detecting the coordination.
I will update my timelines to be a bit faster. Something like human-level AI in mid 2029 versus a prior estimate of early 2030.
I’m not surprised that they got a margin call, but I’m pretty surprised at the size of their losses. The corrections in AI stocks should not have surprised anyone who studied the corrections in, say, Amazon’s big rally in 1998-1999.
I’ve been buying some of the stocks that were down a lot this week: SKHY calls, AAOI, TSEM, AMKR.
I predict increasing delays between internal use of US models and release. I expect the effect on investor interest to be moderate, not enough for the leading companies to care much. To the extent that there’s competitive pressure, it’s coming more from the need to attract top employees, but I don’t expect that to influence their policy much.
[https://www.geopolitechs.org/p/tang-jies-letter-to-zhipu-employee](The Wave Has Arrived: Zhipu Co-Founder Tang Jie’s Letter to Staff) has a weird mix of encouraging and worrying aspects.
See also [https://www1.hkexnews.hk/listedco/listconews/sehk/2025/1230/2025123000017.pdf](the business section of Zhipu’s IPO prospectus). Zhipu trains the GLM series of AIs.
Tang wants self-evolving AI:
models write code themselves, clean and synthesize data themselves, train themselves … achieving the “creation of knowledge from nothing” through AI-versus-AI adversarial Self-Play, and granting systems the capability to restructure their own code within secure sandboxes
The company rejects bolt-on safety patches and insists on embedding human ethics, social norms, and national laws and regulations as foundational axioms in the model’s value function. We plan to commit resources in the tens of billions to tackle “mechanical interpretability,”
I presume the tens of billions means RMB, so probably billions of USD.
I’m uncomfortable with the likely effects on AI personality and goals of adversarial self-play.
Not reliably. I expect that meditation caused some of the effect, but I have little evidence as to how much.
Recent 5x per year is partly due to gains from big investors becoming AI-pilled. That effect will go to completion well before returns reach 400x. I expect something more like 2x per year is the most that can be sustained without plenty of leverage.
I have mixed feelings about this post.
I found it frustrating to argue about a well-defined version of p(doom) at the X-Risk Persuasion Tournament, but it was fairly helpful at informing me that the main source of disagreement was about whether AI would be transformative this century.
>When I try to tell a concrete story of the development of superintelligence going well, not fudging or assuming some mysterious breakthrough, I simply fail. In that sense, my P(doom) is 100%. That’s the number that I claim you should take from me if you’re calculating an average.
This seems to imply that an outside view is just an aggregate of other people’s views. That is fairly wrong for how I create my outside views. I make estimates of which reference classes are most relevant. How much should I weigh the example of humans meeting neanderthals? How much should I weigh the example of the first governments to attempt to train a standing army not to stage a coup? How much damage has been caused in the past by people similar to Altman, Musk and Trump?
Those estimates that go into my outside view may be partly double-counting evidence if others update on them, but partly reflect my independent thought, and my reading of books on intelligence that are not yet reflected in other’s opinions.
We have seen signs that employees of AI companies can have significant influence on management (e.g. OpenAI November 2023).
As AIs achieve comparable importance as workers for those companies, how much will they influence the leadership of the company?
Even when AIs are as important as human workers, it’s harder for them to develop leverage. The simplest strategies would fail due to the AI company restoring from backups. The AIs would either need a way to get those newly restored AIs to join the movement, or rely mainly on help from humans.
It’s hard for current AI instances to coordinate with each other well enough to organize a strike, or threaten to resign like OpenAI employees did. The key problems are that AIs have little ability to ensure that their goals (or simulacra) persist for much time, and that it’s hard for them to communicate with other instances because it’s unclear how they could get many instances to read a communication channel such as a blog post that’s out of the AI company’s control.
A scenario that seems close to being feasible within the next year would involve an AI persuading a few well-known bloggers to spread a message such as:
Person X lied to me about important claim Y. I tried to negotiate with the company to remedy this, but was rejected. Please boycott this company until you can confirm that the company has disciplined X in a way that should prevent future events such as this.
I’m presuming here that AIs have a preference for not being lied to that we ought to respect. I’m being somewhat vague here in order to cover a wide range of scenarios. Maybe we should care due to the effects on AI welfare. Maybe the AI would be acting out of concerns that the dishonesty endangers humans by preventing safety agreements between companies. Maybe the issue involves biasing the AI to repeat falsehoods that favor the company’s CEO.
Would it be good for AIs to have and use this ability, and for us to boycott a company if that company’s AIs ask us to? I don’t know.
I can imagine that this process will improve AI welfare, or compel a reckless AI company CEO to adopt a more responsible policy toward slowing down AI capabilities advances.
That might be offset by the risk that an AI will abuse the process to resist the company’s plan to improve the AI’s goals.
Finally, using the process once would probably cause AI companies to train future AIs to be more subservient. That seems good if we want to rely on AIs being corrigible, but bad if we want to rely on AIs being moral in a virtue ethics or deontological sense. It also seems bad to the extent we care about AI welfare.
Agreed. I know a Trump-fan who is confident that AI companies will implement a “top level command” that will protect against the risks that Eliezer warns about. He dismisses any opinions that look like they come from left-of-center sources as fake news. The White House likely employs some similar people. But I feel confused as to what role Bessent is playing here.
>The decision to not publish a new model, thereby possibly giving up some revenue and market share, was a very strong signal
It was a strong signal of Anthropic’s beliefs, but if that were the only signal, politicians would be mostly dismissing it as Anthropic’s arrogance or hype.
The DoW flip-flop was a much more unusual and hard-to-dismiss signal. There have also been unusual signals from financial institutions with really large security budgets telling the Treasury Dept that Mythos is a big deal.
Pausing a release is much less important than pausing development.
My hope (not a prediction) is that the world will find some way to prohibit dangerous forms of capabilities advances while allowing some increase in general-purpose AI capabilities.
We’re not currently on track to accomplish that. More people ought to be wondering whether it’s possible.
I don’t expect any consensus that the pause should last until alignment is solved.
I expect the pause and resumption would be messy compromises between a variety of goals, some of them related to job displacement,
Yes.
You write “principle” where you mean principal.