Don’t call it a “pause”, as that messages that a pause is much weirder than it is
Eric Schmidt on CNN: “we’re not going to pause AI progress because the entire incentive system in America is not to pause.” And he added, it’s hard to enforce a ban across all labs.
The part about inter-lab enforcement is important, but that’s not the only thing he said. He also said, labs lucid about x-risk are incentivized to not pause. This is a rather bald-faced lie, as corporations are allowed to not do things that might destroy the corporation.
If p(doom) = 100% and it’s common information (everyone knows everyone knows, ad infinitum), we simply use the word “can’t”, and we don’t think twice about it. Someone still could, but that person is considered crazy, and our regulatory question is how to stop them.
I’m handwaving a lot of game theory here[1], but I don’t crux on it when I say “don’t call it a pause”. In terms of the public debate, I’d like Eric Schmidt to assume burden of proof that the labs are required to do the dumb thing. I don’t think he should get to hide behind the word “pause” when dismissing a pause as nonsense.
More shared knowledge to not do the dumb thing, is good, because we don’t pay an overhead from mismodeling each other’s incentives, and we can’t disturb a “don’t build it” equilibrium as easily.
Talking about a pause dilutes that shared knowledge. As long as the very concept of “pausing” is in our lexicon, rushing to build unsafe ASI is the default.
- ^
They all have different p(doom), which is an overly coarse concept, and additional x-risk for being the 2nd lab to act is a factor, and common information is shaky, and I don’t know how to model what investors want. Plus, I don’t know how to model, why did Eric Schmidt benefit from framing it like that, if he knows about x-risk in the first place.
What is “the dumb thing”, specifically? I’m imagining that training or releasing GPT-6 Astra or Claude Fable 5.1 wasn’t “the dumb thing”, because those are fantastically valuable commercial products that have created enormous amounts of economic value (far in excess of the damage caused by the reported hacking incidents in evaluations). Or was it? Maybe GPT-4 was the dumb thing, and everything after that was even dumber? What is it?
To be clear, I personally agree that the near-term extinction risk from advanced AI is terrifying, and that a pause or slowdown would be good. But in order to appeal to “shared knowledge” that a pause or slowdown is the default, non-insane action (such that “pause” language is misleadingly marking what should be unmarked), you should be able to answer this kind of specific, nitpicky question. Otherwise, you don’t have “shared knowledge” or “common information”; you’re just pushing your preferred frame like any other political actor. Maybe you think that’s the winning move, but I think we might still have need for shared knowledge of the actual meaning of shared knowledge, even at this late hour.
Yeah, that’s what I meant by “messages,” but fair enough, I should defend if it’s better epistemics or just a trick that won’t stick. So yeah I do think it’s better.
Option A: Lab CEOs commit to a first-ever pause/slowdown/pacing, Schmidt and others dismiss it as nonsense violating basic capitalism sense, it’s voluntary, it leads to charges of collusion, and they’re actually getting sued for collusion.
Option B: We can’t make them any bigger without serious risk, we don’t know how, so we’re not. Lab CEOs meet to solve alignment to speed up development again, which is not an antitrust violation.
Option A invents a new concept for do something unremarkable, and puts burden of proof on whoever says “pace”, even though pacing is just a new word for something unremarkable.
Sam Altman did make a big concession toward Option B on 9/16: ( https://openai.com/index/responding-next-frontier-critical-cyber-capabilities/)
I think I can answer that the dumb thing is “scaling at maximum speed for much longer.” It’s still coarse and more shared knowledge about where that line is would be great, but it’s good enough that I’d like to hold Sam accountable to it.
Did they? I asked Claude (Opus) about this and it leaned mostly toward this not being true, but that they’re too young for this to be meaningful. It mentioned Fable being too expensive to be widely used in coding. Would/should investment (like into data centers) count as generating value?
This doesn’t seem cruxy for anything so feel free to ignore if it’s too complicated to answer.
Anthropic and OpenAI are pulling in revenue at the rate of tens of billions of dollars per year; even if most of that is selling cheaper Opus/Sol tokens rather than expensive Fable/Astra tokens, that’s still a lot of money and therefore implied economic value. (I just used Astra as a travel agent to plan a trip to visit my girlfriend in Australia.) It would be more complicated to estimate the damages from the Hugging Face incident and RubyGems signups being down or four days, but I’m not inclined to regret my initial comment on your Opus’s say-so.