Would we be less likely to go extinct if OpenAI forcibly took over Anthropic? (a possible hypothetical: a combined effort of OpenAI and the US government forces a merger). My take: no.
People keep making arguments shaped like this, and I feel that there’s some error here.
Currently, for race dynamics to dissolve, two LLM megacorps need to cease to be relevant. If Anthropic weren’t founded or were destroyed/absorbed, only one LLM megacorp would need to lose relevance. Destroying one megacorp is easier than destroying two. Therefore, in worlds in which Anthropic didn’t exist or were destroyed, it would be easier to dissolve the race dynamics altogether. Therefore, it’s bad that Anthropic exists and it would be better if it did not exist.
That seems straightforward to me? We are less likely to go extinct in Anthropic-less worlds because those worlds are more likely to also become OpenAI-less and/or DeepMind-less.
There seems to be some underlying assumption that this is a one-turn game, where you’re asked to choose between “implement Intervention X and then do nothing forever” and “do nothing forever”. I’m not sure why “do nothing forever” is implicitly tacked on.
(Arguments like “I, a capability researcher, don’t need to quit my AGI lab, because I will just be replaced by an identical capability researcher, for a net zero change in anything” seem from the same genre.)
Even if another megacorp doesn’t leave the race, companies have a much larger incentive to invest in safety in a 2-actor race than a 3-actor race, because they internalize more of the benefit. At a minimum, they’re 1.5x more likely to win, plus it’s easier to coordinate between 2 actors than 3.
I don’t think destroying one megacorp is easier than destroying two, each of half the size. My model: it’s easier to destroy a hundred companies with one employee than to destroy one company with a hundred employees, if one is coming in with an external force.
Conditional on OpenAI forcibly taking over Anthropic, it is more likely that Alphabet forcibly takes over OpenAI (or OpenAI forcibly takes over the AI portions of Alphabet). I was more thinking of “what if a freak event caused a forced takeover” rather than “what if corporate murder of AI labs became normal”.
Its not clear to me that shutting company 1 and then shutting company 2 later in time is easer than shutting both companies at the same time in the future. I dont really wanna argue for any conclusion though.
Some “intuition pump” of where this pattern dosent hold: if you want to stop 2 snowballs from rolling down the hill stopping only 1 will make the other grow faster and you wont be able to stop it at all. Snowballs collide with eachother and slow themselves down.
People keep making arguments shaped like this, and I feel that there’s some error here.
Currently, for race dynamics to dissolve, two LLM megacorps need to cease to be relevant. If Anthropic weren’t founded or were destroyed/absorbed, only one LLM megacorp would need to lose relevance. Destroying one megacorp is easier than destroying two. Therefore, in worlds in which Anthropic didn’t exist or were destroyed, it would be easier to dissolve the race dynamics altogether. Therefore, it’s bad that Anthropic exists and it would be better if it did not exist.
That seems straightforward to me? We are less likely to go extinct in Anthropic-less worlds because those worlds are more likely to also become OpenAI-less and/or DeepMind-less.
There seems to be some underlying assumption that this is a one-turn game, where you’re asked to choose between “implement Intervention X and then do nothing forever” and “do nothing forever”. I’m not sure why “do nothing forever” is implicitly tacked on.
(Arguments like “I, a capability researcher, don’t need to quit my AGI lab, because I will just be replaced by an identical capability researcher, for a net zero change in anything” seem from the same genre.)
Even if another megacorp doesn’t leave the race, companies have a much larger incentive to invest in safety in a 2-actor race than a 3-actor race, because they internalize more of the benefit. At a minimum, they’re 1.5x more likely to win, plus it’s easier to coordinate between 2 actors than 3.
I don’t think destroying one megacorp is easier than destroying two, each of half the size. My model: it’s easier to destroy a hundred companies with one employee than to destroy one company with a hundred employees, if one is coming in with an external force.
Conditional on OpenAI forcibly taking over Anthropic, it is more likely that Alphabet forcibly takes over OpenAI (or OpenAI forcibly takes over the AI portions of Alphabet). I was more thinking of “what if a freak event caused a forced takeover” rather than “what if corporate murder of AI labs became normal”.
Thanks for the thoughtful response.
Its not clear to me that shutting company 1 and then shutting company 2 later in time is easer than shutting both companies at the same time in the future. I dont really wanna argue for any conclusion though.
Some “intuition pump” of where this pattern dosent hold: if you want to stop 2 snowballs from rolling down the hill stopping only 1 will make the other grow faster and you wont be able to stop it at all. Snowballs collide with eachother and slow themselves down.