For me the missing argument in this comment thread is the following: Has anyone spelled out the arguments for how it’s supposed to help us, even incrementally, if one AI lab (rather than all of them) drops out of the AI race? Suppose whichever AI lab is most receptive to social censure could actually be persuaded to drop out; don’t we then just end in an Evaporative Cooling of Group Beliefs situation where the remaining participants in the race are all the more intransigent?
Has anyone spelled out the arguments for how it’s supposed to help us, even incrementally, if one AI lab (rather than all of them) drops out of the AI race?
An AI lab dropping out helps in two ways:
timelines get longer because the smart and accomplished AI capabilities engineers formerly employed by this lab are no longer working on pushing for SOTA models/no longer have access to tons of compute/are no longer developing new algorithms to improve performance even holding compute constant. So there is less aggregate brainpower, money, and compute dedicated to making AI more powerful, meaning the rate of AI capability increase is slowed. With longer timelines, there is more time for AI safety research to develop past its pre-paradigmatic stage, for outreach effort to mainstream institutions to start paying dividends in terms of shifting public opinion at the highest echelons, for AI governance strategies to be employed by top international actors, and for moonshots like uploading or intelligence augmentation to become more realistic targets.
race dynamics become less problematic because there is one less competitor other top labs have to worry about, so they don’t need to pump out top models quite as quickly to remain relevant/retain tons of funding from investors/ensure they are the ones who personally end up with a ton of power when more capable AI is developed.
I believe these arguments, frequently employed by LW users and alignment researchers, are indeed valid. But I believe their impact will be quite small, or at the very least meaningfully smaller than what other people on this site likely envision.
And since I believe the evaporative cooling effects you’re mentioning are also real (and quite important), I indeed conclude pushing Anthropic to shut down is bad and counterproductive.
the smart and accomplished AI capabilities engineers formerly employed by this lab are no longer working on pushing for SOTA models/no longer have access to tons of compute/are no longer developing new algorithms to improve performance
For that to be case, instead of engineers entering another company, we should suggest other tasks. There are very questionable technologies shipped indeed (for example, social media with automatic recommendation algorithms) but someone would have to connect the engineers to the tasks.
I agree with sunwillrise but I think there is an even stronger argument for why it would be good for an AI company to drop out of the race. It is a strong jolt that has a good chance of waking up the world to AI risk. It sends a clear message:
We were paper billionaires and we were on track to be actual billionaires, but we gave that up because we were too concerned that the thing we were building could kill literally everyone. Other companies are still building it. They should stop too.
I don’t know exactly what effect that would have on public discourse, but the effect would be large.
A board firing a CEO is a pretty normal thing to happen, and it was very unclear that the firing had anything to do with safety concerns because the board communicated so little.
A big company voluntarily shutting down because its product is too dangerous is (1) a much clearer message and (2) completely unprecedented, as far as I know.
In my ideal world, the company would be very explicit that they are shutting down specifically because they are worried about AGI killing everyone.
For me the missing argument in this comment thread is the following: Has anyone spelled out the arguments for how it’s supposed to help us, even incrementally, if one AI lab (rather than all of them) drops out of the AI race? Suppose whichever AI lab is most receptive to social censure could actually be persuaded to drop out; don’t we then just end in an Evaporative Cooling of Group Beliefs situation where the remaining participants in the race are all the more intransigent?
An AI lab dropping out helps in two ways:
timelines get longer because the smart and accomplished AI capabilities engineers formerly employed by this lab are no longer working on pushing for SOTA models/no longer have access to tons of compute/are no longer developing new algorithms to improve performance even holding compute constant. So there is less aggregate brainpower, money, and compute dedicated to making AI more powerful, meaning the rate of AI capability increase is slowed. With longer timelines, there is more time for AI safety research to develop past its pre-paradigmatic stage, for outreach effort to mainstream institutions to start paying dividends in terms of shifting public opinion at the highest echelons, for AI governance strategies to be employed by top international actors, and for moonshots like uploading or intelligence augmentation to become more realistic targets.
race dynamics become less problematic because there is one less competitor other top labs have to worry about, so they don’t need to pump out top models quite as quickly to remain relevant/retain tons of funding from investors/ensure they are the ones who personally end up with a ton of power when more capable AI is developed.
I believe these arguments, frequently employed by LW users and alignment researchers, are indeed valid. But I believe their impact will be quite small, or at the very least meaningfully smaller than what other people on this site likely envision.
And since I believe the evaporative cooling effects you’re mentioning are also real (and quite important), I indeed conclude pushing Anthropic to shut down is bad and counterproductive.
For that to be case, instead of engineers entering another company, we should suggest other tasks. There are very questionable technologies shipped indeed (for example, social media with automatic recommendation algorithms) but someone would have to connect the engineers to the tasks.
I agree with sunwillrise but I think there is an even stronger argument for why it would be good for an AI company to drop out of the race. It is a strong jolt that has a good chance of waking up the world to AI risk. It sends a clear message:
I don’t know exactly what effect that would have on public discourse, but the effect would be large.
Larger than the OpenAI board fiasco? I doubt it.
A board firing a CEO is a pretty normal thing to happen, and it was very unclear that the firing had anything to do with safety concerns because the board communicated so little.
A big company voluntarily shutting down because its product is too dangerous is (1) a much clearer message and (2) completely unprecedented, as far as I know.
In my ideal world, the company would be very explicit that they are shutting down specifically because they are worried about AGI killing everyone.
I make the case here for stopping based on deontological rather than consequentialist reasons.