If AI development is more insight-driven than compute-driven, then there is more room for sudden progress that gains a decisive advantage over other labs and govs before getting noticed (other entities suspecting the lab getting close to ASI with non-neglible confidence) and reacting. This allows the lab to control the singleton instead of the mainstream labs and govs, and in this situation, the lab might escape from race dynamics.
However, this scenario results in a random lab controlling a singleton. While it’s not as hopeless as a singleton built by a racing entity, it doesn’t look really hopeful either.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity. However, it’s still difficult to ensure the good intentions are still there when they actually get to ASI.
I think there is an interesting middle ground here too. You have the dichotomy Insight versus compute driven.
I would generally agree with your assessment.
Within the compute driven paradigm I’d propose a sub delineation. Intelligence able to generate novel insights versus intelligence able to succeed only in highly specified domains.
It is possible that LLM intelligence only scales in verifiable domains, but not in “insight” driven domains where you end up with a an AI that can solve math problems, identify vulnerabilities, and code in a vastly superhuman way, but is notably hobbled in other domains.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity.
FWIW, I think that this has mostly the effect of just adding fuel to the fire, because the government takes over the project regardless of the intentions of the company.
For this to be different, the “insight” would have to accelerate progress to ASI so much that the company can build ASI in a very short time while staying under the radar, including having very few employees and using little compute.
If you buy my arguments in the section “Why Technical AI safety agendas do not address this problem”, the appearance of such an insight would actually be extremely bad: AI safety is always more bottlenecked on humans compared to capabilities, and this company has very few humans!
Partially agree.
If AI development is more insight-driven than compute-driven, then there is more room for sudden progress that gains a decisive advantage over other labs and govs before getting noticed (other entities suspecting the lab getting close to ASI with non-neglible confidence) and reacting. This allows the lab to control the singleton instead of the mainstream labs and govs, and in this situation, the lab might escape from race dynamics.
However, this scenario results in a random lab controlling a singleton. While it’s not as hopeless as a singleton built by a racing entity, it doesn’t look really hopeful either.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity. However, it’s still difficult to ensure the good intentions are still there when they actually get to ASI.
I think there is an interesting middle ground here too. You have the dichotomy Insight versus compute driven.
I would generally agree with your assessment.
Within the compute driven paradigm I’d propose a sub delineation. Intelligence able to generate novel insights versus intelligence able to succeed only in highly specified domains.
It is possible that LLM intelligence only scales in verifiable domains, but not in “insight” driven domains where you end up with a an AI that can solve math problems, identify vulnerabilities, and code in a vastly superhuman way, but is notably hobbled in other domains.
FWIW, I think that this has mostly the effect of just adding fuel to the fire, because the government takes over the project regardless of the intentions of the company.
For this to be different, the “insight” would have to accelerate progress to ASI so much that the company can build ASI in a very short time while staying under the radar, including having very few employees and using little compute.
If you buy my arguments in the section “Why Technical AI safety agendas do not address this problem”, the appearance of such an insight would actually be extremely bad: AI safety is always more bottlenecked on humans compared to capabilities, and this company has very few humans!