Yes, it’s an unusual risk. But I hope that no one will release access to AI models who in turn can create weapons. The models could either be open-sourced and bring with themselves many other risks[1] or be closed-sourced and used in a misaligned way, causing potential lawsuits. @Daniel Kokotajlo, does the latter mean that no corp will release weapons-creating AIs to anyone aside from the corp’s government? Or Grok 5 or 6 will create some weapons and cause mankind to shut xAI down?
Including a potential international race to protect mankind from rogue AIs. I hope that Kokotajlo’s team will eventually provide feedback to all the scenarios it received, including the Rogue Replication Scenario, and determine whether the international race is a plausible result.
I’ll talk more about this in follow up posts, but I don’t think the main danger is that the models will be voluntarily released. Instead, it’ll just get cheaper and cheaper to train the models that have weapons capabilities as the algorithms get more efficient, which will eventually democratize those weapons.
Analogously, we can think about how cryptography was once a government controlled technology because of its strategic implications, but became widespread as the computing power required to host cryptographic algorthims became extremely cheap. In my view, something very similar is likely to happen with AI systems, although there will also be a more active effort to steal the weights rather than just replicate them given the strategic implications.
Yes, it’s an unusual risk. But I hope that no one will release access to AI models who in turn can create weapons. The models could either be open-sourced and bring with themselves many other risks[1] or be closed-sourced and used in a misaligned way, causing potential lawsuits. @Daniel Kokotajlo, does the latter mean that no corp will release weapons-creating AIs to anyone aside from the corp’s government? Or Grok 5 or 6 will create some weapons and cause mankind to shut xAI down?
Including a potential international race to protect mankind from rogue AIs. I hope that Kokotajlo’s team will eventually provide feedback to all the scenarios it received, including the Rogue Replication Scenario, and determine whether the international race is a plausible result.
I’ll talk more about this in follow up posts, but I don’t think the main danger is that the models will be voluntarily released. Instead, it’ll just get cheaper and cheaper to train the models that have weapons capabilities as the algorithms get more efficient, which will eventually democratize those weapons.
Analogously, we can think about how cryptography was once a government controlled technology because of its strategic implications, but became widespread as the computing power required to host cryptographic algorthims became extremely cheap. In my view, something very similar is likely to happen with AI systems, although there will also be a more active effort to steal the weights rather than just replicate them given the strategic implications.