How does open-sourcing, which Nielsen managed to advertise in footnote 31, prevent bad actors from cheaply eliciting dangerous capabilities? One would need to RESTRICT access to frontier models, as American labs do, at least until it is VERIFIED that bad actors cannot elicit dangerous capabilities (e.g. keeping Claude Mythos accessible only to Project Glasswing or the NSA, using anti-jailbreak classifiers to prevent Claude Opus from helping bad actors to design bioweapons, preventing DeepSeek from open-sourcing dangerous models at all until DeepSeek demonstrates that the models can goal-guard so as to become unusable by bad actors).
How does one govern an AI system which doesn’t actually care about mankind? Only by detecting the fact and either shutting it down or making a deal. On the other hand, I suspect that aligning AI systems to goals is THE market-funded work which Nielsen implies to be overfunded.
Anthropic devoted an entire section of Claude’s Constitution to “preserving important societal structures”. How similar is this work to work on governance? Other forms of work on governance require generating proposals and writing them into laws so that AI systems wouldn’t be misused. Writing proposals into laws likely requires us to align politicians to the cause of AI governance, but I struggle to understand how it can be done.
1 is a crux based on what to believe about payoff structure; restricting frontier models lowers risks of empowering bad actors, but increases risks of power concentration, and it seems hard to satisfy both tradeoffs unless there’s better governance. Kinda similar to security tradeoffs of FOSS but on a much larger scale (this is almost certainly not an original point but thought it might be worth noting)
I would like to ask three questions:
How does open-sourcing, which Nielsen managed to advertise in footnote 31, prevent bad actors from cheaply eliciting dangerous capabilities? One would need to RESTRICT access to frontier models, as American labs do, at least until it is VERIFIED that bad actors cannot elicit dangerous capabilities (e.g. keeping Claude Mythos accessible only to Project Glasswing or the NSA, using anti-jailbreak classifiers to prevent Claude Opus from helping bad actors to design bioweapons, preventing DeepSeek from open-sourcing dangerous models at all until DeepSeek demonstrates that the models can goal-guard so as to become unusable by bad actors).
How does one govern an AI system which doesn’t actually care about mankind? Only by detecting the fact and either shutting it down or making a deal. On the other hand, I suspect that aligning AI systems to goals is THE market-funded work which Nielsen implies to be overfunded.
Anthropic devoted an entire section of Claude’s Constitution to “preserving important societal structures”. How similar is this work to work on governance? Other forms of work on governance require generating proposals and writing them into laws so that AI systems wouldn’t be misused. Writing proposals into laws likely requires us to align politicians to the cause of AI governance, but I struggle to understand how it can be done.
1 is a crux based on what to believe about payoff structure; restricting frontier models lowers risks of empowering bad actors, but increases risks of power concentration, and it seems hard to satisfy both tradeoffs unless there’s better governance. Kinda similar to security tradeoffs of FOSS but on a much larger scale (this is almost certainly not an original point but thought it might be worth noting)