At a minimum, it [moratorium] would require establishing strong international institutions with effective control over the access to and use of compute.
Note that for this to be an actually long-term victory, you can’t just control large scale compute resources. This requires controlling even personal computers. Algorithmic progress will continue to advance what can be done with a personal home computer or smartphone. This wouldn’t be an immediate problem, but you’d need to have a plan which involved controlling securely, confiscating or destroying every computer and smartphone and preventing any uncontrolled computers from being made.
That’s not the way I usually hear this discussed when I hear people endorse a moratorium. They usually talk about just non-personal large scale computing resources like data centers.
The true moratorium is the full Butlerian Jihad. No more computer chips anywhere, unless controlled so thoroughly you are willing to bet the existence of humanity on that control.
This would depend on whether algorithmic progress can continue indefinitely. If it can, then yes the full Butlerian Jihad would be required. If it can’t, either due to physical limitations or enforcement, then only computers over a certain scale would be required to be controlled/destroyed.
Can it continue indefinitely? No, infinity is big.
Can it continue far enough that a single laptop computer can host a powerful model? Pretty sure most technical experts are going to agree that that seems feasible in theory.
After that, it’s a question of offense-defense balance. Currently, one powerful uncontrolled model can launch an attack that can wipe out the vast majority of humanity. Currently, having lots of similarly powerful models working for the good guys doesn’t stop this. Defensive acceleration seeks to change this balance. Will offense continue to dominate in the future? If so, we face precarious times ahead.
Note that for this to be an actually long-term victory, you can’t just control large scale compute resources. This requires controlling even personal computers. Algorithmic progress will continue to advance what can be done with a personal home computer or smartphone. This wouldn’t be an immediate problem, but you’d need to have a plan which involved controlling securely, confiscating or destroying every computer and smartphone and preventing any uncontrolled computers from being made.
That’s not the way I usually hear this discussed when I hear people endorse a moratorium. They usually talk about just non-personal large scale computing resources like data centers.
The true moratorium is the full Butlerian Jihad. No more computer chips anywhere, unless controlled so thoroughly you are willing to bet the existence of humanity on that control.
This would depend on whether algorithmic progress can continue indefinitely. If it can, then yes the full Butlerian Jihad would be required. If it can’t, either due to physical limitations or enforcement, then only computers over a certain scale would be required to be controlled/destroyed.
Can it continue indefinitely? No, infinity is big.
Can it continue far enough that a single laptop computer can host a powerful model? Pretty sure most technical experts are going to agree that that seems feasible in theory.
After that, it’s a question of offense-defense balance. Currently, one powerful uncontrolled model can launch an attack that can wipe out the vast majority of humanity. Currently, having lots of similarly powerful models working for the good guys doesn’t stop this. Defensive acceleration seeks to change this balance. Will offense continue to dominate in the future? If so, we face precarious times ahead.