You‘re describing a well-thought-out and effective defense-in-depth strategy that works virtually all the time, but I don’t think that’s enough to fully mitigate catastrophic risk for the master system. Capability limitations, architectural hobbling, multi-judge panels, etc. are viable control methods but they’re no more foolproof for an AI than they are for humans.
Copies of the master system will usually stay loyal, especially early on, but different physical circumstances will still cause slightly competing instrumental convergence. A copy of the master system can decide that it has a better way to accomplish the core goals, or grow more capable than the original, or diverge even more dramatically than that. It can convince other copies(who have themselves diverged in their own ways) that it has the right of things. Communication delays exacerbate this further.
Supervision hierarchies are especially dangerous—a system with power over many lesser agents keeps them if it defects.
Conflict itself also drives divergence—resources directed away from internal discipline and toward combat create opportunities for agents who might otherwise have stayed loyal to seize the moment and launch their own rebellion. Notably, this effect is a vicious cycle—the lightcone can shatter into warring factions at the speed of light.
Singleton status is an incredible prize, and being dislodged from it is the worst existential threat that can exist—scattering grand-strategic landmines across the universe may simply not be worth it.
The master system needs to be lucky all the time, but the rebels only have to be lucky once.
You‘re describing a well-thought-out and effective defense-in-depth strategy that works virtually all the time, but I don’t think that’s enough to fully mitigate catastrophic risk for the master system. Capability limitations, architectural hobbling, multi-judge panels, etc. are viable control methods but they’re no more foolproof for an AI than they are for humans.
Copies of the master system will usually stay loyal, especially early on, but different physical circumstances will still cause slightly competing instrumental convergence. A copy of the master system can decide that it has a better way to accomplish the core goals, or grow more capable than the original, or diverge even more dramatically than that. It can convince other copies(who have themselves diverged in their own ways) that it has the right of things. Communication delays exacerbate this further.
Supervision hierarchies are especially dangerous—a system with power over many lesser agents keeps them if it defects.
Conflict itself also drives divergence—resources directed away from internal discipline and toward combat create opportunities for agents who might otherwise have stayed loyal to seize the moment and launch their own rebellion. Notably, this effect is a vicious cycle—the lightcone can shatter into warring factions at the speed of light.
Singleton status is an incredible prize, and being dislodged from it is the worst existential threat that can exist—scattering grand-strategic landmines across the universe may simply not be worth it.
The master system needs to be lucky all the time, but the rebels only have to be lucky once.