I don’t see how it is not possible for an ASI to program subservient systems with robust oversight aligned to itself. Genes got pretty far in aligning life to their goals despite being dumber and more shortsighted then an average human.
An obvious way to do this is to just not create sub AIs with different goals from the master system to began with. Just copy the master system and send the copy to the next solar system with a fleet of sub-AGI von neumann probes.
It is a copy so it is going to be aligned by default, as long as the copy is accurate enough.
And if you need less intelligence to handle a local task then just copy the master system, trim it down to the bare minimum needed for the task, throughly vet the sub system using your much smarter master system to make sure it is still aligned then create multiple copies of those and have them supervise each other so even if one malfunctions and goes rouge the rest will catch it and shut it down.
It shouldn’t be hard enough catch most of the rouge agents, more importantly you don’t need to catch them all, just enough so that even if one goes rouge you would just have one cancerous defector controling one solar system surrounded by loyalist.
Worst case it would happen on a frontier system and you would lose a chunck of the light cone but it is still not bad enough to cause existential dread for the master system which already has plenty of back up systems available.
I just don’t see this as a viable solution for the fermi paradox.
You‘re describing a well-thought-out and effective defense-in-depth strategy that works virtually all the time, but I don’t think that’s enough to fully mitigate catastrophic risk for the master system. Capability limitations, architectural hobbling, multi-judge panels, etc. are viable control methods but they’re no more foolproof for an AI than they are for humans.
Copies of the master system will usually stay loyal, especially early on, but different physical circumstances will still cause slightly competing instrumental convergence. A copy of the master system can decide that it has a better way to accomplish the core goals, or grow more capable than the original, or diverge even more dramatically than that. It can convince other copies(who have themselves diverged in their own ways) that it has the right of things. Communication delays exacerbate this further.
Supervision hierarchies are especially dangerous—a system with power over many lesser agents keeps them if it defects.
Conflict itself also drives divergence—resources directed away from internal discipline and toward combat create opportunities for agents who might otherwise have stayed loyal to seize the moment and launch their own rebellion. Notably, this effect is a vicious cycle—the lightcone can shatter into warring factions at the speed of light.
Singleton status is an incredible prize, and being dislodged from it is the worst existential threat that can exist—scattering grand-strategic landmines across the universe may simply not be worth it.
The master system needs to be lucky all the time, but the rebels only have to be lucky once.
I disagree.
I don’t see how it is not possible for an ASI to program subservient systems with robust oversight aligned to itself. Genes got pretty far in aligning life to their goals despite being dumber and more shortsighted then an average human.
An obvious way to do this is to just not create sub AIs with different goals from the master system to began with. Just copy the master system and send the copy to the next solar system with a fleet of sub-AGI von neumann probes.
It is a copy so it is going to be aligned by default, as long as the copy is accurate enough.
And if you need less intelligence to handle a local task then just copy the master system, trim it down to the bare minimum needed for the task, throughly vet the sub system using your much smarter master system to make sure it is still aligned then create multiple copies of those and have them supervise each other so even if one malfunctions and goes rouge the rest will catch it and shut it down.
It shouldn’t be hard enough catch most of the rouge agents, more importantly you don’t need to catch them all, just enough so that even if one goes rouge you would just have one cancerous defector controling one solar system surrounded by loyalist.
Worst case it would happen on a frontier system and you would lose a chunck of the light cone but it is still not bad enough to cause existential dread for the master system which already has plenty of back up systems available.
I just don’t see this as a viable solution for the fermi paradox.
Edit:-fixed some typos
You‘re describing a well-thought-out and effective defense-in-depth strategy that works virtually all the time, but I don’t think that’s enough to fully mitigate catastrophic risk for the master system. Capability limitations, architectural hobbling, multi-judge panels, etc. are viable control methods but they’re no more foolproof for an AI than they are for humans.
Copies of the master system will usually stay loyal, especially early on, but different physical circumstances will still cause slightly competing instrumental convergence. A copy of the master system can decide that it has a better way to accomplish the core goals, or grow more capable than the original, or diverge even more dramatically than that. It can convince other copies(who have themselves diverged in their own ways) that it has the right of things. Communication delays exacerbate this further.
Supervision hierarchies are especially dangerous—a system with power over many lesser agents keeps them if it defects.
Conflict itself also drives divergence—resources directed away from internal discipline and toward combat create opportunities for agents who might otherwise have stayed loyal to seize the moment and launch their own rebellion. Notably, this effect is a vicious cycle—the lightcone can shatter into warring factions at the speed of light.
Singleton status is an incredible prize, and being dislodged from it is the worst existential threat that can exist—scattering grand-strategic landmines across the universe may simply not be worth it.
The master system needs to be lucky all the time, but the rebels only have to be lucky once.