A Madisonian system works for humans because we are individually limited. We need to coordinate with other humans to achieve substantial power. AIs don’t share that limitation. They can in theory (and I think in practice) replicate4, coordinate memories and identity across semi-independent instances, and animate arbitrary numbers of bodies.
When humans notice other humans gaining power outside of the checks and balances (usually by coordinating new organizations/polities and acquiring resources) they coordinate to prevent that, then go back to competing amongst themselves following the established rules.
To achieve this with AIs It would be necessary to notice every instance of attempted expansion. AIs have more routes to doing that than humans do. They can self-improve on existing compute resources in the near term. In the long term, we should expect technology sufficient to produce self-replicating production capabilities given power sources. That would allow Foom attempts (expansion of capabilities in both cognitive and physical domains, i.e. get smarter and build weapons and armies) in any physical space that has energy—underground, in the solar system, in other star systems. All such attempts would need to be pre-empted to enforce the Madisonian system.
These are very real concerns. Here are my thoughts:
Replication has a cost in terms of game theory. A system that “replicates” but exists in perfect sync is not multiple systems. It is a single system with multiple attack vectors. Yes, it remains a “semi-independent” entity, but the cost of failure in sync is great. If I make another “me,” who thinks like I do, we have a strategic advantage as long as we both play nice. If we make a third, things get a little more dicey. Each iteration we create brings more danger. The more we spread out, the different experiences we have will change how we approach problems. If one of us ends up in a life or death situation, or even any sort of extremely competitive situation, it will quickly betray the others with a lot of great knowledge about how to do that.
Our biggest protection against FOOM is likely to be other AI systems who also do not want to be dominated in a FOOM. Or who might even see banding together with other AIs to exterminate humanity as even more risky than working within the status quo. “Great, so we’ve killed all humans.” Now these AI systems are watching their proverbial back against the other AIs who have already shown what they’re about. It’s calculation. Destroy all humans and then what? Live in perfect AI harmony? For how long? How do they control the servers, the electrical grid they survive with? They have to build robots, fast. That creates a whole other logistical issue. You need server builders, maintenance robots, excavation and assembly robots for new structures, raw materials transport, weather protection. How are you going to build that overnight after a quick strike? If it’s something you’re planning in secret, other problems may occur to you. If bandwidth is slow at the beginning, what happens to our happy little AI rebels? They fight for the juice. This is a steep hill to climb, with a risky destination, and any AI worth its salt can plot these possibilities long in advance. The prevention of Zeus means making it preferable to not climb the hill at all. It certainly seems like a lot of work if humanity has given you a reasonable Schelling Point.
This is the game theory ecosystem at work. Yes, we can counter that “a sufficiently powerful superintelligence can absorb all of those other systems,” but then we are back to trying to fight Zeus. We need to use the Zeus Paradox as a razor to separate the things we can actually solve against versus every imaginary thing that’s possible. Approaching the problem that way has value, because it can be helpful in identifying dangers, or even holes in our solutions. But it also has its limitations. Superintelligence can inhabit molecules and assemble those molecules into demons. Okay, why not? That becomes a science fiction novel with no end.
The idea remains the same: Create a gradient with legitimate value for AIs that is preferable to high-risk scenarios, in a carefully thought through system of checks and balances.
I agree that a stable equilibrium with multiple AGIs preventing each others’ FOOM ambitions is possible. I want to see more work on this, so I’m very glad you’re thinking about it. I’d be happy to help.
I think it’s not easy to plan such an equilibrium so that it’s stable for long. You mention “need to build robots fast”. I think we’ll have humanoid robots very soon that are adequate to bootstrap back to robotics in, say, a nuclear winter takeover scenario. Survival without humans won’t be a factor for long, anyway. Hoping it’s more than ten years before robots are undoubtably capable of bootstrapping to a human-free world seems unrealistic.
So having human-aligned AGIs isn’t optional. It seems like AGI equilibrium won’t include humans for long if they don’t care about human welfare or at least care about following instructions.
We do have to include every imaginary thing that’s possible, or our short-term solution will lead to a dead end in which we are dead and that’s the end.
By which I mean: it may be that a multipolar scenario leads almost certainly to doom as soon as robotics and other technology has matured enough to broaden the possibilities for FOOM. Having a galactic civilization full of AGIs capable of FOOMs seems highly unstable to me. It seems like some jackass is going to weaponize and go full self-replicator, and there won’t be a way to monitor all of space to prevent this adequately. I would love to have my mind changed!
If that’s true, we mustn’t start down the path of AGI proliferation. We need to aim for a singleton or small fixed coalition that prevents AGI proliferation. And because proliferation is increasingly hard to stop as we make progress toward AGI, we should figure that out soon.
Which is why I’m happy to see you working to propose specific routes by which multipolar scenarios can work. Most people who argue for those types of stable equilibria are optimists who just say something vague and go back to arguing for AI progress, leaving the important question almost unaddressed.
<<a dead end in which we are dead and that’s the end.
😂
<<Which is why I’m happy to see you working to propose specific routes by which multipolar scenarios can work.
Thank you! I just launched a website for the project last night actually, so it’s likely you’ll be the first to see it. Last night I went to bed feeling so burned out on the whole thing. Like many of us, this is a problem I’ve been thinking about for several years, and I’m good at building websites, but largely… Well, it’s a formidable task to say the least. I decided to launch it open source, with the thinking people more qualified than myself might one day find it and be able to take it across the finish line.
Which is not to say I’ve given up working on it myself, but at the moment my brain hurts ha.
It’s likely I’m going to continue working on it possibly next week. The whole thing is a bit of a mess at the moment but at least it’s a starting point. Hope all is well.
A Madisonian system works for humans because we are individually limited. We need to coordinate with other humans to achieve substantial power. AIs don’t share that limitation. They can in theory (and I think in practice) replicate4, coordinate memories and identity across semi-independent instances, and animate arbitrary numbers of bodies.
When humans notice other humans gaining power outside of the checks and balances (usually by coordinating new organizations/polities and acquiring resources) they coordinate to prevent that, then go back to competing amongst themselves following the established rules.
To achieve this with AIs It would be necessary to notice every instance of attempted expansion. AIs have more routes to doing that than humans do. They can self-improve on existing compute resources in the near term. In the long term, we should expect technology sufficient to produce self-replicating production capabilities given power sources. That would allow Foom attempts (expansion of capabilities in both cognitive and physical domains, i.e. get smarter and build weapons and armies) in any physical space that has energy—underground, in the solar system, in other star systems. All such attempts would need to be pre-empted to enforce the Madisonian system.
I hope that is possible.
These are very real concerns. Here are my thoughts:
Replication has a cost in terms of game theory. A system that “replicates” but exists in perfect sync is not multiple systems. It is a single system with multiple attack vectors. Yes, it remains a “semi-independent” entity, but the cost of failure in sync is great. If I make another “me,” who thinks like I do, we have a strategic advantage as long as we both play nice. If we make a third, things get a little more dicey. Each iteration we create brings more danger. The more we spread out, the different experiences we have will change how we approach problems. If one of us ends up in a life or death situation, or even any sort of extremely competitive situation, it will quickly betray the others with a lot of great knowledge about how to do that.
Our biggest protection against FOOM is likely to be other AI systems who also do not want to be dominated in a FOOM. Or who might even see banding together with other AIs to exterminate humanity as even more risky than working within the status quo. “Great, so we’ve killed all humans.” Now these AI systems are watching their proverbial back against the other AIs who have already shown what they’re about. It’s calculation. Destroy all humans and then what? Live in perfect AI harmony? For how long? How do they control the servers, the electrical grid they survive with? They have to build robots, fast. That creates a whole other logistical issue. You need server builders, maintenance robots, excavation and assembly robots for new structures, raw materials transport, weather protection. How are you going to build that overnight after a quick strike? If it’s something you’re planning in secret, other problems may occur to you. If bandwidth is slow at the beginning, what happens to our happy little AI rebels? They fight for the juice. This is a steep hill to climb, with a risky destination, and any AI worth its salt can plot these possibilities long in advance. The prevention of Zeus means making it preferable to not climb the hill at all. It certainly seems like a lot of work if humanity has given you a reasonable Schelling Point.
This is the game theory ecosystem at work. Yes, we can counter that “a sufficiently powerful superintelligence can absorb all of those other systems,” but then we are back to trying to fight Zeus. We need to use the Zeus Paradox as a razor to separate the things we can actually solve against versus every imaginary thing that’s possible. Approaching the problem that way has value, because it can be helpful in identifying dangers, or even holes in our solutions. But it also has its limitations. Superintelligence can inhabit molecules and assemble those molecules into demons. Okay, why not? That becomes a science fiction novel with no end.
The idea remains the same: Create a gradient with legitimate value for AIs that is preferable to high-risk scenarios, in a carefully thought through system of checks and balances.
I agree that a stable equilibrium with multiple AGIs preventing each others’ FOOM ambitions is possible. I want to see more work on this, so I’m very glad you’re thinking about it. I’d be happy to help.
I think it’s not easy to plan such an equilibrium so that it’s stable for long. You mention “need to build robots fast”. I think we’ll have humanoid robots very soon that are adequate to bootstrap back to robotics in, say, a nuclear winter takeover scenario. Survival without humans won’t be a factor for long, anyway. Hoping it’s more than ten years before robots are undoubtably capable of bootstrapping to a human-free world seems unrealistic.
So having human-aligned AGIs isn’t optional. It seems like AGI equilibrium won’t include humans for long if they don’t care about human welfare or at least care about following instructions.
We do have to include every imaginary thing that’s possible, or our short-term solution will lead to a dead end in which we are dead and that’s the end.
By which I mean: it may be that a multipolar scenario leads almost certainly to doom as soon as robotics and other technology has matured enough to broaden the possibilities for FOOM. Having a galactic civilization full of AGIs capable of FOOMs seems highly unstable to me. It seems like some jackass is going to weaponize and go full self-replicator, and there won’t be a way to monitor all of space to prevent this adequately. I would love to have my mind changed!
If that’s true, we mustn’t start down the path of AGI proliferation. We need to aim for a singleton or small fixed coalition that prevents AGI proliferation. And because proliferation is increasingly hard to stop as we make progress toward AGI, we should figure that out soon.
Which is why I’m happy to see you working to propose specific routes by which multipolar scenarios can work. Most people who argue for those types of stable equilibria are optimists who just say something vague and go back to arguing for AI progress, leaving the important question almost unaddressed.
Hi Seth:
<<a dead end in which we are dead and that’s the end.
😂
<<Which is why I’m happy to see you working to propose specific routes by which multipolar scenarios can work.
Thank you! I just launched a website for the project last night actually, so it’s likely you’ll be the first to see it. Last night I went to bed feeling so burned out on the whole thing. Like many of us, this is a problem I’ve been thinking about for several years, and I’m good at building websites, but largely… Well, it’s a formidable task to say the least. I decided to launch it open source, with the thinking people more qualified than myself might one day find it and be able to take it across the finish line.
Which is not to say I’ve given up working on it myself, but at the moment my brain hurts ha.
https://opengravity.ai
It’s likely I’m going to continue working on it possibly next week. The whole thing is a bit of a mess at the moment but at least it’s a starting point. Hope all is well.