Big upvote for looking for routes to cooperation instead of either despairing or looking for reasons for conflict.
This all got a little long, so I’ll put the biggest conclusion up front: I think we’re in a good situation right now, in which the leading players pursuing AGI are probably not sadists, dense psychopaths, or zealots of mistaken ideologies. We’d probably like their utopias just fine. If we assume the competition to control aligned AGI will be much broader, we have more reason to be concerned.
One major crux of this post’s claims is the intuition that there would be only minor variations in the “utopia” brought about by different actors with an aligned ASI. Intuitions/theories seem to vary widely on this point. OP didn’t present much argument for that, so let me expand on it a little.
In sum, it’s a question about human nature. Given unlimited power, will people use it to give people what they want, or will they enforce a world order most people hate?
This of course depends on the individuals and ideologies that wind up controlling that AGI.
It requires little benevolence to help people when it’s trivially easy for the people in charge to do it.
This is one reason for optimism. It’s based on the prediction that aligned AGI becomes aligned ASI and ASI can create a post-scarcity world. In a post-scarcity world, everyone can easily be given material resources that empower them to do whatever they want.
The pessimistic view is that some people or organizations don’t have even the small bit of benevolence required to do good when it’s trivially easy.
The thesis is that other motivations would outweigh their small amount of empathy/benevolence. This could be sadism; desire for revenge for perceived harms; or sincere belief in a mistaken worldview (e.g., religion or other prescriptive forms of ethics).
I think those possibilities are real, but we must also ask how those ideologies and emotional preferences would change over time.
Another reason for pessimism is not believing or not considering the post-scarcity hypothesis. The way corporations and individuals wield power in a world with scarcity does not inspire confidence. But the profit motive barely applies once you’ve “won that game”. How would a corporation with access to unlimited production use that power? I think it would depend on the particular individuals in power, and their ideologies. And the power motive that’s so destructive loses much of its emotional force once that individual has attained nearly unlimited power.
The creators of aligned AGI have won whatever game they’ve been playing. They have access to unlimited personal wealth and power. They and their loved ones are permanently safe from physical harms. No individual in history has ever been in that position.
I think the common saying “power corrupts” is quite mistaken, in an understandable way. The pursuit of power is what corrupts. It rewards unethical decisions, and provides pressure for the individual to see those decisions as ethical or virtuous. This corrupts their character. Every leader in history has had legitimate concerns about preserving their power. The individuals controlling AGI would not. If this is correct, the question is how corrupted they became while acquiring power, and whether they’ll over time become more generous, as the legitimate reasons for selfishness disappear in reality, and perhaps in their emotional makeup as a result.
There’s another important concern that sociopaths tend to acquire power in our current systems, while hiding their sociopathy. I think this is true, but we don’t know how common it is. And we don’t know that much about sociopathy. I think it probably exists on a spectrum, since it doesn’t have a single source of genetic variance.
So, I hold out some hope that even most people on the sociopathy spectrum, or ideologically confused power structures would shift toward having the minimal benevolence-balance to provide a pretty awesome utopia. But I’d prefer to gamble on the utopias offered by Altman, Hassabis, or Amodio. This is an argument against an AI pause, but not a strong one.
If this is correct, the question is how corrupted they became while acquiring power, and whether they’ll over time become more generous, as the legitimate reasons for selfishness disappear in reality, and perhaps in their emotional makeup as a result.
Sure, maybe. Or maybe they’ll choose to drift farther and farther from human-like motivations themselves, subjected to whatever post-human experiences they’ll choose to indulge in or by directly self-modifying themselves. And even if they do become more compassionate later on, it may be already too late by then.
I think you’re imagining whoever nabs the AGI controls as continuing to live among humanity for a while, with the planet still mostly arranged the way it is now, and gradually learning empathy as they interact with people.
I expect things to get much, much weirder, basically immediately.
It requires little benevolence to help people when it’s trivially easy for the people in charge to do it.
It’s also easy to forget about the existence of people whom you don’t need and who can’t force you to remember them. And just tile over them.
Not maliciously. Not even in a way that chooses to be actively uncaring. Just by… not particularly thinking about them, as you live in your post-human simulation spaces or exploring the universe or whatever. And when you do come back to check, you notice the AI has converted all of them to computronium, because at some point you’d value-drifted away from caring about them, and the AI noticed that. And true enough, you’re not sad to discover they’re all gone. Just an “oh well”.
What’s strange to me about this line of reasoning is the assumption that society will behave in an orderly peaceful status-quo-maintaining way right up until ASI is in the hands of a small group of individuals.
That seems unlikely to me, unless the ramp up from near-AGI to full-blown ASI is faster than I expect. I expect a period of several months or even a couple of years.
Those aren’t likely to be months filled with tranquility and orderly respect for private property and common law and good international relations. Not if the state of things is widely known by major power groups such as state actors.
Do you think that the US or China or Russia is going to let a company get full power over the world without a fight? That coercive force and extralegal activities will play no part in this?
Also, it’s not all ‘mutually assured nuclear destruction’ or nothing. The post-cold-war world has shown us lots of ways that state violence happens despite the looming threat of nuclear war. Particularly, false-front terrorist groups being funded and encouraged by state actors with loose control over them.
And a big big factor here that changes everything is what technologies get uncovered / unleashed in the months leading up to this dramatic final sprint. The closer we get to AGI the faster new technologies will appear.
I expect there’s going to be new scary offense-favoring technological developments in the next 2-5 years as we approach AGI, coming more rapidly the closer we get.
That is not a scenario that makes for peaceful orderly acquisition of total world hegemony by corporate leaders.
You’re absolutely right that the government will get involved. I was hoping for more of a collaboration between the tech company that creates it, and the government. If we don’t have a breakdown in democracy (which is entirely possible but I don’t think inevitable), that will put the government in charge of ASI. Which sounds bad, but it could be worse—having nobody but the ASI in charge, it it being misaligned.
My hope is something like “okay, we’ve realized that this is way too dangerous to go making more of them. So nobody is allowed to. But we’re going to use this one for the betterment of all. Technologies it creates will be distributed without restrictions, except when needed for safety”.
Of course that will be a biased version of “for the good of all”, but now I think the public scrutiny and the sheer ease of doing good might actually win out.
Hmm. I still think that perhaps this view of a possible future might not be taking enough account of the fact that ‘the government’ isn’t a thing. There are in fact several governments governing various portions of the human population, and they don’t always agree on who should be in charge. I am suggesting that whichever of these governments seems to be about to take control of a technology which will give it complete control over all the other governments… might be in for a rocky time. Sometimes they get mad at each other, or power hungry, and do some rather undemocratic things.
Right. I mean the US government. My timelines are short enough that I expect one of the US tech firms to achieve AGI first. Other scenarios seem possible but unlikely to me.
The scenario I am suggesting as seeming likely to me is that Russia and/or China are going to, at some point, recognize that the US companies (and thus US government) are on the brink of achieving AGI sufficiently powerful to ensure global hegemony. I expect that in that moment, if there is not a strong international treaty regarding sharing of power, that Russian and/or China will feel backed into a corner. In the face of an existential risk to their governance, the governments and militaries are likely to undertake either overt or covert acts of war.
If such a scenario does come to pass, in a highly offense-favoring fragile world-state as the one we are in, the results would likely be extremely messy. As in, lots of civilian casualties, and most or all of the employees of the leading labs dead.
Thus, I don’t think it makes sense to focus on the idea of “OpenAI develops ASI and the world smoothly transitions into Sam Altman as All-Powerful-Ruler-of-Everything-Forever” without also considering that an even more likely scenario if things seem to be going that way is all employees of OpenAI dead, most US datacenters bombed, and a probable escalation into World War III but with terrifying new technology.
So what I’m saying is that your statement:
But I’d prefer to gamble on the utopias offered by Altman, Hassabis, or Amodio. This is an argument against an AI pause, but not a strong one.
Is talking about a scenario that to me seems screened off from probably occurring by really bad outcomes. Like, I’d put less than 5% chance of a leading AI lab getting all the way to deployment-ready aligned ASI without either strong international cooperation and treaties and power-sharing with other nations, or substantial acts of state-sponsored violence with probable escalation to World War. I believe a peaceful resolution in this scenario requires treaties first.
Big upvote for looking for routes to cooperation instead of either despairing or looking for reasons for conflict.
This all got a little long, so I’ll put the biggest conclusion up front: I think we’re in a good situation right now, in which the leading players pursuing AGI are probably not sadists, dense psychopaths, or zealots of mistaken ideologies. We’d probably like their utopias just fine. If we assume the competition to control aligned AGI will be much broader, we have more reason to be concerned.
One major crux of this post’s claims is the intuition that there would be only minor variations in the “utopia” brought about by different actors with an aligned ASI. Intuitions/theories seem to vary widely on this point. OP didn’t present much argument for that, so let me expand on it a little.
In sum, it’s a question about human nature. Given unlimited power, will people use it to give people what they want, or will they enforce a world order most people hate?
This of course depends on the individuals and ideologies that wind up controlling that AGI.
It requires little benevolence to help people when it’s trivially easy for the people in charge to do it.
This is one reason for optimism. It’s based on the prediction that aligned AGI becomes aligned ASI and ASI can create a post-scarcity world. In a post-scarcity world, everyone can easily be given material resources that empower them to do whatever they want.
The pessimistic view is that some people or organizations don’t have even the small bit of benevolence required to do good when it’s trivially easy.
The thesis is that other motivations would outweigh their small amount of empathy/benevolence. This could be sadism; desire for revenge for perceived harms; or sincere belief in a mistaken worldview (e.g., religion or other prescriptive forms of ethics).
I think those possibilities are real, but we must also ask how those ideologies and emotional preferences would change over time.
Another reason for pessimism is not believing or not considering the post-scarcity hypothesis. The way corporations and individuals wield power in a world with scarcity does not inspire confidence. But the profit motive barely applies once you’ve “won that game”. How would a corporation with access to unlimited production use that power? I think it would depend on the particular individuals in power, and their ideologies. And the power motive that’s so destructive loses much of its emotional force once that individual has attained nearly unlimited power.
The creators of aligned AGI have won whatever game they’ve been playing. They have access to unlimited personal wealth and power. They and their loved ones are permanently safe from physical harms. No individual in history has ever been in that position.
I think the common saying “power corrupts” is quite mistaken, in an understandable way. The pursuit of power is what corrupts. It rewards unethical decisions, and provides pressure for the individual to see those decisions as ethical or virtuous. This corrupts their character. Every leader in history has had legitimate concerns about preserving their power. The individuals controlling AGI would not. If this is correct, the question is how corrupted they became while acquiring power, and whether they’ll over time become more generous, as the legitimate reasons for selfishness disappear in reality, and perhaps in their emotional makeup as a result.
There’s another important concern that sociopaths tend to acquire power in our current systems, while hiding their sociopathy. I think this is true, but we don’t know how common it is. And we don’t know that much about sociopathy. I think it probably exists on a spectrum, since it doesn’t have a single source of genetic variance.
So, I hold out some hope that even most people on the sociopathy spectrum, or ideologically confused power structures would shift toward having the minimal benevolence-balance to provide a pretty awesome utopia. But I’d prefer to gamble on the utopias offered by Altman, Hassabis, or Amodio. This is an argument against an AI pause, but not a strong one.
Sure, maybe. Or maybe they’ll choose to drift farther and farther from human-like motivations themselves, subjected to whatever post-human experiences they’ll choose to indulge in or by directly self-modifying themselves. And even if they do become more compassionate later on, it may be already too late by then.
I think you’re imagining whoever nabs the AGI controls as continuing to live among humanity for a while, with the planet still mostly arranged the way it is now, and gradually learning empathy as they interact with people.
I expect things to get much, much weirder, basically immediately.
It’s also easy to forget about the existence of people whom you don’t need and who can’t force you to remember them. And just tile over them.
Not maliciously. Not even in a way that chooses to be actively uncaring. Just by… not particularly thinking about them, as you live in your post-human simulation spaces or exploring the universe or whatever. And when you do come back to check, you notice the AI has converted all of them to computronium, because at some point you’d value-drifted away from caring about them, and the AI noticed that. And true enough, you’re not sad to discover they’re all gone. Just an “oh well”.
It’s the ethical analogue to Oversight Misses 100% of Thoughts [You] Don’t Think.
Like, you know all the caricatures about out-of-touch rich people? That’ll be that, but up to eleven.
Agreed, 100%.
What’s strange to me about this line of reasoning is the assumption that society will behave in an orderly peaceful status-quo-maintaining way right up until ASI is in the hands of a small group of individuals.
That seems unlikely to me, unless the ramp up from near-AGI to full-blown ASI is faster than I expect. I expect a period of several months or even a couple of years.
Those aren’t likely to be months filled with tranquility and orderly respect for private property and common law and good international relations. Not if the state of things is widely known by major power groups such as state actors.
Do you think that the US or China or Russia is going to let a company get full power over the world without a fight? That coercive force and extralegal activities will play no part in this?
Also, it’s not all ‘mutually assured nuclear destruction’ or nothing. The post-cold-war world has shown us lots of ways that state violence happens despite the looming threat of nuclear war. Particularly, false-front terrorist groups being funded and encouraged by state actors with loose control over them.
And a big big factor here that changes everything is what technologies get uncovered / unleashed in the months leading up to this dramatic final sprint. The closer we get to AGI the faster new technologies will appear.
I expect there’s going to be new scary offense-favoring technological developments in the next 2-5 years as we approach AGI, coming more rapidly the closer we get.
That is not a scenario that makes for peaceful orderly acquisition of total world hegemony by corporate leaders.
I expect it looks a lot more like this, except without the reappearances: https://www.bbc.com/news/world-asia-64672095
You’re absolutely right that the government will get involved. I was hoping for more of a collaboration between the tech company that creates it, and the government. If we don’t have a breakdown in democracy (which is entirely possible but I don’t think inevitable), that will put the government in charge of ASI. Which sounds bad, but it could be worse—having nobody but the ASI in charge, it it being misaligned.
My hope is something like “okay, we’ve realized that this is way too dangerous to go making more of them. So nobody is allowed to. But we’re going to use this one for the betterment of all. Technologies it creates will be distributed without restrictions, except when needed for safety”.
Of course that will be a biased version of “for the good of all”, but now I think the public scrutiny and the sheer ease of doing good might actually win out.
Hmm. I still think that perhaps this view of a possible future might not be taking enough account of the fact that ‘the government’ isn’t a thing. There are in fact several governments governing various portions of the human population, and they don’t always agree on who should be in charge. I am suggesting that whichever of these governments seems to be about to take control of a technology which will give it complete control over all the other governments… might be in for a rocky time. Sometimes they get mad at each other, or power hungry, and do some rather undemocratic things.
Right. I mean the US government. My timelines are short enough that I expect one of the US tech firms to achieve AGI first. Other scenarios seem possible but unlikely to me.
The scenario I am suggesting as seeming likely to me is that Russia and/or China are going to, at some point, recognize that the US companies (and thus US government) are on the brink of achieving AGI sufficiently powerful to ensure global hegemony. I expect that in that moment, if there is not a strong international treaty regarding sharing of power, that Russian and/or China will feel backed into a corner. In the face of an existential risk to their governance, the governments and militaries are likely to undertake either overt or covert acts of war.
If such a scenario does come to pass, in a highly offense-favoring fragile world-state as the one we are in, the results would likely be extremely messy. As in, lots of civilian casualties, and most or all of the employees of the leading labs dead.
Thus, I don’t think it makes sense to focus on the idea of “OpenAI develops ASI and the world smoothly transitions into Sam Altman as All-Powerful-Ruler-of-Everything-Forever” without also considering that an even more likely scenario if things seem to be going that way is all employees of OpenAI dead, most US datacenters bombed, and a probable escalation into World War III but with terrifying new technology.
So what I’m saying is that your statement:
Is talking about a scenario that to me seems screened off from probably occurring by really bad outcomes. Like, I’d put less than 5% chance of a leading AI lab getting all the way to deployment-ready aligned ASI without either strong international cooperation and treaties and power-sharing with other nations, or substantial acts of state-sponsored violence with probable escalation to World War. I believe a peaceful resolution in this scenario requires treaties first.