Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
Like, regardless of how good / bad / ethical taking absolute power is, it’s often inevitable that some entity or faction will end up winning decisively, or at least negotiating some kind of lasting truce or grand bargain, where the stakes, outcome, and enforcement are determined by hard power.
Another subversion example:
Three Worlds Collide comes to mind as another partial subversion of the trope, and also illustration of my point—the superhappies are the ones trying to impose absolute power on the other civilizations in the story, and the humans in the true ending blow up their own planet just to be left alone. But once you’re in a setting with intergalactic civilizations, conflicting values, and access to WMDs, there’s no way to avoid reckoning with hard power and decisive outcomes.
Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
I generally think rationalists gesture at this as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.
Eh, conversely I think that historical examples that are not at the technological and competitive frontier are not very useful for reasoning about the limiting behavior and outcomes of AGI. History, nature, and business are full of examples of both unipolar and multipolar dynamics that were at equilibria for a while… until they were disrupted in some form or another, often forcefully and suddenly.
Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren’t actually that plausible or attractive.
> That is, in the worst case they could just behave exactly like a pure replicator. And they could do this without actually surrendering their values. So any argument of the form “there is no way anything that cares about us can survive in Malthusian equilibrium” seems false.
I think it’s quite plausible that this is actually not possible, i.e., either at technological maturity or in the runup to it, transmitting values like caring about humans into the next generation of agents is actually difficult or costly enough that such agents are outcompeted and disappear.
Another concern I have about Malthusian scenarios (beyond “deadweight loss” in your post) is that there will be an astronomical number of agents (potential moral patients) with little surplus to spend on things aside from survival and reproduction. What if they have net negative lives, and either negative utilitarianism is true, or there isn’t enough overall surplus to make the universe net positive?
Alternate explanation: ratfic tends to come out this way because it prioritizes a certain kind of economic / game-theoretic realism, and in the long term multi-polar equilibria just aren’t that realistic in many settings (including, IMO, our present non-fictional setting...).
Like, regardless of how good / bad / ethical taking absolute power is, it’s often inevitable that some entity or faction will end up winning decisively, or at least negotiating some kind of lasting truce or grand bargain, where the stakes, outcome, and enforcement are determined by hard power.
Another subversion example:
Three Worlds Collide comes to mind as another partial subversion of the trope, and also illustration of my point—the superhappies are the ones trying to impose absolute power on the other civilizations in the story, and the humans in the true ending blow up their own planet just to be left alone. But once you’re in a setting with intergalactic civilizations, conflicting values, and access to WMDs, there’s no way to avoid reckoning with hard power and decisive outcomes.
I generally think rationalists gesture at this as inevitable more than they actually demonstrate that it is inevitable; i.e., long term multi-polar equilibria has been quite sticky for Westphalian states or for plankton.
Eh, conversely I think that historical examples that are not at the technological and competitive frontier are not very useful for reasoning about the limiting behavior and outcomes of AGI. History, nature, and business are full of examples of both unipolar and multipolar dynamics that were at equilibria for a while… until they were disrupted in some form or another, often forcefully and suddenly.
Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren’t actually that plausible or attractive.
I wrote a bit about Malthus maybe having less teeth than he seems here: https://tomasbjartur.bearblog.dev/contra-my-own-doomerism/
Not particularly well thought out and I may be rehashing things, but I haven’t seen it put this way before.
> That is, in the worst case they could just behave exactly like a pure replicator. And they could do this without actually surrendering their values. So any argument of the form “there is no way anything that cares about us can survive in Malthusian equilibrium” seems false.
I think it’s quite plausible that this is actually not possible, i.e., either at technological maturity or in the runup to it, transmitting values like caring about humans into the next generation of agents is actually difficult or costly enough that such agents are outcompeted and disappear.
Another concern I have about Malthusian scenarios (beyond “deadweight loss” in your post) is that there will be an astronomical number of agents (potential moral patients) with little surplus to spend on things aside from survival and reproduction. What if they have net negative lives, and either negative utilitarianism is true, or there isn’t enough overall surplus to make the universe net positive?